upgrade from awx 13 to 14, postgres failed to restart #7854

linuxcube · 2020-08-06T19:37:38Z

ISSUE TYPE

Bug Report

SUMMARY

postgres failed to restart from upgrade of awx v13 to v14

ENVIRONMENT

AWX version: 14
AWX install method: docker
Ansible version: 2.8.3-1
Operating System: RHEL 8.2
Web Browser: Chrome/Firefox

STEPS TO REPRODUCE

docker stop ansible/awx:14.0.0 ansible/awx:14.0.0 redis postgres:10
rm rf /root/awx
git clone https://github.com/ansible/awx.git
cd awx/installer/
ansible-playbook -i inventory install.yml
docker ps

EXPECTED RESULTS

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0effec6e2ab ansible/awx:14.0.0 "tini -- /usr/bin/la…" 31 minutes ago Up About a minute 8052/tcp awx_task
a3a328a5ffc3 ansible/awx:14.0.0 "tini -- /bin/sh -c …" 31 minutes ago Up About a minute 0.0.0.0:80->8052/tcp awx_web
8dd7063400a8 postgres:10 "docker-entrypoint.s…" 32 minutes ago Restarting (1) 30 seconds ago awx_postgres
e0eb72177943 redis "docker-entrypoint.s…" 2 weeks ago Up About a minute 6379/tcp awx_redis

ACTUAL RESULTS

postgres stuck in restart mode

ADDITIONAL INFORMATION

No additional details collected

ryanpetrello · 2020-08-06T19:45:53Z

docker logs 8dd7063400a8 ?

shanemcd · 2020-08-06T19:47:36Z

Did you copy over your old inventory file with your passwords?

ryanpetrello · 2020-08-06T19:50:56Z

What @shanemcd said:

https://github.com/ansible/awx/blob/devel/INSTALL.md#upgrading-from-previous-versions

If you blew away your inventory file, upgrading is unlikely to work.

agrisvv · 2020-08-06T20:11:17Z

centos 8 from awx 13 to 14 after upgrade as usual with old inventory settings.
postgres docker restarting..


Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".

linuxcube · 2020-08-06T20:15:59Z

I did both, with and without old inventoy file....the outcome was the same. Postgres not starting.

…

On Thu, Aug 6, 2020, 12:47 PM Shane McDonald ***@***.***> wrote: Did you copy over your old inventory file with your passwords? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7854 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7QQLFV3OJZMEPPECVMPITR7MCGNANCNFSM4PW5EJ5A> .

linuxcube · 2020-08-06T20:16:57Z

Hi agris....looks like some of us are in the same boat as you after upgrade.

…

On Thu, Aug 6, 2020, 1:11 PM Agris ***@***.***> wrote: centos 8 from awx 13 to 14 after upgrade as usual with old inventory settings. postgres docker restarting.. Data page checksums are disabled. initdb: directory "/var/lib/postgresql/data" exists but is not empty If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. initdb: directory "/var/lib/postgresql/data" exists but is not empty If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7854 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7QQLH5NXF4OC4BIQIC3LDR7ME7HANCNFSM4PW5EJ5A> .

egmar · 2020-08-06T20:24:48Z

@linuxphone see this change 06fd131 , perhaps its the cause

agrisvv · 2020-08-06T21:01:21Z

@linuxphone see this change 06fd131 , perhaps its the cause

modified installer reverted changes as in commit and installed again, but now it started with empty db.. will try to check where it gone..

linuxcube · 2020-08-06T21:35:32Z

Hi Agris, That entry is already committed to the latest code, so this is not the solution….the issue remains. From: Agris <notifications@github.com> Sent: Thursday, August 6, 2020 2:02 PM To: ansible/awx <awx@noreply.github.com> Cc: linuxphone <lgrullonbb@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [ansible/awx] upgrade from awx 13 to 14, postgres failed to restart (#7854) @linuxphone <https://github.com/linuxphone> see this change 06fd131 <06fd131> , perhaps its the cause modified installer reverted changes as in commit and installed again, but now it started with empty db.. will try to check where it gone.. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7854 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7QQLFHZ7LP6GUV5YCSK73R7MK3BANCNFSM4PW5EJ5A> . <https://github.com/notifications/beacon/AD7QQLDJYSRIPIXRITGCECTR7MK3BA5CNFSM4PW5EJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOE7ZERNA.gif>

amtilghman · 2020-08-06T22:56:54Z

I tested a devel build 2 days ago and had the same issue, was thinking it would be fixed before 14.0.0 (or i would have made an issue :) )

Just tried 12.0.0 > 14.0.0

Error was identical to person above:

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

linuxcube · 2020-08-06T22:59:14Z

Hi Amtilghman, It looks like we need a small patch for this….hopefully not too many users start upgrading and face the same issue. From: amtilghman <notifications@github.com> Sent: Thursday, August 6, 2020 3:57 PM To: ansible/awx <awx@noreply.github.com> Cc: linuxphone <lgrullonbb@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [ansible/awx] upgrade from awx 13 to 14, postgres failed to restart (#7854) I tested a devel build 2 days ago and had the same issue, was hoping it would be fixed before 14.0.0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7854 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7QQLHXH2DEVXNNR2NELOTR7MYMFANCNFSM4PW5EJ5A> . <https://github.com/notifications/beacon/AD7QQLDQQUETWVCB6WESZVTR7MYMFA5CNFSM4PW5EJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOE7ZOSQI.gif>

ekinof · 2020-08-07T09:16:32Z

Same with me, I just tried to upgrade from 12.0.0 to 14.0.0, my postgres container doesn't work. I downgrade to 13.0.0 and now it works.

kenchrcum · 2020-08-07T09:59:01Z

Hello,
I had the same issue, I solved it by using
docker-compose down && docker-compose up -d
It seemed to me the problem happened because the old AWX was still running when I started the installer.

Goeger · 2020-08-07T10:07:42Z

@linuxphone see this change 06fd131 , perhaps its the cause

had the same issue and to change this file to the old version solved it for me

dotslashcmd · 2020-08-07T11:06:52Z

It seems that in the docker-compose generated by the V14 installer, the env variable PGDATA for the postgres container is missing.

Docker-compose v 13 :

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data/pgdata:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      PGDATA: /var/lib/postgresql/data/pgdata
      http_proxy: 
      https_proxy: 
      no_proxy:

Docker-compose v 14 :

postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:

linuxcube · 2020-08-07T13:06:36Z

@linuxphone see this change 06fd131 , perhaps its the cause

had the same issue and to change this file to the old version solved it for me

This didnt fix it for me, I tried putting back the old entries and reinstalling....same issue with postgress...it wont restart normally.

linuxcube · 2020-08-07T13:10:21Z

It seems that in the docker-compose generated by the V14 installer, the env variable PGDATA for the postgres container is missing.

Docker-compose v 13 :

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data/pgdata:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      PGDATA: /var/lib/postgresql/data/pgdata
      http_proxy: 
      https_proxy: 
      no_proxy:

Docker-compose v 14 :

postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:

Putting the old entries back, appears to fix it....I will try again in my VAL environment.....for those having the issues, ensure you put back the old entries from released v13 and rerun the installer. Will report back shortly with my VAL results.

linuxcube · 2020-08-07T13:14:00Z

It seems that in the docker-compose generated by the V14 installer, the env variable PGDATA for the postgres container is missing.

Docker-compose v 13 :

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data/pgdata:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      PGDATA: /var/lib/postgresql/data/pgdata
      http_proxy: 
      https_proxy: 
      no_proxy:

Docker-compose v 14 :

postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:

Team, I was able to fix this under my VAL environment...its working as expected....thank you all for the support.

bile0026 · 2020-08-10T13:51:15Z

Same issue here upgrading from 12.0.0 > 14.0.0. Postgres won't start.

cd into ~/.awx/awxcompose and running docker-compose down && docker-compose up -d did solve the issue though, so looks like I'm back in business.

shanemcd · 2020-08-10T14:57:23Z

Hi folks. Apologies for this, the breakage was not intentional. It does seem like this broke in 06fd131. Since fixing this would lead to more complications for future upgrades, I am inclined to roll forward and direct people towards the workarounds described above. Sorry again and thank you for living on the bleeding edge of a fast-moving project. 😄

ghost · 2020-09-01T17:26:03Z

@shanemcd Why is this not specified in the upgrade documentation or at least the release notes?

If the upgrade is no longer seamless it needs to be mentioned somewhere that going from < v14 to v14 or higher requires additional manual intervention.

ryanpetrello · 2020-09-01T17:31:37Z

@nvonwolf this isn't an issue we noticed until users reported it on upgrade (which is after we created the changelog/release notes)

Does this seem reasonable cc @shanemcd?

#8054

ghost · 2020-09-01T17:35:23Z

Thanks for the quick response. Issue is solved for me after finding this thread but other may not be so lucky.

danielbostock · 2020-10-19T06:42:23Z

Same issue here upgrading from 12.0.0 > 14.0.0. Postgres won't start.

cd into ~/.awx/awxcompose and running docker-compose down && docker-compose up -d did solve the issue though, so looks like I'm back in business.

This also fixed my issue. Thanks.

awxbot added the type:bug label Aug 6, 2020

blomquisg added the component:installer label Aug 7, 2020

blomquisg assigned Spredzy Aug 7, 2020

ryanpetrello unassigned Spredzy Aug 10, 2020

shanemcd closed this as completed Aug 10, 2020

jmaughmer mentioned this issue Aug 14, 2020

docker-compose upgrading from previous versions to 14.0.0 is broken (awx_postgres container) #7906

Closed

shanemcd mentioned this issue Oct 20, 2020

Awx 13.x to 15.x upgrade....Data not being migrated #8418

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade from awx 13 to 14, postgres failed to restart #7854

upgrade from awx 13 to 14, postgres failed to restart #7854

linuxcube commented Aug 6, 2020

ryanpetrello commented Aug 6, 2020

shanemcd commented Aug 6, 2020

ryanpetrello commented Aug 6, 2020 •

edited

Loading

agrisvv commented Aug 6, 2020

linuxcube commented Aug 6, 2020 via email

linuxcube commented Aug 6, 2020 via email

egmar commented Aug 6, 2020

agrisvv commented Aug 6, 2020

linuxcube commented Aug 6, 2020 via email

amtilghman commented Aug 6, 2020 •

edited

Loading

linuxcube commented Aug 6, 2020 via email

ekinof commented Aug 7, 2020

kenchrcum commented Aug 7, 2020

Goeger commented Aug 7, 2020

dotslashcmd commented Aug 7, 2020 •

edited

Loading

linuxcube commented Aug 7, 2020

linuxcube commented Aug 7, 2020

linuxcube commented Aug 7, 2020

bile0026 commented Aug 10, 2020 •

edited

Loading

shanemcd commented Aug 10, 2020 •

edited

Loading

ghost commented Sep 1, 2020

ryanpetrello commented Sep 1, 2020

ghost commented Sep 1, 2020

danielbostock commented Oct 19, 2020

upgrade from awx 13 to 14, postgres failed to restart #7854

upgrade from awx 13 to 14, postgres failed to restart #7854

Comments

linuxcube commented Aug 6, 2020

ISSUE TYPE

SUMMARY

ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ADDITIONAL INFORMATION

ryanpetrello commented Aug 6, 2020

shanemcd commented Aug 6, 2020

ryanpetrello commented Aug 6, 2020 • edited Loading

agrisvv commented Aug 6, 2020

linuxcube commented Aug 6, 2020 via email

linuxcube commented Aug 6, 2020 via email

egmar commented Aug 6, 2020

agrisvv commented Aug 6, 2020

linuxcube commented Aug 6, 2020 via email

amtilghman commented Aug 6, 2020 • edited Loading

linuxcube commented Aug 6, 2020 via email

ekinof commented Aug 7, 2020

kenchrcum commented Aug 7, 2020

Goeger commented Aug 7, 2020

dotslashcmd commented Aug 7, 2020 • edited Loading

linuxcube commented Aug 7, 2020

linuxcube commented Aug 7, 2020

linuxcube commented Aug 7, 2020

bile0026 commented Aug 10, 2020 • edited Loading

shanemcd commented Aug 10, 2020 • edited Loading

ghost commented Sep 1, 2020

ryanpetrello commented Sep 1, 2020

ghost commented Sep 1, 2020

danielbostock commented Oct 19, 2020

ryanpetrello commented Aug 6, 2020 •

edited

Loading

amtilghman commented Aug 6, 2020 •

edited

Loading

dotslashcmd commented Aug 7, 2020 •

edited

Loading

bile0026 commented Aug 10, 2020 •

edited

Loading

shanemcd commented Aug 10, 2020 •

edited

Loading