Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when upgrading awx-operator from 2.1.0 to 2.2.0 #1673

Open
3 tasks done
racheldaguenethahn opened this issue Dec 21, 2023 · 10 comments
Open
3 tasks done

Error when upgrading awx-operator from 2.1.0 to 2.2.0 #1673

racheldaguenethahn opened this issue Dec 21, 2023 · 10 comments

Comments

@racheldaguenethahn
Copy link

racheldaguenethahn commented Dec 21, 2023

Please confirm the following

  • I agree to follow this project's code of conduct.
  • I have checked the current issues for duplicates.
  • I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hi,

I'm unable to upgrade AWX instance due to an issue on the migration of the managed postgresql DB.
Something is wrong into the main_projectupdateevent table.
Does someone know how to solve this? Table entries to delete?
Logs from the awx-manager container of awx-operator 2.2.0 pod:

 TASK [Migrate the database if the K8s resources were updated.] ********************************
fatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last):
File \"/usr/bin/awx-manage\", line 8, in <module>
  sys.exit(manage())
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage
  execute_from_command_line(sys.argv)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line
  utility.execute()
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv
  self.execute(*args, **cmd_options)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute
  output = self.handle(*args, **options)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper
  res = handle_func(*args, **kwargs)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle
  post_migrate_state = executor.migrate(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate
  state = self._migrate_all_forwards(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards
  state = self.apply_migration(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration
  state = migration.apply(state, schema_editor)
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply
  operation.database_forwards(
File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards
  raise ValueError(\nValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).\n", "stderr_lines": ["Traceback (most recent call last):", "  File \"/usr/bin/awx-manage\", line 8, in <module>", "    sys.exit(manage())", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage", "    execute_from_command_line(sys.argv)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line", "    utility.execute()", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", "    post_migrate_state = executor.migrate(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", "    state = self._migrate_all_forwards(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", "    state = self.apply_migration(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", "    operation.database_forwards(", "  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards", "    raise ValueError(", "ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid)."], "stdout": "Operations to perform:
Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations:
Applying main.0184_django_indexes...", "stdout_lines": ["Operations to perform:", "  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", "  Applying main.0184_django_indexes..."]}

Thank in advance
Rachel

AWX Operator version

2.1.0

AWX version

22.2.0

Kubernetes platform

openshift

Kubernetes/Platform version

4.12.37

Modifications

no

Steps to reproduce

helm pull awx-operator/awx-operator --version 2.2.0
tar -zxf awx-operator-2.2.0.tgz

kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxbackups.awx.ansible.com.yaml
kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxrestores.awx.ansible.com.yaml
kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxs.awx.ansible.com.yaml

helm upgrade --install awx-operator ./awx-operator

Expected results

awx-operator is upgraded successfully
AWX instance is upgraded successfully

Actual results

awx-operator is upgraded
AWX instance is waiting for database migration

The following errors appears into awx-operator/awx-manager container

`
TASK [Migrate the database if the K8s resources were updated.] ********************************
fatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py", line 200, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
res = handle_func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
post_migrate_state = executor.migrate(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
state = self._migrate_all_forwards(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
state = self.apply_migration(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
state = migration.apply(state, schema_editor)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
operation.database_forwards(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py", line 977, in database_forwards
raise ValueError(\nValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).\n", "stderr_lines": ["Traceback (most recent call last):", " File "/usr/bin/awx-manage", line 8, in ", " sys.exit(manage())", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py", line 200, in manage", " execute_from_command_line(sys.argv)", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 442, in execute_from_command_line", " utility.execute()", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 436, in execute", " self.fetch_command(subcommand).run_from_argv(self.argv)", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv", " self.execute(*args, **cmd_options)", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute", " output = self.handle(*args, **options)", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle", " post_migrate_state = executor.migrate(", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate", " state = self._migrate_all_forwards(", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards", " state = self.apply_migration(", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration", " state = migration.apply(state, schema_editor)", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply", " operation.database_forwards(", " File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py", line 977, in database_forwards", " raise ValueError(", "ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid)."], "stdout": "Operations to perform:
Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations:
Applying main.0184_django_indexes...", "stdout_lines": ["Operations to perform:", " Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", " Applying main.0184_django_indexes..."]}

`

Additional information

No response

Operator Logs

No response

@jessicamack
Copy link
Member

Hello @racheldaguenethahn, we've had several fixes around similar issues. Are you able to upgrade to the latest version?

@racheldaguenethahn
Copy link
Author

racheldaguenethahn commented Jan 8, 2024

Hi @jessicamack , thank you for your answer.
Yes I can upgrade to the latest version.
Do you have a recommended upgrade path? Or can I upgrade directly from awx-operator 2.1.0 to awx-operator 2.10.0?

@fosterseth
Copy link
Member

you should be able to upgrade from 2.1.0 to 2.10.0

@racheldaguenethahn
Copy link
Author

Hi, I upgraded awx-operator from 2.1.0 to 2.10.0.
The upgrade succeeded but now awx is blocked on a page saying that it's upgrading.
When I try to access the URL, I'm redirected to https://privateurl/migrations_notran/ (see attachment)
awx

From awx-web pod I can see the following error

192.168.84.1 - - [15/Jan/2024:08:11:27 +0000] "GET /static/media/logo-black.svg HTTP/1.1" 404 548 "https://privateurl/migrations_notran/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" "172.27.0.250" 2024/01/15 08:11:27 [error] 19#19: *92 open() "/var/lib/awx/public/static//media/logo-black.svg" failed (2: No such file or directory), client: 192.168.84.1, server: _, request: "GET /static/media/logo-black.svg HTTP/1.1", host: "privateurl", referrer: "https://privateurl/migrations_notran/" 192.168.84.1 - - [15/Jan/2024:08:11:37 +0000] "GET / HTTP/1.1" 302 5 "https://privateurl/migrations_notran/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" "172.27.0.250" [pid: 24|app: 0|req: 36/98] 192.168.84.1 () {64 vars in 1342 bytes} [Mon Jan 15 08:11:37 2024] GET / => generated 0 bytes in 77 msecs (HTTP/1.1 302) 5 headers in 220 bytes (1 switches on core 0) 192.168.84.1 - - [15/Jan/2024:08:11:38 +0000] "GET /migrations_notran/ HTTP/1.1" 200 2112 "https://privateurl/migrations_notran/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" "172.27.0.250" [pid: 22|app: 0|req: 36/99] 192.168.84.1 () {66 vars in 1378 bytes} [Mon Jan 15 08:11:37 2024] GET /migrations_notran/ => generated 2112 bytes in 71 msecs (HTTP/1.1 200) 7 headers in 261 bytes (1 switches on core 0)

@jessicamack
Copy link
Member

The migrations_notrans URL is served when migrations haven't finished. This is usually because migrations are in progress but have not finished, however, if migrations error then it'll also serve that page. Can you provide the logs from the awx-task and postgres containers as well?

@khorn7sk
Copy link

@racheldaguenethahn I have the same issue, in my case pod awx-task- doesn't run(by security restrictions).
Can you check? awx-task up and running?

@racheldaguenethahn
Copy link
Author

@jessicamack I cannot provide you the logs from my production environment as I reverted to version 2.1.0.

However, I deployed another awx-operator 2.1.0 + awx instance into a sandbox cluster. This instance uses an unmanaged postgresql DB containing a backup of the production environment.
Then, I upgraded this sandbox instance to 2.10.0 and I have the same result as in production => stucked on migration.

Find in attachment the logs of the postgres13 pod.
postgres13.log

The logs from awx-task pod are not very useful...

[wait-for-migrations] Waiting for database migrations... [wait-for-migrations] Attempt 1 [wait-for-migrations] Waiting 0.5 seconds before next attempt [wait-for-migrations] Attempt 2 [wait-for-migrations] Waiting 1 seconds before next attempt [wait-for-migrations] Attempt 3 [wait-for-migrations] Waiting 2 seconds before next attempt [wait-for-migrations] Attempt 4 [wait-for-migrations] Waiting 4 seconds before next attempt [wait-for-migrations] Attempt 5 [wait-for-migrations] Waiting 8 seconds before next attempt [wait-for-migrations] Attempt 6 [wait-for-migrations] Waiting 16 seconds before next attempt [wait-for-migrations] Attempt 7 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 8 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 9 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 10 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 11 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 12 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 13 [wait-for-migrations] Waiting 30 seconds before next attempt [wait-for-migrations] Attempt 14

@racheldaguenethahn
Copy link
Author

@khorn7sk yes the awx-task pod is up and running but waiting for the DB migration

@racheldaguenethahn
Copy link
Author

Hi, I saw that there was a new version for awx-operator.
I tried to upgrade awx-operator from 2.1.0 to 2.11.0

It still fails on the DB with the following issues

`
2024-02-06 10:47:47.449 UTC [76] ERROR: relation "conf_setting" does not exist at character 158
2024-02-06 10:47:47.449 UTC [76] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_HOST' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1
2024-02-06 10:47:47.451 UTC [76] ERROR: relation "conf_setting" does not exist at character 158
2024-02-06 10:47:47.451 UTC [76] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_PORT' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1
2024-02-06 10:47:47.940 UTC [78] ERROR: relation "conf_setting" does not exist at character 158
2024-02-06 10:47:47.940 UTC [78] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_HOST' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1
2024-02-06 10:47:47.941 UTC [78] ERROR: relation "conf_setting" does not exist at character 158
2024-02-06 10:47:47.941 UTC [78] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'LOG_AGGREGATOR_PORT' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC LIMIT 1
2024-02-06 10:47:55.306 UTC [81] ERROR: relation "main_unifiedjob" does not exist at character 1362
2024-02-06 10:47:55.306 UTC [81] STATEMENT: SELECT "main_unifiedjob"."id", "main_unifiedjob"."polymorphic_ctype_id", "main_unifiedjob"."modified", "main_unifiedjob"."description", "main_unifiedjob"."created_by_id", "main_unifiedjob"."modified_by_id", "main_unifiedjob"."name", "main_unifiedjob"."execution_environment_id", "main_unifiedjob"."old_pk", "main_unifiedjob"."emitted_events", "main_unifiedjob"."unified_job_template_id", "main_unifiedjob"."created", "main_unifiedjob"."launch_type", "main_unifiedjob"."schedule_id", "main_unifiedjob"."execution_node", "main_unifiedjob"."controller_node", "main_unifiedjob"."cancel_flag", "main_unifiedjob"."status", "main_unifiedjob"."failed", "main_unifiedjob"."started", "main_unifiedjob"."dependencies_processed", "main_unifiedjob"."finished", "main_unifiedjob"."canceled_on", "main_unifiedjob"."elapsed", "main_unifiedjob"."job_args", "main_unifiedjob"."job_cwd", "main_unifiedjob"."job_env", "main_unifiedjob"."job_explanation", "main_unifiedjob"."start_args", "main_unifiedjob"."result_traceback", "main_unifiedjob"."celery_task_id", "main_unifiedjob"."instance_group_id", "main_unifiedjob"."preferred_instance_groups_cache", "main_unifiedjob"."task_impact", "main_unifiedjob"."organization_id", "main_unifiedjob"."installed_collections", "main_unifiedjob"."ansible_version", "main_unifiedjob"."host_status_counts", "main_unifiedjob"."work_unit_id" FROM "main_unifiedjob" WHERE ("main_unifiedjob"."dependencies_processed" AND "main_unifiedjob"."status" IN ('pending', 'waiting', 'running') AND NOT ("main_unifiedjob"."launch_type" = 'sync') AND NOT ("main_unifiedjob"."polymorphic_ctype_id" = 68 AND "main_unifiedjob"."polymorphic_ctype_id" IS NOT NULL)) ORDER BY "main_unifiedjob"."created" ASC
2024-02-06 10:47:55.316 UTC [82] ERROR: relation "django_content_type" does not exist at character 106
2024-02-06 10:47:55.316 UTC [82] STATEMENT: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE ("django_content_type"."app_label" = 'main' AND "django_content_type"."model" = 'workflowapproval') LIMIT 21
2024-02-06 10:48:05.314 UTC [86] ERROR: relation "main_towerschedulestate" does not exist at character 91
2024-02-06 10:48:05.314 UTC [86] STATEMENT: SELECT "main_towerschedulestate"."id", "main_towerschedulestate"."schedule_last_run" FROM "main_towerschedulestate" WHERE "main_towerschedulestate"."id" = 1 LIMIT 21
2024-02-06 10:48:15.329 UTC [88] ERROR: relation "django_content_type" does not exist at character 106
2024-02-06 10:48:15.329 UTC [88] STATEMENT: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE ("django_content_type"."app_label" = 'main' AND "django_content_type"."model" = 'workflowapproval') LIMIT 21
2024-02-06 10:48:15.339 UTC [89] ERROR: relation "main_unifiedjob" does not exist at character 1362
2024-02-06 10:48:15.339 UTC [89] STATEMENT: SELECT "main_unifiedjob"."id", "main_unifiedjob"."polymorphic_ctype_id", "main_unifiedjob"."modified", "main_unifiedjob"."description", "main_unifiedjob"."created_by_id", "main_unifiedjob"."modified_by_id", "main_unifiedjob"."name", "main_unifiedjob"."execution_environment_id", "main_unifiedjob"."old_pk", "main_unifiedjob"."emitted_events", "main_unifiedjob"."unified_job_template_id", "main_unifiedjob"."created", "main_unifiedjob"."launch_type", "main_unifiedjob"."schedule_id", "main_unifiedjob"."execution_node", "main_unifiedjob"."controller_node", "main_unifiedjob"."cancel_flag", "main_unifiedjob"."status", "main_unifiedjob"."failed", "main_unifiedjob"."started", "main_unifiedjob"."dependencies_processed", "main_unifiedjob"."finished", "main_unifiedjob"."canceled_on", "main_unifiedjob"."elapsed", "main_unifiedjob"."job_args", "main_unifiedjob"."job_cwd", "main_unifiedjob"."job_env", "main_unifiedjob"."job_explanation", "main_unifiedjob"."start_args", "main_unifiedjob"."result_traceback", "main_unifiedjob"."celery_task_id", "main_unifiedjob"."instance_group_id", "main_unifiedjob"."preferred_instance_groups_cache", "main_unifiedjob"."task_impact", "main_unifiedjob"."organization_id", "main_unifiedjob"."installed_collections", "main_unifiedjob"."ansible_version", "main_unifiedjob"."host_status_counts", "main_unifiedjob"."work_unit_id" FROM "main_unifiedjob" WHERE (NOT "main_unifiedjob"."dependencies_processed" AND "main_unifiedjob"."status" IN ('pending') AND NOT ("main_unifiedjob"."launch_type" = 'sync') AND NOT ("main_unifiedjob"."polymorphic_ctype_id" = 68 AND "main_unifiedjob"."polymorphic_ctype_id" IS NOT NULL)) ORDER BY "main_unifiedjob"."created" ASC
2024-02-06 10:48:35.353 UTC [91] ERROR: relation "main_instance" does not exist at character 666
2024-02-06 10:48:35.353 UTC [91] STATEMENT: SELECT "main_instance"."id", "main_instance"."uuid", "main_instance"."hostname", "main_instance"."ip_address", "main_instance"."created", "main_instance"."modified", "main_instance"."version", "main_instance"."cpu", "main_instance"."memory", "main_instance"."errors", "main_instance"."last_seen", "main_instance"."health_check_started", "main_instance"."last_health_check", "main_instance"."capacity", "main_instance"."capacity_adjustment", "main_instance"."enabled", "main_instance"."managed_by_policy", "main_instance"."cpu_capacity", "main_instance"."mem_capacity", "main_instance"."node_type", "main_instance"."node_state", "main_instance"."listener_port" FROM "main_instance" WHERE "main_instance"."node_state" IN ('ready', 'unavailable', 'installed') ORDER BY "main_instance"."hostname" ASC
2024-02-06 10:48:35.373 UTC [94] ERROR: relation "main_instancegroup" does not exist at character 666
2024-02-06 10:48:35.373 UTC [94] STATEMENT: DECLARE "_django_curs_139787606607680_sync_1" NO SCROLL CURSOR WITH HOLD FOR SELECT "main_instancegroup"."id", "main_instancegroup"."name", "main_instancegroup"."created", "main_instancegroup"."modified", "main_instancegroup"."is_container_group", "main_instancegroup"."credential_id", "main_instancegroup"."pod_spec_override", "main_instancegroup"."admin_role_id", "main_instancegroup"."use_role_id", "main_instancegroup"."read_role_id", "main_instancegroup"."max_concurrent_jobs", "main_instancegroup"."max_forks", "main_instancegroup"."policy_instance_percentage", "main_instancegroup"."policy_instance_minimum", "main_instancegroup"."policy_instance_list" FROM "main_instancegroup" WHERE "main_instancegroup"."is_container_group"
2024-02-06 10:48:35.373 UTC [92] ERROR: relation "main_towerschedulestate" does not exist at character 91
2024-02-06 10:48:35.373 UTC [92] STATEMENT: SELECT "main_towerschedulestate"."id", "main_towerschedulestate"."schedule_last_run" FROM "main_towerschedulestate" WHERE "main_towerschedulestate"."id" = 1 LIMIT 21
2024-02-06 10:48:35.375 UTC [94] ERROR: cursor "_django_curs_139787606607680_sync_1" does not exist
2024-02-06 10:48:35.375 UTC [94] STATEMENT: CLOSE "_django_curs_139787606607680_sync_1"

`

@racheldaguenethahn
Copy link
Author

racheldaguenethahn commented Apr 4, 2024

Hello, I still have this issue.
Current awx-operator version is 2.1.0.
When trying to upgrade my AWX instance by upgrading the operator to 2.2.0, it fails.

Going into the awx-web pod to run awx-manage command line I have the same issue:

bash-5.1$ awx-manage showmigrations
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
[X] 0007_alter_validators_add_error_messages
[X] 0008_alter_user_username_max_length
[X] 0009_alter_user_last_name_max_length
[X] 0010_alter_group_name_max_length
[X] 0011_update_proxy_permissions
[X] 0012_alter_user_first_name_max_length
conf
[X] 0001_initial
[X] 0002_v310_copy_tower_settings
[X] 0003_v310_JSONField_changes
[X] 0004_v320_reencrypt
[X] 0005_v330_rename_two_session_settings
[X] 0006_v331_ldap_group_type
[X] 0007_v380_rename_more_settings
[X] 0008_subscriptions
[X] 0009_rename_proot_settings
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
main
[X] 0001_initial
[X] 0002_squashed_v300_release (18 squashed migrations)
[X] 0003_squashed_v300_v303_updates (9 squashed migrations)
[X] 0004_squashed_v310_release (6 squashed migrations)
[X] 0005_squashed_v310_v313_updates (3 squashed migrations)
[X] 0006_v320_release
[X] 0007_v320_data_migrations
[X] 0008_v320_drop_v1_credential_fields
[X] 0009_v322_add_setting_field_for_activity_stream
[X] 0010_v322_add_ovirt4_tower_inventory
[X] 0011_v322_encrypt_survey_passwords
[X] 0012_v322_update_cred_types
[X] 0013_v330_multi_credential
[X] 0014_v330_saved_launchtime_configs
[X] 0015_v330_blank_start_args
[X] 0016_v330_non_blank_workflow
[X] 0017_v330_move_deprecated_stdout
[X] 0018_v330_add_additional_stdout_events
[X] 0019_v330_custom_virtualenv
[X] 0020_v330_instancegroup_policies
[X] 0021_v330_declare_new_rbac_roles
[X] 0022_v330_create_new_rbac_roles
[X] 0023_v330_inventory_multicred
[X] 0024_v330_create_user_session_membership
[X] 0025_v330_add_oauth_activity_stream_registrar
[X] 0026_v330_delete_authtoken
[X] 0027_v330_emitted_events
[X] 0028_v330_add_tower_verify
[X] 0030_v330_modify_application
[X] 0031_v330_encrypt_oauth2_secret
[X] 0032_v330_polymorphic_delete
[X] 0033_v330_oauth_help_text
[X] 0034_v330_delete_user_role
[X] 0035_v330_more_oauth2_help_text
[X] 0036_v330_credtype_remove_become_methods
[X] 0037_v330_remove_legacy_fact_cleanup
[X] 0038_v330_add_deleted_activitystream_actor
[X] 0039_v330_custom_venv_help_text
[X] 0040_v330_unifiedjob_controller_node
[X] 0041_v330_update_oauth_refreshtoken
[X] 0042_v330_org_member_role_deparent
[X] 0043_v330_oauth2accesstoken_modified
[X] 0044_v330_add_inventory_update_inventory
[X] 0045_v330_instance_managed_by_policy
[X] 0046_v330_remove_client_credentials_grant
[X] 0047_v330_activitystream_instance
[X] 0048_v330_django_created_modified_by_model_name
[X] 0049_v330_validate_instance_capacity_adjustment
[X] 0050_v340_drop_celery_tables
[X] 0051_v340_job_slicing
[X] 0052_v340_remove_project_scm_delete_on_next_update
[X] 0053_v340_workflow_inventory
[X] 0054_v340_workflow_convergence
[X] 0055_v340_add_grafana_notification
[X] 0056_v350_custom_venv_history
[X] 0057_v350_remove_become_method_type
[X] 0058_v350_remove_limit_limit
[X] 0059_v350_remove_adhoc_limit
[X] 0060_v350_update_schedule_uniqueness_constraint
[X] 0061_v350_track_native_credentialtype_source
[X] 0062_v350_new_playbook_stats
[X] 0063_v350_org_host_limits
[X] 0064_v350_analytics_state
[X] 0065_v350_index_job_status
[X] 0066_v350_inventorysource_custom_virtualenv
[X] 0067_v350_credential_plugins
[X] 0068_v350_index_event_created
[X] 0069_v350_generate_unique_install_uuid
[X] 0070_v350_gce_instance_id
[X] 0071_v350_remove_system_tracking
[X] 0072_v350_deprecate_fields
[X] 0073_v360_create_instance_group_m2m
[X] 0074_v360_migrate_instance_group_relations
[X] 0075_v360_remove_old_instance_group_relations
[X] 0076_v360_add_new_instance_group_relations
[X] 0077_v360_add_default_orderings
[X] 0078_v360_clear_sessions_tokens_jt
[X] 0079_v360_rm_implicit_oauth2_apps
[X] 0080_v360_replace_job_origin
[X] 0081_v360_notify_on_start
[X] 0082_v360_webhook_http_method
[X] 0083_v360_job_branch_override
[X] 0084_v360_token_description
[X] 0085_v360_add_notificationtemplate_messages
[X] 0086_v360_workflow_approval
[X] 0087_v360_update_credential_injector_help_text
[X] 0088_v360_dashboard_optimizations
[X] 0089_v360_new_job_event_types
[X] 0090_v360_WFJT_prompts
[X] 0091_v360_approval_node_notifications
[X] 0092_v360_webhook_mixin
[X] 0093_v360_personal_access_tokens
[X] 0094_v360_webhook_mixin2
[X] 0095_v360_increase_instance_version_length
[X] 0096_v360_container_groups
[X] 0097_v360_workflowapproval_approved_or_denied_by
[X] 0098_v360_rename_cyberark_aim_credential_type
[X] 0099_v361_license_cleanup
[X] 0100_v370_projectupdate_job_tags
[X] 0101_v370_generate_new_uuids_for_iso_nodes
[X] 0102_v370_unifiedjob_canceled
[X] 0103_v370_remove_computed_fields
[X] 0104_v370_cleanup_old_scan_jts
[X] 0105_v370_remove_jobevent_parent_and_hosts
[X] 0106_v370_remove_inventory_groups_with_active_failures
[X] 0107_v370_workflow_convergence_api_toggle
[X] 0108_v370_unifiedjob_dependencies_processed
[X] 0109_v370_job_template_organization_field
[X] 0110_v370_instance_ip_address
[X] 0111_v370_delete_channelgroup
[X] 0112_v370_workflow_node_identifier
[X] 0113_v370_event_bigint
[X] 0114_v370_remove_deprecated_manual_inventory_sources
[X] 0115_v370_schedule_set_null
[X] 0116_v400_remove_hipchat_notifications
[X] 0117_v400_remove_cloudforms_inventory
[X] 0118_add_remote_archive_scm_type
[X] 0119_inventory_plugins
[X] 0120_galaxy_credentials
[X] 0121_delete_toweranalyticsstate
[X] 0122_really_remove_cloudforms_inventory
[X] 0123_drop_hg_support
[X] 0124_execution_environments
[X] 0125_more_ee_modeling_changes
[X] 0126_executionenvironment_container_options
[X] 0127_reset_pod_spec_override
[X] 0128_organiaztion_read_roles_ee_admin
[X] 0129_unifiedjob_installed_collections
[X] 0130_ee_polymorphic_set_null
[X] 0131_undo_org_polymorphic_ee
[X] 0132_instancegroup_is_container_group
[X] 0133_centrify_vault_credtype
[X] 0134_unifiedjob_ansible_version
[X] 0135_schedule_sort_fallback_to_id
[X] 0136_scm_track_submodules
[X] 0137_custom_inventory_scripts_removal_data
[X] 0138_custom_inventory_scripts_removal
[X] 0139_isolated_removal
[X] 0140_rename
[X] 0141_remove_isolated_instances
[X] 0142_update_ee_image_field_description
[X] 0143_hostmetric
[X] 0144_event_partitions
[X] 0145_deregister_managed_ee_objs
[X] 0146_add_insights_inventory
[X] 0147_validate_ee_image_field
[X] 0148_unifiedjob_receptor_unit_id
[X] 0149_remove_inventory_insights_credential
[X] 0150_rename_inv_sources_inv_updates
[X] 0151_rename_managed_by_tower
[X] 0152_instance_node_type
[X] 0153_instance_last_seen
[X] 0154_set_default_uuid
[X] 0155_improved_health_check
[X] 0156_capture_mesh_topology
[X] 0157_inventory_labels
[X] 0158_make_instance_cpu_decimal
[X] 0159_deprecate_inventory_source_UoPU_field
[X] 0160_alter_schedule_rrule
[X] 0161_unifiedjob_host_status_counts
[X] 0162_alter_unifiedjob_dependent_jobs
[X] 0163_convert_job_tags_to_textfield
[X] 0164_remove_inventorysource_update_on_project_update
[X] 0165_task_manager_refactor
[X] 0166_alter_jobevent_host
[X] 0167_project_signature_validation_credential
[X] 0168_inventoryupdate_scm_revision
[X] 0169_jt_prompt_everything_on_launch
[X] 0170_node_and_link_state
[X] 0171_add_health_check_started
[X] 0172_prevent_instance_fallback
[X] 0173_instancegroup_max_limits
[X] 0174_ensure_org_ee_admin_roles
[X] 0175_workflowjob_is_bulk_job
[X] 0176_inventorysource_scm_branch
[X] 0177_instance_group_role_addition
[X] 0178_instance_group_admin_migration
[X] 0179_change_cyberark_plugin_names
[X] 0180_add_hostmetric_fields
[X] 0181_hostmetricsummarymonthly
[X] 0182_constructed_inventory
[X] 0183_pre_django_upgrade
[ ] 0184_django_indexes
oauth2_provider
[X] 0001_initial
[X] 0002_auto_20190406_1805
[X] 0003_auto_20201211_1314
[X] 0004_auto_20200902_2022
[X] 0005_auto_20211222_2352
sessions
[X] 0001_initial
sites
[X] 0001_initial
[X] 0002_alter_domain_unique
social_django
[X] 0001_initial (2 squashed migrations)
[X] 0002_add_related_name (2 squashed migrations)
[X] 0003_alter_email_max_length (2 squashed migrations)
[X] 0004_auto_20160423_0400 (2 squashed migrations)
[X] 0005_auto_20160727_2333 (1 squashed migrations)
[X] 0006_partial
[X] 0007_code_timestamp
[X] 0008_partial_timestamp
[X] 0009_auto_20191118_0520
[X] 0010_uid_db_index
sso
[X] 0001_initial
[X] 0002_expand_provider_options
[X] 0003_convert_saml_string_to_list
taggit
[X] 0001_initial
[X] 0002_auto_20150616_2121
[X] 0003_taggeditem_add_unique_index
[X] 0004_alter_taggeditem_content_type_alter_taggeditem_tag
[X] 0005_auto_20220424_2025

Only 0184_django_indexes is failing

bash-5.1$ awx-manage migrate main
Operations to perform:
Apply all migrations: main
Running migrations:
Applying main.0184_django_indexes...Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py", line 200, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
res = handle_func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
post_migrate_state = executor.migrate(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
state = self._migrate_all_forwards(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
state = self.apply_migration(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
state = migration.apply(state, schema_editor)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
operation.database_forwards(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py", line 977, in database_forwards
raise ValueError(
ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).

Any ideas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants