custom facts can break fact cache

[wenottingham]

ISSUE TYPE

• Bug Report

SUMMARY

User-forwarded traceback:

2020-02-13 19:10:29,258 ERROR    awx.main.tasks job 909 (successful) Final run hook errored.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 1411, in run
    self.final_run_hook(self.instance, status, private_data_dir, fact_modification_times, isolated_manager_instance=isolated_manager_instance)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 1864, in final_run_hook
    fact_modification_times,
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/models/jobs.py", line 821, in finish_job_fact_cache
    ansible_local_system_id = ansible_facts.get('ansible_local', {}).get('insights', {}).get('system_id', None)
AttributeError: 'str' object has no attribute 'get'

User stated they were returning custom windows facts.

Assumption - this needs bulletproofed against arbitrary data structures in these namespaces.

ENVIRONMENT

• AWX version: current

[chrismeyersfsu]

Steps to recreate (not sure if this is what the user did, but it's a way to recreate).

• put "hello world" in /etc/ansible/facts.d/insights.facts
• run a job template with facts on

[fvzwieten]

I made a local.ps1 file from this template:
'{
"role":"{{ role }}"
}'

When executed in powershell console this output's:
{
"role":"webserver"
}

It gave this in :

=> /var/log/tower/task_system.log <==
2020-02-13 19:10:29,258 ERROR awx.main.tasks job 909 (successful) Final run hook errored.
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 1411, in run
self.final_run_hook(self.instance, status, private_data_dir, fact_modification_times, isolated_manager_instance=isolated_manager_instance)
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/tasks.py", line 1864, in final_run_hook
fact_modification_times,
File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/models/jobs.py", line 821, in finish_job_fact_cache
ansible_local_system_id = ansible_facts.get('ansible_local', {}).get('insights', {}).get('system_id', None)
AttributeError: 'str' object has no attribute 'get'

I needed to repair DB after that. fact caching was broken.

[ryanpetrello]

put "hello world" in /etc/ansible/facts.d/insights.facts

Needs to be .fact, not .facts.

[kdelee]

put "hello world" in /etc/ansible/facts.d/insights.facts

Needs to be .fact, not .facts.

Tried this out with the host in question being localhost in the development environment. I am still running code before your fix and I'm not seeing the bug. Might need some help to reproduce

[ryanpetrello]

@kdelee have you turned on fact cache for a JT, and run something like gather_facts.yml?

[kdelee]

😲 ok did the fact cache step but failed on gather_facts.yml will try that

[kdelee]

❗ recreated traceback in log:

awx_1        | 2020-03-05 17:15:22,513 ERROR    awx.main.tasks job 120 (successful) Final run hook errored.
awx_1        | Traceback (most recent call last):
awx_1        |   File "/awx_devel/awx/main/tasks.py", line 1435, in run
awx_1        |     self.final_run_hook(self.instance, status, private_data_dir, fact_modification_times, isolated_manager_instance=isolated_manager_instance)
awx_1        |   File "/awx_devel/awx/main/tasks.py", line 1901, in final_run_hook
awx_1        |     fact_modification_times,
awx_1        |   File "/awx_devel/awx/main/models/jobs.py", line 832, in finish_job_fact_cache
awx_1        |     ansible_local_system_id = ansible_facts.get('ansible_local', {}).get('insights', {}).get('system_id', None)
awx_1        | AttributeError: 'str' object has no attribute 'get'

Then updated to latest devel that contains your bug fix, upon running gather_facts.yml on same host that has the bad facts, do not see any kind of trace back and facts are stored normally.

Calling this verified. Not planning to automate this test case unless you all think this is high priority, ping if you do.