Send also workflows as part of unified jobs and send all changes to jobs #6176

Ladas · 2020-03-05T11:49:28Z

SUMMARY

Workflows do not have a record in main_job, therefore the JOIN
was ignoring those. We need to do LEFT JOIN to include also
workflows.

It can take several hours for a job to go from pending to successful/failed state and we need to also send the job with a changed state, otherwise the analytics will be incorrect.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

Analytics

FIXES

#6200

Partially fixing: #6201 (sending workflows as part of unified jobs)

Works with current analytics processor?

Currently the processor will break, since it requires Org name and ID as NOT NULL. I will limit this constraints for only template jobs. (will insert PR here)

TODO

figure out if we want to add organization relation from workflow_template (for some reason workflow_job misses it)
adjust analytics processor to deal with missing org for workflows https://github.com/RedHatInsights/tower-analytics-backend/pull/262/commits/793721f624e5fcd07c2089a158f76935e4c20ad7

Ladas · 2020-03-05T11:49:59Z

cc @rooftopcellist @wenottingham @benthomasson

softwarefactory-project-zuul · 2020-03-05T12:04:24Z

Build succeeded.

awx-api-lint : SUCCESS in 5m 15s
awx-api : SUCCESS in 12m 09s
awx-ui : SUCCESS in 7m 48s
awx-ui-next : SUCCESS in 11m 56s
awx-swagger : SUCCESS in 12m 28s
awx-detect-schema-change : SUCCESS in 14m 19s (non-voting)
awx-ansible-modules : SUCCESS in 6m 26s

softwarefactory-project-zuul · 2020-03-05T14:15:50Z

Build succeeded.

awx-api-lint : SUCCESS in 6m 35s
awx-api : SUCCESS in 13m 34s
awx-ui : SUCCESS in 6m 49s
awx-ui-next : SUCCESS in 11m 58s
awx-swagger : SUCCESS in 10m 40s
awx-detect-schema-change : SUCCESS in 10m 55s (non-voting)
awx-ansible-modules : SUCCESS in 4m 56s

softwarefactory-project-zuul · 2020-03-06T11:01:28Z

Build succeeded.

awx-api-lint : SUCCESS in 5m 26s
awx-api : SUCCESS in 16m 03s
awx-ui : SUCCESS in 7m 16s
awx-ui-next : SUCCESS in 16m 31s
awx-swagger : SUCCESS in 11m 18s
awx-detect-schema-change : SUCCESS in 11m 27s (non-voting)
awx-ansible-modules : SUCCESS in 5m 30s

ryanpetrello · 2020-03-06T15:07:28Z

We may want to hold off on this until @AlanCoding's change here lands: #5726

ryanpetrello · 2020-03-06T15:08:04Z

awx/main/analytics/collectors.py

-                                 JOIN main_organization ON main_organization.id = main_project.organization_id
-                                 WHERE main_unifiedjob.created > {} 
+                                 LEFT JOIN main_project ON main_project.unifiedjobtemplate_ptr_id = main_job.project_id
+                                 LEFT JOIN main_organization ON main_organization.id = main_project.organization_id


This join is going to change after @AlanCoding's PR merges.

cool, I am fine fixing this another way. Just keep in mind this was basically filtering out everything except jobs. But we want other stuff like workflowapproval, worflowjob, projectupdate etc. , so we'll need the left join, since not everything has org associated

Also @AlanCoding is your PR going to be backported? We need this backported to all versions sending data to analytics, so we get the workflow and other data.

no, we do not plan on backporting. that would be very complicated

Hm, ok, so can we merge this first? Or it doesn't matter for easier backport?

We should wait for @AlanCoding's organization schema to land in devel before changing this in devel.

If you want to open a PR like this in a backport branch, we'll have to carry a different implementation for that database schema anyways.

sounds good, I'll backport and wait for @AlanCoding here

AlanCoding · 2020-03-06T16:51:55Z

It also seems like we are not able to get a link to organizations
from workflows? When looking at:
<tower_url>#/organizations?organization_search=page_size:20;order_by:name

I don't follow that URL. If you were searching from /api/v2/unified_jobs/ or /api/v2/workflow_jobs/, or the corresponding template endpoints, and using organization, then yes, I see why that wouldn't work, and that will change with the other PR I have open because it moves the field from the workflow tables to the unified tables

Ladas · 2020-03-06T17:45:39Z

@AlanCoding the tower url was just supposed to show that on the organization page, we don't show any links to workflows. And as I was scanning the db, we only store organization relation in workflow template, not workflow_job.

ryanpetrello · 2020-03-17T12:24:23Z

@Ladas we've merged @AlanCoding's work - this is probably ready for a rebase and another look now.

AlanCoding · 2020-03-17T14:13:36Z

why was the created filter changed to modified? I think those were both indexed the same, but this is confusing. What do you think that modified represents? And how does that differ from finished? I'm sorry, I'm still groking what data it is that you are trying to obtain.

Ladas · 2020-03-17T18:07:32Z

@AlanCoding the use of modified is described in the issue (I suppose finished can be used also). If we use created, then finished/modified can be hours later and if we'd already sent the payload, we wouldn't send the job again. We need to send the job again if it's finished (or change occurred) so the analytics contains the terminal state.

AlanCoding · 2020-03-17T19:47:59Z

I may not have access to the issue. But finished is indexed

awx/awx/main/models/unified_jobs.py

Lines 631 to 637 in 4a5e596

    
           finished = models.DateTimeField( 
        
               null=True, 
        
               default=None, 
        
               editable=False, 
        
               help_text=_("The date and time the job finished execution."), 
        
               db_index=True, 
        
           )

can't say the same for modified. So it seems like that'll work as a drop-in replacement here.

And the organization field should be on the unified jobs table now, but I think there are more problems you're trying to solve here which I'm not fully aware of.

Ladas · 2020-03-18T07:54:24Z

@AlanCoding I see. Although finished doesn't have NOT NULL constraint. And from what I see, it is being set only when really finished. So it will not catch jobs stuck in pending etc. (so things we are trying to detect and alert upon)

So it looks like we have 2 paths:

using modified

GOOD: we'll catch all changes
BAD: we'll always have to do a sequential scan of unified jobs

using both created and finished

GOOD: we'll use indexes
BAD: while we should catch all jobs, the state it is stuck on doesn't have to be correct

Looks like 2) is good enough, we just need take in account that when finished is NULL, we might not have the right status. (and e.g. change the alert for too many pending jobs to too many unfinished jobs)

For the orgs, in analytics I expect them to be there for job type. So it's ok that the rest of unified jobs don't have it. But we should send all unified jobs for analytics except what is explicitly filtered out in where condition.

softwarefactory-project-zuul · 2020-03-18T09:38:02Z

Build succeeded.

awx-api-lint : SUCCESS in 4m 22s
awx-api : SUCCESS in 9m 33s
awx-ui : SUCCESS in 6m 38s
awx-ui-next : SUCCESS in 11m 43s
awx-swagger : SUCCESS in 11m 44s
awx-detect-schema-change : SUCCESS in 13m 44s (non-voting)
awx-ansible-modules : SUCCESS in 5m 52s

Ladas · 2020-03-18T09:48:44Z

@AlanCoding ok I changed the condition to (main_unifiedjob.created > {0} OR main_unifiedjob.finished > {0}) and I've added the left join, so the unified jobs without org are not filtered out.

Can you review also the backports please? https://github.com/ansible/tower/pull/4180 and https://github.com/ansible/tower/pull/4179

Btw. we should probably add index to modified column, so we can get the 100% correct data to analytics. (even if that can't be backported)

wenottingham · 2020-04-07T18:05:37Z

What still needs done here? Just a review?

cc @AlanCoding @rooftopcellist

ryanpetrello · 2020-04-07T18:07:30Z

@wenottingham likely that and verification from somebody.

chrismeyersfsu · 2020-04-08T20:49:58Z

@Ladas I submitted a PR against your PR. https://github.com/Ladas/awx/pull/1/files incorporate that test and I'm good with merging this PR 👍

Ladas · 2020-04-09T08:50:20Z

@chrismeyersfsu thank you

btw. this PR's condition works for backports nicely. But as extension for the next major AWX release, the best would be if we could add index on modified column and use that instead of created+finished. (then we could track exact state of a job, in a case it gets stuck in a non terminal state)

softwarefactory-project-zuul · 2020-04-09T09:06:07Z

Build failed.

awx-api-lint : SUCCESS in 18m 10s
awx-api : FAILURE in 16m 53s
awx-ui : SUCCESS in 20m 05s
awx-ui-next : SUCCESS in 21m 56s
awx-swagger : FAILURE in 18m 01s
awx-detect-schema-change : FAILURE in 16m 54s (non-voting)
awx-ansible-modules : FAILURE in 17m 04s

ryanpetrello · 2020-04-09T13:16:00Z

@Ladas the CI failures here suggest to me that this branch needs to be rebased against devel.

It can take several hours for a job to go from pending to successful/failed state and we need to also send the job with a changed state, otherwise the analytics will be incorrect.

Workflows do not have a record in main_job, therefore the JOIN was ignoring those. We need to do LEFT JOIN to include also workflows. It also seems like we are not able to get a link to organizations from workflows? When looking at: <tower_url>#/organizations?organization_search=page_size:20;order_by:name We don't seem to list a relation to workflows. Is it possible to get it from somewhere?

Use created and finished, which are indexed, to try to fetch all states of jobs. If job is not finished, we might not get the right terminal status, but that should be ok for now.

* unified_jobs output should include derived jobs i.e. project update, inventory update, job * This PR adds a test to ensure that.

Ladas · 2020-04-09T13:22:12Z

@ryanpetrello thanks for the hint, rebased

softwarefactory-project-zuul · 2020-04-09T13:36:38Z

Build succeeded.

awx-api-lint : SUCCESS in 5m 59s
awx-api : SUCCESS in 9m 15s
awx-ui : SUCCESS in 7m 52s
awx-ui-next : SUCCESS in 9m 17s
awx-swagger : SUCCESS in 14m 46s
awx-detect-schema-change : SUCCESS in 12m 52s (non-voting)
awx-ansible-modules : SUCCESS in 9m 13s

wenottingham · 2020-04-13T17:17:19Z

@ryanpetrello ok for mergeit with @chrismeyersfsu's approval?

softwarefactory-project-zuul · 2020-04-13T18:41:35Z

Build succeeded (gate pipeline).

awx-api-lint : SUCCESS in 6m 41s
awx-api : SUCCESS in 20m 52s
awx-ui : SUCCESS in 13m 37s
awx-ui-next : SUCCESS in 17m 32s
awx-swagger : SUCCESS in 15m 39s
awx-detect-schema-change : SUCCESS in 15m 26s (non-voting)
awx-ansible-modules : SUCCESS in 7m 04s
awx-push-new-schema : FAILURE in 12m 03s (non-voting)

[backport 4.2] removed hostname check when editing hostname on existing host (ansible#13057)

awxbot added the type:bug label Mar 5, 2020

Ladas closed this Mar 5, 2020

Ladas force-pushed the send_also_workflows_as_part_of_unified_jobs branch from 2346c1f to 4fcd2c5 Compare March 5, 2020 13:48

Ladas reopened this Mar 5, 2020

Ladas changed the title ~~Send also workflows as part of unified jobs~~ [WIP] Send also workflows as part of unified jobs Mar 6, 2020

Ladas force-pushed the send_also_workflows_as_part_of_unified_jobs branch from 748f6cd to 4a5e596 Compare March 6, 2020 10:44

Ladas mentioned this pull request Mar 6, 2020

Send also workflows as part of unified jobs into analytics #6201

Closed

ryanpetrello reviewed Mar 6, 2020

View reviewed changes

AlanCoding added the component:analytics label Mar 6, 2020

Ladas mentioned this pull request Mar 18, 2020

Use modified to check if job should be sent to analytics #6179

Closed

Ladas changed the title ~~[WIP] Send also workflows as part of unified jobs~~ Send also workflows as part of unified jobs and send all changes to jobs Mar 18, 2020

Ladas force-pushed the send_also_workflows_as_part_of_unified_jobs branch 2 times, most recently from 7223908 to 274fead Compare March 18, 2020 09:23

wenottingham mentioned this pull request Apr 7, 2020

Send job and template nodes to analytics #6204

Merged

blomquisg assigned chrismeyersfsu Apr 8, 2020

chrismeyersfsu approved these changes Apr 8, 2020

View reviewed changes

Ladas and others added 4 commits April 9, 2020 15:20

Use modified to check if job should be sent to analytics

b722711

It can take several hours for a job to go from pending to successful/failed state and we need to also send the job with a changed state, otherwise the analytics will be incorrect.

Use indexed timestamps

6a86af5

Use created and finished, which are indexed, to try to fetch all states of jobs. If job is not finished, we might not get the right terminal status, but that should be ok for now.

test analytics table output

1f9f869

* unified_jobs output should include derived jobs i.e. project update, inventory update, job * This PR adds a test to ensure that.

Ladas force-pushed the send_also_workflows_as_part_of_unified_jobs branch from 405ecd3 to 1f9f869 Compare April 9, 2020 13:21

ryanpetrello added the mergeit label Apr 13, 2020

softwarefactory-project-zuul bot merged commit c414fd6 into ansible:devel Apr 13, 2020

AlanCoding pushed a commit to AlanCoding/awx that referenced this pull request Jan 4, 2023

Merge pull request ansible#6176 from sarabrajsingh/release_4.2

ad81f5c

[backport 4.2] removed hostname check when editing hostname on existing host (ansible#13057)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send also workflows as part of unified jobs and send all changes to jobs #6176

Send also workflows as part of unified jobs and send all changes to jobs #6176

Ladas commented Mar 5, 2020 •

edited

Loading

Ladas commented Mar 5, 2020

softwarefactory-project-zuul bot commented Mar 5, 2020

softwarefactory-project-zuul bot commented Mar 5, 2020

softwarefactory-project-zuul bot commented Mar 6, 2020

ryanpetrello commented Mar 6, 2020

ryanpetrello Mar 6, 2020

Ladas Mar 6, 2020

AlanCoding Mar 6, 2020

Ladas Mar 6, 2020

ryanpetrello Mar 6, 2020

Ladas Mar 9, 2020

AlanCoding commented Mar 6, 2020

Ladas commented Mar 6, 2020

ryanpetrello commented Mar 17, 2020

AlanCoding commented Mar 17, 2020

Ladas commented Mar 17, 2020

AlanCoding commented Mar 17, 2020

Ladas commented Mar 18, 2020

softwarefactory-project-zuul bot commented Mar 18, 2020

Ladas commented Mar 18, 2020 •

edited

Loading

wenottingham commented Apr 7, 2020

ryanpetrello commented Apr 7, 2020

chrismeyersfsu commented Apr 8, 2020 •

edited

Loading

Ladas commented Apr 9, 2020

softwarefactory-project-zuul bot commented Apr 9, 2020

ryanpetrello commented Apr 9, 2020

Ladas commented Apr 9, 2020

softwarefactory-project-zuul bot commented Apr 9, 2020

wenottingham commented Apr 13, 2020

softwarefactory-project-zuul bot commented Apr 13, 2020

Send also workflows as part of unified jobs and send all changes to jobs #6176

Send also workflows as part of unified jobs and send all changes to jobs #6176

Conversation

Ladas commented Mar 5, 2020 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

FIXES

Works with current analytics processor?

TODO

Ladas commented Mar 5, 2020

softwarefactory-project-zuul bot commented Mar 5, 2020

softwarefactory-project-zuul bot commented Mar 5, 2020

softwarefactory-project-zuul bot commented Mar 6, 2020

ryanpetrello commented Mar 6, 2020

ryanpetrello Mar 6, 2020

Choose a reason for hiding this comment

Ladas Mar 6, 2020

Choose a reason for hiding this comment

AlanCoding Mar 6, 2020

Choose a reason for hiding this comment

Ladas Mar 6, 2020

Choose a reason for hiding this comment

ryanpetrello Mar 6, 2020

Choose a reason for hiding this comment

Ladas Mar 9, 2020

Choose a reason for hiding this comment

AlanCoding commented Mar 6, 2020

Ladas commented Mar 6, 2020

ryanpetrello commented Mar 17, 2020

AlanCoding commented Mar 17, 2020

Ladas commented Mar 17, 2020

AlanCoding commented Mar 17, 2020

Ladas commented Mar 18, 2020

softwarefactory-project-zuul bot commented Mar 18, 2020

Ladas commented Mar 18, 2020 • edited Loading

wenottingham commented Apr 7, 2020

ryanpetrello commented Apr 7, 2020

chrismeyersfsu commented Apr 8, 2020 • edited Loading

Ladas commented Apr 9, 2020

softwarefactory-project-zuul bot commented Apr 9, 2020

ryanpetrello commented Apr 9, 2020

Ladas commented Apr 9, 2020

softwarefactory-project-zuul bot commented Apr 9, 2020

wenottingham commented Apr 13, 2020

softwarefactory-project-zuul bot commented Apr 13, 2020

Ladas commented Mar 5, 2020 •

edited

Loading

Ladas commented Mar 18, 2020 •

edited

Loading

chrismeyersfsu commented Apr 8, 2020 •

edited

Loading