Performance improvements to ETLv2 action that loads queues for jobs realm

[eiffel777]

The ETLv2 action for populating the modw.queue dimension table selects all the records in mod_hpcdb.hpcdb_jobs to get all queues that have been ingested. It seems unnecessary to try and re-ingest all the queues every time the ingestor runs. It also takes a long time, around 300 seconds. Instead, I think the action should get the queues from jobs listed in the mod_hpcdb.hpcdb_jobs_to_ingest table. This approach is much quicker, around 1 second or less.

Tests performed

Tested in docker and on metrics-staging

Checklist:

• The pull request description is suitable for a Changelog entry
• The milestone is set correctly on the pull request
• The appropriate labels have been added to the pull request

[jpwhite4]

Just want to confirm that the queue table is definitly not truncated by this action. I couldn't see where the 'truncate' flag was set.

[eiffel777]

@jpwhite4 Yes, this action does not truncate the queue table. The default for truncate_destination is false as set in classes/ETL/aOptions.php and truncate_destination is not changed for this action or pipeline. Also, when running this action on metrics-staging I do not see the queue table being truncated.