Parallelize optimal_snr jobs

[GarethCabournDavies]

Given certain approximants and amount of injections, optimal_snr jobs can take a long time causing jobs to be evicted/restarted in the workflow

This splits optimal_snr jobs in the same way as the findtrigs jobs, and recombines using a new executable, pycbc_optimal_snr_merge

This will break current configs due to changing the way optimal_snr is used and optimal_snr_merge config requirements.

Newly required config tags:

• in [executables], optimal_snr_merge = ${which:pycbc_optimal_snr_merge}
• in [workflow-optimal-snr]: parallelization-factor = 5
• add [optimal_snr_merge]

Tested in multiifo workflow, but as the 2-ifo workflow calls it in the same way, this should work there as well

[titodalcanton]

Instead of adding the new script pycbc_optimal_snr_merge, would it be possible to just use ligolw_add?

[GarethCabournDavies]

Instead of adding the new script pycbc_optimal_snr_merge, would it be possible to just use ligolw_add?

I'm not sure - I hadn't seen that functionality before
I'm not sure how it could work within a workflow, as surely it would still need an executable to wrap it in pegasus?

[titodalcanton]

I am thinking about this:

[executables]
optimal_snr_merge = ligolw_add

I do not see an obvious reason why a specific script would be needed instead, as it is just merging tables.

[ahnitz]

@GarethDaviesGW ligolw_add is an executable. I think @titodalcanton is saying why invent a new executable instead of using an existing one that likely already can do this. It takes options slightly differently, but otherwise, I think it should work.

[GarethCabournDavies]

thanks - I hadnt seen the executable - I'll put that in rather than the optimal_snr_merge code

[GarethCabournDavies]

I've updated to use the ligolw_add executable as well and removed the optimal_snr_merge

[GarethCabournDavies]

The last codeclimate and codebeat issues are ones which we have generally ignored before - the overwriting of create_node in particular looks to be something done a lot elsewhere

[titodalcanton]

This looks good to me, but I am a bit too rusty on the workflow code to feel comfortable approving that change. Maybe best if @spxiwh or @ahnitz give the final approval.

[ahnitz]

Assuming you tested that the workflow builds and the dax looks sane, this seems fine.

[GarethCabournDavies]

tested with and without the parallelize option in config for workflow, and using the updated optimal_snr exe