add option to restrict single-Path jobs of hltIntegrationTests to a subset of triggers

[missirol]

PR description:

This PRs adds a functionality to the utility hltIntegrationTests.

Currently, given a HLT menu with N Paths (excluding HLTriggerFinalPath), hltIntegrationTests runs N+1 jobs: one job running the full HLT menu in question, and N jobs each running only one of the N Paths of the menu standalone. This PR adds the option to run the 1-Path-standalone jobs for only a subset M of the Paths in the menu (M < N). This can be useful when testing an update which is known to modify only M Paths, where one normally wants to run (a) the full HLT menu, and (b) standalone jobs only for the M Paths in question, given that the other N-M Paths are unchanged.

This is implemented by adding the command-line argument --paths to hltIntegrationTests. In analogy to the --paths argument of hltGetConfiguration, one can specify patterns to select certain Paths using a comma-separated list. One can also specify Paths to be ignored/dropped by prepending the character - to a given pattern (e.g. --paths "*HLT*,-HLTriggerFinalPath" to ignore the Path HLTriggerFinalPath). The script hltListPaths is updated accordingly (technically, changing its CLI in a non-backward-compatible way). More details are provided in the help messages of the scripts updated in this PR.

Note.

• hltIntegrationTests [MENU] -x "--paths A,B,C" (before and after this PR) corresponds to running 4 jobs: one with a menu containing only 3 Paths (A, B, C), plus 1-Path jobs for each of those 3 Paths (A, B, C).
• hltIntegrationTests [MENU] --paths A,B,C (possible only with this PR) also corresponds to running 4 jobs: one with a 'full' menu containing all its Paths (possibly many more than A, B and C), plus 1-Path jobs only for the 3 Paths specified via --paths (i.e. A, B and C).

Minor code improvements are included as well.

Merely technical. No changes expected.

PR validation:

Manual tests, e.g.

hltIntegrationTests \
/dev/CMSSW_14_0_0/GRun \
-n 100 --mc -d tmp \
--paths "*PFScouting*" \
-x "--globaltag 140X_mcRun3_2024_realistic_v1" \
-i /store/relval/CMSSW_14_0_0_pre3/RelValQCD_Pt_1800_2400_14/GEN-SIM-DIGI-RAW/PU_140X_mcRun3_2024_realistic_v1_STD_2024_PU-v1/2580000/0bf93d4d-de46-4781-bfe4-78eda3f202f6.root

----------------------------
Starting hltIntegrationTests
----------------------------
Creating HLT menu from ConfDB configuration:
> hltGetConfiguration /dev/CMSSW_14_0_0/GRun --process TEST20240218193609 --max-events 100 --no-prescale --no-output --mc --input /store/relval/CMSSW_14_0_0_pre3/RelValQCD_Pt_1800_2400_14/GEN-SIM-DIGI-RAW/PU_140X_mcRun3_2024_realistic_v1_STD_2024_PU-v1/2580000/0bf93d4d-de46-4781-bfe4-78eda3f202f6.root --globaltag 140X_mcRun3_2024_realistic_v1
Preparing list of trigger Paths to be tested standalone (paths.txt)
Will run full menu and 7 triggers standalone over 100 events, with 4 jobs in parallel
Preparing single-trigger configurations
Running...
	full menu dump
	DST_Run3_DoubleMuon_PFScoutingPixelTracking_v2
	DST_Run3_DoubleEG_PFScoutingPixelTracking_v2
	DST_Run3_JetHT_PFScoutingPixelTracking_v22
	DST_Run3_EG30_PFScoutingPixelTracking_v22
	DST_Run3_DoubleMu3_PFScoutingPixelTracking_v22
	DST_Run3_EG16_EG12_PFScoutingPixelTracking_v22
	MC_Run3_PFScoutingPixelTracking_v22
Comparing the results of running each path by itself with those from the full menu
WARNING: path DST_Run3_DoubleEG_PFScoutingPixelTracking_v2 did not accept any events
WARNING: path DST_Run3_EG16_EG12_PFScoutingPixelTracking_v22 did not accept any events
--------------------------
hltIntegrationTests PASSED
--------------------------
exit status: 0

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

To be backported to 14_0_X for HLT-menu development in 2024.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44005/38932

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @missirol for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @mmusich, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@silviodonato, @Martin-Grunewald this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[Martin-Grunewald]

please test

[Martin-Grunewald]

Ah that's useful - can you please make a backport to 14_0?

[missirol]

Ah that's useful - can you please make a backport to 14_0?

Sure, it's in #44007 (same branch).

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bce158/37546/summary.html
COMMIT: 043b424
CMSSW: CMSSW_14_1_X_2024-02-18-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44005/37546/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test test-das-selected-lumis had ERRORS

Comparison Summary

Summary:

• You potentially removed 3 lines from the logs
• Reco comparison results: 63 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3248569
• DQMHistoTests: Total failures: 1189
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247358
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[Martin-Grunewald]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bce158/37548/summary.html
COMMIT: 043b424
CMSSW: CMSSW_14_1_X_2024-02-19-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44005/37548/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 94 lines to the logs
• Reco comparison results: 65 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3248569
• DQMHistoTests: Total failures: 1189
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247358
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[Martin-Grunewald]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1