make hltFindDuplicates work again

[missirol]

The utility hltFindDuplicates seems to be broken in recent CMSSW releases (it has not been touched in the last few years, I don't know when it stopped working). This PR aims to introduce minimal updates to make it work again.

While testing, I also noticed that the text files produced as outputs are not fully reproducible, because the labels assigned to the groups of duplicate modules (hltGroup1, hltGroup2, etc) are not assigned deterministically. In order to avoid this and make outputs reproducible, the group label hltGroup[0-9]+ is replaced with a string based on the content of the group (the string contains the type of the plugin followed by the label of the first module of the group in alphabetic order).

A unit test is also added as an attempt to make sure that this utility does not break again in the future. In order to make the unit test fast (a few seconds), the test currently uses only the PRef menu (the simplest menu in the release with some duplicate modules).

Merely technical. No changes expected.

FYI: @fwyzard (in case Andrea is the one who wrote this script, which is what I would guess based on CMSHLT-2099)

PR validation:

Manual checks, and the new unit test passes.

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

N/A

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43644/38304

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @Martin-Grunewald, @mmusich can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @silviodonato this is something you requested to watch as well.
@sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

[mmusich]

@cmsbuild, please test

• I guess we're still not swamped by test jobs ;)

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-597024/36689/summary.html
COMMIT: d591c3b
CMSSW: CMSSW_14_0_X_2024-01-02-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43644/36689/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 106 lines from the logs
• Reco comparison results: 5 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3247277
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247255
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[mmusich]

+hlt

• differences in the PR log changes appear to be spurious

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[missirol]

I noticed that, using the initial implementation in this PR, modules with nested PSets were not treated correctly. The version in the latest commit should fix this. While fixing this, I changed the unit test to use a dedicated input file (rather than one of the HLT menus), requiring not just that the script runs, but that it also returns the expected results. Other improvements are also included (treatment of the label assigned to groups of duplicate modules, printouts for debugging that can be shown using --verbosity-level). Coincidentally, this version seems faster, as running it on the GRun menu now took 50 seconds compared to the ~250+ seconds reported in previous comments (and the outputs of the latest version are the expected ones). This is intended to be the final version.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43644/38345

• This PR adds an extra 20KB to repository

[cmsbuild]

Pull request #43644 was updated. @Martin-Grunewald, @cmsbuild, @mmusich can you please check and sign again.

[mmusich]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-597024/36730/summary.html
COMMIT: 04e3b62
CMSSW: CMSSW_14_0_X_2024-01-07-0000/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43644/36730/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 113 lines to the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3247277
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247252
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[mmusich]

+hlt

• re-sign after the new push

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1