Add HCAL tag HcalRespCorrs_2021_v3.0_mc to Run-3 MC GTs

[tvami]

PR description:

The HCAL group requested the inclusion of the tag HcalRespCorrs_2021_v3.0_mc in all Run-3 MC GTs in [*]. As described in the HN message the difference between the new GTs and the current ones is simply this single-IOV MC tag. For completeness here are the links as well:

2021 design MC GT

• https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_design_v1/120X_mcRun3_2021_design_v3

2021 realistic MC GT

• https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_realistic_v1/120X_mcRun3_2021_realistic_v4

2021 cosmics MC GT

• https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021cosmics_realistic_deco_v1/120X_mcRun3_2021cosmics_realistic_deco_v3

2021 heavy ion MC GT

• https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_realistic_HI_v1/120X_mcRun3_2021_realistic_HI_v3

2023 realistic MC GT

• https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2023_realistic_v1/120X_mcRun3_2023_realistic_v4

2024 realistic MC GT

• https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2024_realistic_v1/120X_mcRun3_2024_realistic_v4

More details in the presentation at the AlCaDB meeting [**].

[*] https://hypernews.cern.ch/HyperNews/CMS/get/calibrations/4444.html
[**] https://indico.cern.ch/event/1066868/#14-fix-of-hcal-response-correc

PR validation:

runTheMatrix.py -l limited,7.23,159.0,11634.0,12434.0,12834.0 --ibeos -j8

As described in the HN message:

• Plot showing ieta (HCAL tower index) distribution of new corrections:
  https://hypernews.cern.ch/HyperNews/CMS/get/AUX/2021/08/17/10:01:18-74638-HcalRespCorrs_2021_v3.0_mc.pdf

• Closure test results with the new corrections.
  https://hypernews.cern.ch/HyperNews/CMS/get/AUX/2021/08/17/10:01:18-86183-IsoTrackN105.pdf

if this PR is a backport please specify the original PR and why you need to backport that PR:

A backport is expected for 12_0_X GTs as well.

cc: @mseidel42 @abdoulline

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34922/24732

• This PR adds an extra 12KB to repository

[cmsbuild]

A new Pull Request was created by @tvami (Tamas Vami) for master.

It involves the following packages:

• Configuration/AlCa (alca)

@malbouis, @yuanchao, @cmsbuild, @tlampen, @pohsun, @francescobrivio, @tvami can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @mmusich, @fabiocos, @tocheng this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[tvami]

test parameters:

• workflows = 7.23,159.0,11634.0,12434.0,12834.0

[tvami]

@cmsbuild , please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd0909/17836/summary.html
COMMIT: 8ae5e4f
CMSSW: CMSSW_12_1_X_2021-08-17-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34922/17836/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-cd0909/7.23_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21

Summary:

• No significant changes to the logs found
• Reco comparison results: 4160 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3000352
• DQMHistoTests: Total failures: 5706
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2994623
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 38 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: found differences in 3 / 38 workflows

[abdoulline]

Is it only me who cannot see "DQM bin by bin comparison"-> "11634.0 overlay GUI (6070)" results, but just an empty page at the end ? ...
While just looking back to PRs submitted several days ago - I see everything's OK - DQM directories/plots present.

[tvami]

I saw the same, and actually already contacted the DQM colleagues yesterday @jfernan2

[tvami]

@cmsbuild , please test

• maybe the "DQM bin by bin comparison" will work this time

[jfernan2]

There is a problem with DQM Development GUI no accepting root files, we have contacted cmsweb managers in order to restart the agents to see if it solves the problem.

@tvami the problem you reported over skype at midnight on RelVal GUI is not related to this one, since it was due to a not existing run in your comparison, and a different (RelVal) GUI

[jfernan2]

@abdoulline if you still want so see the plots in the Bin By Bin tool, use the JSRoot links (no overlay there sorry) which still have the plots

[tvami]

Hi @jfernan2 so I just re-initiated the tests, if I understand correctly this is not expected to help the problem then, right?

[abdoulline]

@jfernan2 thank you for clarifying/providing details.

[jfernan2]

Hi @jfernan2 so I just re-initiated the tests, if I understand correctly this is not expected to help the problem then, right?

I am afraid not :-(

[tvami]

@tvami it has been customary to supply the descriptions of PRs changing autoCond.py with links to the conddb browser displaying the differences of the Global Tags changed in the PR. That greatly helps with the release validation process.
Would it be possible to do it here too (even if trivial)?

Hi @mmusich I did that in the HN msg, but sure, I can copy past it here as well.

[mmusich]

I did that in the HN msg, but sure, I can copy past it here as well.

Thanks. IMHO a self-contained description is superior as it doesn't compel to look for the sources in many different places.

[abdoulline]

Thanks @abdoulline
I was surprised to see changes in OfflinePV or SiStrip/TEC and Tracking folders due to HCAL, but since I cannot overlay the plots it is difficult to tell if the changes are sensible or not.
Do we understand why 11634.912_TTbar_14TeV+2021_DD4hepDB is not affected (geometry?) or why 12834.0_TTbar_14TeV+2024 is not affected either (not real Run3 MC?) ?
Thank you

@jfernan2

• I must admit I cannot say much(anything) about OfflinePV and/or SiStrip/TEC and Tracking .
  Not sure whether actual HCAL changes at L1 (Trigger Primitives) may somehow impact those (?).

• We didn't submit updated tag to DD4HEP-related queue.

• While updated HCAL conditions went to (regular) 2021/2023/2023 GT updates, as mentioned the PR intro by Tamas
  Add HCAL tag HcalRespCorrs_2021_v3.0_mc to Run-3 MC GTs #34922 (comment)
  I don't see wf 12834.0_TTbar_14TeV in the standard PR list(s):

(a) DQM bin-by-bin comparison
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-17-1100+cd0909/44834/

(b) Comparison with the baseline:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-17-1100+cd0909/44834/

[mmusich]

@jfernan2 @abdoulline

I must admit I cannot say much(anything) about OfflinePV and/or SiStrip/TEC and Tracking .
Not sure whether actual HCAL changes at L1 (Trigger Primitives) may somehow impact those (?).

if the change in the HCAL response is sizeable, it might affect the jetCore tracking iteration, which in turn is going to change (minimally) the distribution of on-track quantities in the strips and the list of constituent tracks in the offline PVs.
A quick look to the relmon seem to confirm:

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-18-1100+cd0909/44848/12434.0_TTbar_14TeV+2023+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA/Tracking_Track.html

while there are vast changes in the later tracking iterations, there aren't in the purely pixel seeded ones (triplets / quadruplets)

[jfernan2]

+1

[jfernan2]

Thanks for the instructive explanation

[jfernan2]

Bin By Bin tool should be working by now, in case you want to retrigger the tests to see the overlay comparison

[tvami]

@cmsbuild , please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cd0909/17894/summary.html
COMMIT: 8ae5e4f
CMSSW: CMSSW_12_1_X_2021-08-19-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34922/17894/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-cd0909/7.23_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21

Summary:

• No significant changes to the logs found
• Reco comparison results: 4155 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3000352
• DQMHistoTests: Total failures: 5704
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2994626
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: found differences in 3 / 38 workflows

[jfernan2]

+1
Changes in DQM folders for Run3 MCs due to new GT

[abdoulline]

Now in overlayed plots (even with 10 ev) one can see that the changes correspond to expectations
#34922 (comment) :
(1) overall changes are small;
(2) in the region of interest (e.g. for HE depths=1,3 and |ieta|=28-29) there is RecHits energy re-balancing as expected - a decrease in forward region and an increase in the backward one.

[mseidel42]

(2) in the region of interest (e.g. for HE depths=1,3 and |ieta|=27-29) there is RecHits energy re-balancing as expected - a decrease in forward region and an increase in the backward one.

Hi Salavat, could you attach the plots that you are referring to, please? I could not find them.

[abdoulline]

@mseidel42 for 11634.0 HcalRecHitsD

(a) HE RecHits mean energy vs ieta for depth1 (https://tinyurl.com/cu9spdpp)

(b) idem for depth3 (https://tinyurl.com/4m49evns)

[perrotta]

@abdoulline could you please copy and paste the images of those plots? As such, they can remain as a reference here in the thread even when the DQM histos will get cleared (and at the same time, those like me that now, for some reason that I'm going to investigate, cannot access the DQM server can look at them as well). Thank you!

[abdoulline]

@perrotta please, find the snapshots attached to the previous post #34922 (comment)

[francescobrivio]

+alca

• As discussed during the AlCaDB meeting (https://indico.cern.ch/event/1066868/#14-fix-of-hcal-response-correc) we are ok with these changes

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1

• Updated HCAL tags for Run3 MC