DisplacedRegionalStep track DNN

[aehart]

PR description:

This PR changes the track DNN used by the DisplacedRegionalStep to a model trained on tracks from this iteration. Details for this training can be found in the following presentation:
https://indico.cern.ch/event/1345175/#12-displaced-regional-tracking

See accompanying PR in cms-data/RecoTracker-FinalTrackSelectors#13.

PR validation:

The tracking-only RECO has been run with these changes in CMSSW_14_0_X_2023-11-12-2300, with fake rates and efficiencies in the above talk.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43336/37785

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @aehart (Andrew Hart) for master.

It involves the following packages:

• RecoTracker/FinalTrackSelectors (reconstruction)
• RecoTracker/IterativeTracking (reconstruction)

@cmsbuild, @jfernan2, @mandrenguyen can you please review it and eventually sign? Thanks.
@mmusich, @JanFSchulte, @felicepantaleo, @rovere, @VinInn, @mtosi, @GiacomoSguazzoni, @dgulhan, @missirol, @VourMa, @ebrondol, @gpetruc this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[slava77]

test parameters:

• pull_request = DisplacedRegionalStep track DNN model file cms-data/RecoTracker-FinalTrackSelectors#13
• workflow = 12434.701
• relvals_opt= -w upgrade

[slava77]

@cmsbuild please test

[cmsbuild]

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f172bf/35964/summary.html
COMMIT: d5a94bc
CMSSW: CMSSW_14_0_X_2023-11-20-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43336/35964/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @Dr15Jones Fix tracer log viewer issue with callID #43328
• @iarspider Remove explicit copy-constructor from TotemTimingDigi #43315
• @Michael-Krohn ZDC trigger primitive emulation and HCAL look up table generation #42818
• @missirol Add HLT-Scouting collections to MINIAOD event content (follow-up of #42863) #43327
• @iarspider Fix array-bounds warning in DTMuonMillepede #43313
• @civanch [14_0_X SIM] Replace deprecated method #43321
• @smuzaffar cleanup classlib string List/Format usage #43320
• @nurfikri89 [JMENano] Fix GenJetIdx for AK4 jets matched to low pt genjets #43144
• @aandvalenzuela [XPOG][RECO] Fix DEVEL warnings on deprecated copy-constructors #43316
• @bsunanda Run3-sim151 Modify the run script for the SIM step to get hits in the ZDC #43267
• @bsunanda Run3-sim153 Reorganize the codes relevant to forward detectors among Calo and Forward projects in SimG4CMS #43311
• @wddgit Avoid unnecessary Framework delays closing luminosity blocks (simple version) #43263
• @yuanchao remove try/catch clause on accessing vertex collection #43282
• @swagata87 Adapt PF code to read Hcal thresholds from GT #43025

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f172bf/35964/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f172bf/35964/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test RecoPPSLocalNewT2 had ERRORS

RelVals

ValueError: Undefined workflows: 12434.701

[slava77]

ValueError: Undefined workflows: 12434.701

@aehart
please add the following in class UpgradeWorkflow_displacedRegional in Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

return '2021' in key -> return ('2021' in key or '2023' in key or '2024' in key)

and then also in the steps = add RecoNanoFakeHLT

this will expand the coverage to the default workflows. Currently nothing matches to build a displacedRegional variant

[slava77]

please add the following in class UpgradeWorkflow_displacedRegional in Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

actually, it's probably more practical to go via a separate PR, I made #43353

[slava77]

@cmsbuild please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f172bf/36028/summary.html
COMMIT: d5a94bc
CMSSW: CMSSW_14_0_X_2023-11-22-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43336/36028/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 348 lines to the logs
• Reco comparison results: 11 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3367918
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3367889
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 214 log files, 167 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[slava77]

@iarspider @smuzaffar
I was expecting comparisons to show up for 12434.701_TTbar_14TeV+2023_displacedRegional workflow.

• the matrix outputs are available
• it also got registered in the wf_mapping file
• but then there is nothing in the comparisons folder

[slava77]

@cmsbuild please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f172bf/36064/summary.html
COMMIT: d5a94bc
CMSSW: CMSSW_14_0_X_2023-11-24-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43336/36064/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 44 lines from the logs
• Reco comparison results: 298 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3529118
• DQMHistoTests: Total failures: 2474
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3526622
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 172 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[slava77]

I was expecting comparisons to show up for 12434.701_TTbar_14TeV+2023_displacedRegional workflow.

uhm, the comparisons showed up this time.
I guess there is some time delay in the system before a new workflow can be usable for comparisons (I used the same initial IB at first CMSSW_14_0_X_2023-11-22-2300)

[smuzaffar]

@iarspider @smuzaffar I was expecting comparisons to show up for 12434.701_TTbar_14TeV+2023_displacedRegional workflow.

@slava77 , this is strange. Baseline job to run 12434.701 could not find this workflow in CMSSW_14_0_X_2023-11-22-2300/el8_amd64_gcc12 IB [a] while if I run now the same command again in this IB then it properly returns the workflow number [b]. I just have restarted the baseline job to try to run 12434.701 in CMSSW_14_0_X_2023-11-22-2300/el8_amd64_gcc12 IB and this time it has found the workflow [c]. Is generation of this workflow depend on some external service ?

[a] https://cmssdt.cern.ch/jenkins/job/ib-run-baseline/8250/console
cmd: https://github.com/cms-sw/cms-bot/blob/master/run-ib-pr-matrix.sh#L25C92-L25C92

+ runTheMatrix.py -n -w upgrade -l 12434.701
+ grep '^[1-9][0-9]*\(.[0-9][0-9]*\|\)\s'
+ sed 's| .*||'
+ grep -v ' workflows '
++ cat /tmp/cmsbuild/jenkins/workspace/ib-run-baseline/req.wfs
++ echo
++ sed 's|,$||'
+ WFS=

[b]

Singularity> runTheMatrix.py -n -w upgrade -l 12434.701 | grep -v ' workflows ' | grep '^[1-9][0-9]*\(.[0-9][0-9]*\|\)\s' | sed 's| .*||'
12434.701

[c] https://cmssdt.cern.ch/jenkins/job/ib-run-baseline/8273/console

+ runTheMatrix.py -n -w upgrade -l 12434.701
+ grep -v ' workflows '
+ grep '^[1-9][0-9]*\(.[0-9][0-9]*\|\)\s'
+ sed 's| .*||'
++ cat /data/cmsbld/jenkins/workspace/ib-run-baseline/req.wfs
+ for wf in $(cat $WORKSPACE/req.wfs)
++ echo ' 4.22 4.53 5.1 7.3 8.0 9.0 25.0 135.4 136.731 136.7611 136.793 136.8311 136.874 136.88811 138.4 138.5 139.001 140.53 140.56 141.044 158.01 1306.0 1330.0 2018.1 101.0 312.0 25202.0 1000.0 1001.0 10024.0 10042.0 10224.0 10824.0 11634.0 11634.911 11634.914 12434.0 12434.7 12634.0 13234.0 14034.0 14234.0 23234.0 24834.0 24834.911 24896.0 24900.0 25034.999 250202.181 2500.4  '
++ grep ' 12434.701 '
++ wc -l
+ '[' 0 -eq 0 ']'
+ WFS=12434.701,

[slava77]

@cms-sw/reconstruction-l2
please check this PR.
Thank you.

[jfernan2]

+1
From reco point of view, I cannot judge differences in Physics output

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1

[smuzaffar]

@cms-sw/orp-l2 , last night IBs failed as this PR was merged without merging the cms-data/RecoTracker-FinalTrackSelectors#13 ( and its change in cmsdist repo) .When you merge a cmssw PR then can you please check if requires-external label is set and if it requires a cms-data PR then merge that PR and the corresponding cmsdist PR ( which bot opens once a data PR is merged)?