Adding SiPixel on track charge and shape probs to MiniAOD content #36247

tvami · 2021-11-25T03:17:15Z

PR description:

See request to ReRECO 2017/2018 SM and MET data in [1]. This assumed refitting tracks to get the required low-level pixel reco information. As an alternative (actually it's an improvement, so better than alternative) it was recommended to save the required information in the MiniAOD. This PR does that, i.e. adds SiPixel on track charge and shape probs to MiniAOD content. Since in 2017/2018 the Pixel layer 1 was very noisy, and alternative version that excludes L1 was added as well.

Adding it to RECO/AOD as a new object siPixelTrackProbQXY:

edm::ValueMap<reco::SiPixelTrackProbQXY>    "siPixelTrackProbQXY"       ""        "RECO"
edm::ValueMap<reco::SiPixelTrackProbQXY>    "siPixelTrackProbQXY"       "NoLayer1"        "RECO"

While adding it to MiniAOD as a member of the isolatedTracks collection.

Further details presented in the Tracking POG meeting on Monday [2]

[1] https://its.cern.ch/jira/browse/PDMVRERECO-48
[2] https://indico.cern.ch/event/1098365/#5-proposal-to-use-low-level-pi

PR validation:

Running MiniAOD

cmsDriver.py ReRECO --conditions auto:run2_data --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run2_2018,Configuration/DataProcessing/Utils.addMonitoring --datatier MINIAOD --era Run2_2018,pf_badHcalMitigation --eventcontent MINIAOD --nThreads 8 --no_exec --number 100 --python_filename 4HLT2MiniAOD.py --scenario pp --step RAW2DIGI,L1Reco,RECO,EI,PAT --runUnscheduled --data  --filein root://cms-xrd-global.cern.ch//store/data/Run2018C/SingleMuon/RAW/v1/000/319/337/00000/004ED286-5582-E811-8895-FA163E126126.root --era Run2_2018

before the PR and after the PR the sizes change as follows:
-rw-r--r-- 1 tvami cms 8177333 Nov 24 16:30 ReRECO_RAW2DIGI_L1Reco_RECO_EI_PAT.root
-->
-rw-r--r-- 1 tvami cms 8185759 Nov 24 16:26 ReRECO_RAW2DIGI_L1Reco_RECO_EI_PAT.root
i.e an 1.00103041 increase in size is expected.

A unit test has been implemented as well

Running AOD

cmsDriver.py ReRECO --conditions auto:run2_data --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run2_2018 --datatier AOD --eventcontent AOD --number 100 --python_filename 4HLT2AOD.py --scenario pp --step RAW2DIGI,L1Reco,RECO --data  --filein file:/eos/cms/store/user/cmsbuild/store/data/Run2018C/SingleMuon/RAW/v1/000/319/337/00000/004ED286-5582-E811-8895-FA163E126126.root --fileout file:AOD.root --era Run2_2018

before the PR and after the PR the sizes of the AOD change as follows:
-rw-r--r--. 1 tvami zh 46456023 Dec 6 22:24 AOD.root
-->
-rw-r--r--. 1 tvami zh 47673431 Dec 6 22:23 AOD.root
i.e an 1.0262056 increase in size is expected.

Here is the timing report for before the PR

 Time Summary: 
 - Min event:   3.1252
 - Max event:   42.8974
 - Avg event:   12.985
 - Total loop:  181.876
 - Total init:  67.6784
 - Total job:   249.563
 - EventSetup Lock: 0
 - EventSetup Get:  0
 Event Throughput: 0.549825 ev/s
 CPU Summary: 
 - Total loop:     1176.02
 - Total init:     51.0267
 - Total extra:    0
 - Total children: 0.125992
 - Total job:      1227.06

and after the PR

 Time Summary: 
 - Min event:   3.22021
 - Max event:   41.7722
 - Avg event:   13.0204
 - Total loop:  180.607
 - Total init:  66.0196
 - Total job:   246.629
 - EventSetup Lock: 0
 - EventSetup Get:  0
 Event Throughput: 0.55369 ev/s
 CPU Summary: 
 - Total loop:     1183.12
 - Total init:     50.1868
 - Total extra:    0
 - Total children: 0.117669
 - Total job:      1233.31

Content was validated on MC Signal samples, and on data as well, please see more plots in [2]

if this PR is a backport please specify the original PR and why you need to backport that PR:

Needs a backport in 10_6_X

cc @mmusich

cmsbuild · 2021-11-25T03:22:35Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36247/26887

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/PatCandidates/interface/IsolatedTrack.h modified in PR(s): Adding Disappearing Tracks Cuts (EXO-19-010) to List of saved DeDxHitInfo Cuts in Isolated Tracks Slimming #36225
- File DataFormats/PatCandidates/src/classes_def_objects.xml modified in PR(s): Adding muon MVA ID #36179
- File PhysicsTools/PatAlgos/python/slimming/isolatedTracks_cfi.py modified in PR(s): Adding Disappearing Tracks Cuts (EXO-19-010) to List of saved DeDxHitInfo Cuts in Isolated Tracks Slimming #36225

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36247/26887/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36247/26887/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

RecoTracker/DeDx/plugins/SiPixelTrackProbQXYProducer.cc

slava77 · 2021-11-25T14:38:11Z

Code check has found code style and quality issues

in case it was not noticed already

cmsbuild · 2021-11-25T17:19:08Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36247/26904

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/PatCandidates/interface/IsolatedTrack.h modified in PR(s): Adding Disappearing Tracks Cuts (EXO-19-010) to List of saved DeDxHitInfo Cuts in Isolated Tracks Slimming #36225
- File DataFormats/PatCandidates/src/classes_def_objects.xml modified in PR(s): Adding muon MVA ID #36179
- File PhysicsTools/PatAlgos/python/slimming/isolatedTracks_cfi.py modified in PR(s): Adding Disappearing Tracks Cuts (EXO-19-010) to List of saved DeDxHitInfo Cuts in Isolated Tracks Slimming #36225

cmsbuild · 2021-11-25T17:19:34Z

A new Pull Request was created by @tvami (Tamas Vami) for master.

It involves the following packages:

DataFormats/PatCandidates (reconstruction)
DataFormats/TrackReco (reconstruction)
PhysicsTools/PatAlgos (reconstruction)
RecoTracker/Configuration (reconstruction)
RecoTracker/DeDx (reconstruction)

@jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @felicepantaleo, @hatakeyamak, @emilbols, @mbluj, @demuller, @mmusich, @seemasharmafnal, @mmarionncern, @ahinzmann, @JanFSchulte, @dgulhan, @jdolen, @azotz, @GiacomoSguazzoni, @rovere, @VinInn, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @ebrondol, @mtosi, @mariadalfonso, @AlexDeMoor, @JyothsnaKomaragiri, @cbernet, @gpetruc, @andrzejnovak this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2021-11-25T18:06:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36247/26906

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/PatCandidates/interface/IsolatedTrack.h modified in PR(s): Adding Disappearing Tracks Cuts (EXO-19-010) to List of saved DeDxHitInfo Cuts in Isolated Tracks Slimming #36225
- File DataFormats/PatCandidates/src/classes_def_objects.xml modified in PR(s): Adding muon MVA ID #36179
- File PhysicsTools/PatAlgos/python/slimming/isolatedTracks_cfi.py modified in PR(s): Adding Disappearing Tracks Cuts (EXO-19-010) to List of saved DeDxHitInfo Cuts in Isolated Tracks Slimming #36225

cmsbuild · 2021-11-25T18:07:08Z

Pull request #36247 was updated. @jpata, @cmsbuild, @slava77 can you please check and sign again.

tvami · 2021-11-25T18:44:42Z

@cmsbuild , please test

tvami · 2021-12-15T02:10:43Z

@smuzaffar tests have not come back in 24 hours, should re-trigger them?

smuzaffar · 2021-12-15T10:14:26Z

looks like the profiling job is failing [a] , so re-triggering the test is not going to help.

[a]
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-profiling/146/

19:31:09 processing relval_upgrade
19:31:09 ERROR reading file: relval_upgrade 'HARVESTFakeHLT_trackingMkFit_2017'
19:31:09 Traceback (most recent call last):
19:31:09   File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36247/21237/CMSSW_12_3_X_2021-12-13-1100/bin/slc7_amd64_gcc900/runTheMatrix.py", line 598, in <module>
19:31:09     ret = runSelected(opt)
19:31:09   File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36247/21237/CMSSW_12_3_X_2021-12-13-1100/bin/slc7_amd64_gcc900/runTheMatrix.py", line 23, in runSelected
19:31:09     mrd.prepare(opt.useInput, opt.refRel, opt.fromScratch)
19:31:09   File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36247/21237/CMSSW_12_3_X_2021-12-13-1100/python/Configuration/PyReleaseValidation/MatrixReader.py", line 506, in prepare
19:31:09     self.readMatrix(matrixFile, useInput, refRel, fromScratch)
19:31:09   File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36247/21237/CMSSW_12_3_X_2021-12-13-1100/python/Configuration/PyReleaseValidation/MatrixReader.py", line 251, in readMatrix
19:31:09     if self.relvalModule.steps[stepName] is None:
19:31:09 KeyError: 'HARVESTFakeHLT_trackingMkFit_2017'

mmusich · 2021-12-15T11:01:33Z

looks like the profiling job is failing [a] , so re-triggering the test is not going to help.

to solve this, PR #36477 is needed (see also #36347 (comment))

tvami · 2021-12-15T14:18:41Z

@cmsbuild , please abort

cmsbuild · 2021-12-15T14:41:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36247/27386

This PR adds an extra 176KB to repository
Found files with invalid states:
- RecoTracker/DeDx/test/ProbQXYFromMiniAOD.py:
  - Added: 63380ba
  - Deleted: b10da8b
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/PatCandidates/src/classes_def_objects.xml modified in PR(s): Fix calculation of smeared MET on MiniAOD #36165, Adding muon MVA ID #36179

cmsbuild · 2021-12-15T14:41:27Z

Pull request #36247 was updated. @perrotta, @gouskos, @clacaputo, @fabiocos, @cmsbuild, @fgolf, @slava77, @jpata, @mariadalfonso, @qliphy, @davidlange6 can you please check and sign again.

tvami · 2021-12-15T14:42:28Z

@cmsbuild , please test with #36477

cmsbuild · 2021-12-15T22:30:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecf9e7/21289/summary.html
COMMIT: 75b2f0e
CMSSW: CMSSW_12_3_X_2021-12-15-1100/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36247/21289/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 3250719
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3250697
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

tvami · 2021-12-16T19:00:07Z

The idea from Slava to use the DeDxHitInfo's clusters and an approximation for the angles seems to be giving compatible results with my proposed solution. I'm removing the urgent label and will modify the PR so it keeps the producer only. That might still be useful for a possible future skim.

tvami · 2021-12-16T19:20:24Z

I'm closing this PR too, I'll open up a new one that will only contain the producer, maybe that actually could be attached to the skim development as well!

tvami marked this pull request as draft November 25, 2021 03:17

tvami changed the title ~~Draft: Adding SiPixel on track charge and shape probs to MiniAOD content~~ Adding SiPixel on track charge and shape probs to MiniAOD content Nov 25, 2021

cmsbuild added this to the CMSSW_12_2_X milestone Nov 25, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Nov 25, 2021

tvami mentioned this pull request Nov 25, 2021

[10_6_X] Adding SiPixel on track charge and shape probs to MiniAOD content #36244

Closed

cmsbuild added code-checks-rejected and removed code-checks-pending labels Nov 25, 2021

tvami commented Nov 25, 2021

View reviewed changes

RecoTracker/DeDx/plugins/SiPixelTrackProbQXYProducer.cc Outdated Show resolved Hide resolved

tvami force-pushed the probQInMiniAODMaster branch from e3d255f to 77a80d9 Compare November 25, 2021 17:10

cmsbuild added code-checks-pending and removed code-checks-rejected labels Nov 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 25, 2021

tvami force-pushed the probQInMiniAODMaster branch from 77a80d9 to c9dc302 Compare November 25, 2021 17:58

cmsbuild added code-checks-pending and removed code-checks-approved labels Nov 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 25, 2021

tvami marked this pull request as ready for review November 25, 2021 18:43

mmusich mentioned this pull request Dec 15, 2021

remove unused step from mkFit special workflow #36477

Merged

cmsbuild added tests-pending and removed tests-started labels Dec 15, 2021

tvami force-pushed the probQInMiniAODMaster branch from 2001fee to ddde69d Compare December 15, 2021 14:30

cmsbuild added code-checks-pending operations-pending and removed code-checks-approved labels Dec 15, 2021

Add proc modifier

75b2f0e

tvami force-pushed the probQInMiniAODMaster branch from ddde69d to 75b2f0e Compare December 15, 2021 14:34

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 15, 2021

cmsbuild added requires-external tests-started and removed tests-pending labels Dec 15, 2021

cmsbuild added tests-approved and removed tests-started labels Dec 15, 2021

cmsbuild mentioned this pull request Dec 16, 2021

Remove Event Interpretation (EI) from relval steps #36465

Merged

cmsbuild removed the urgent label Dec 16, 2021

tvami closed this Dec 16, 2021

mmusich mentioned this pull request Mar 29, 2022

review content saved in miniAOD in DeDxHitInfo to possibly reduce the size on file #36405

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding SiPixel on track charge and shape probs to MiniAOD content #36247

Adding SiPixel on track charge and shape probs to MiniAOD content #36247

tvami commented Nov 25, 2021 •

edited

Loading

cmsbuild commented Nov 25, 2021

slava77 commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

tvami commented Nov 25, 2021

tvami commented Dec 15, 2021

smuzaffar commented Dec 15, 2021

mmusich commented Dec 15, 2021 •

edited

Loading

tvami commented Dec 15, 2021

cmsbuild commented Dec 15, 2021

cmsbuild commented Dec 15, 2021

tvami commented Dec 15, 2021

cmsbuild commented Dec 15, 2021

tvami commented Dec 16, 2021

tvami commented Dec 16, 2021

Adding SiPixel on track charge and shape probs to MiniAOD content #36247

Adding SiPixel on track charge and shape probs to MiniAOD content #36247

Conversation

tvami commented Nov 25, 2021 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Nov 25, 2021

slava77 commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

tvami commented Nov 25, 2021

tvami commented Dec 15, 2021

smuzaffar commented Dec 15, 2021

mmusich commented Dec 15, 2021 • edited Loading

tvami commented Dec 15, 2021

cmsbuild commented Dec 15, 2021

cmsbuild commented Dec 15, 2021

tvami commented Dec 15, 2021

cmsbuild commented Dec 15, 2021

Comparison Summary

tvami commented Dec 16, 2021

tvami commented Dec 16, 2021

tvami commented Nov 25, 2021 •

edited

Loading

mmusich commented Dec 15, 2021 •

edited

Loading