Use akCs4PFJets for candidate-based tagInfos for HI workflows #31674

mandrenguyen · 2020-10-05T18:23:14Z

PR description:

Update candidate based tagInfos to use HI jet collection (akCs4PFJets) in HI workflows.
This fixes an issue created in #30898, which is causing crashes in several HI workflows ( #31670 ).
The issue is that the b-tagging validation clones the 'patJets', but changes from the track-based tagInfos that we use to the candidate-based ones used nowadays in pp.
The solution is to update the candidate-based tagInfos to use akCs4PFJets, which was anyway on the to-do list for Run 3, when we plan update to those in the HI b-tagging sequence. This issue was spotted running wfs like 158.1.

There was also a trivial change to the name of the cleaned heavy ion genJet module, which had a conflict (affected wf 150, for example).

PR validation:

Tested on wf 150, 158.1 and 158.01. Please test on the other wfs identified in #31670

if this PR is a backport please specify the original PR and why you need to backport that PR:

Before submitting your pull requests, make sure you followed this checklist:

verify that the PR is really intended for the chosen branch
verify that changes follow CMS Naming, Coding, And Style Rules
verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

cmsbuild · 2020-10-05T18:23:40Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-05T18:30:07Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31674/18802

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/PatAlgos/python/producersHeavyIons/heavyIonJets_cff.py modified in PR(s): Legacy csvV2 calibrations for 2018 PbPb data #31660

cmsbuild · 2020-10-05T18:30:27Z

A new Pull Request was created by @mandrenguyen (Matthew Nguyen) for master.

It involves the following packages:

DQMOffline/RecoB
PhysicsTools/PatAlgos
RecoBTag/ImpactParameter
RecoBTag/SoftLepton

@perrotta, @andrius-k, @kmaeshima, @ErnestaP, @cmsbuild, @jfernan2, @fioriNTU, @slava77, @jpata, @santocch can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @hatakeyamak, @emilbols, @peruzzim, @seemasharmafnal, @mmarionncern, @ahinzmann, @smoortga, @jdolen, @ferencek, @rociovilar, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @andrzejnovak, @clelange, @riga, @JyothsnaKomaragiri, @gpetruc, @mariadalfonso this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

slava77 · 2020-10-05T19:05:28Z

@cmsbuild please test workflow 158.1, 158.2, 158.3, 159.1, 159.3, 159.4, 300.0, 301.0, 302.0, 130.0, 311.0, 312.0

cmsbuild · 2020-10-05T19:05:47Z

The tests are being triggered in jenkins.
Test Parameters:

MATRIX_EXTRAS = 158.1,301.0,158.3,158.2,302.0,130.0,300.0,159.1,159.3,159.4,311.0,312.0
CMSSW_11_2_X_2020-10-05-1200/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/9741/console Started: 2020/10/05 21:07

cmsbuild · 2020-10-05T21:23:18Z

-1

Tested at: 8474eb7

CMSSW: CMSSW_11_2_X_2020-10-05-1200
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e08935/9741/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

Unit Tests:

I found errors in the following unit tests:

---> test TestDQMOfflineConfiguration100 had ERRORS
---> test TestDQMOfflineConfiguration50 had ERRORS

cmsbuild · 2020-10-05T21:23:20Z

Comparison job queued.

slava77 · 2020-10-05T21:49:42Z

---> test TestDQMOfflineConfiguration100 had ERRORS

Exception: 'Phase2C11M9' is not a valid option for '--era'

looks unrelated to this PR

---> test TestDQMOfflineConfiguration50 had ERRORS

has the same issue.

cmsbuild · 2020-10-05T22:49:03Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e08935/9741/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/130.0_SinglePiPt10+SinglePiPt10+DIGI+RECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/158.1_QCD_Pt_80_120_13_HI+QCD_Pt_80_120_13_HI+DIGIHI2018PPRECO+RECOHI2018PPRECO+HARVESTHI2018PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/158.2_PhotonJets_Pt_10_13_HI+PhotonJets_Pt_10_13_HI+DIGIHI2018PPRECO+RECOHI2018PPRECO+HARVESTHI2018PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/158.3_ZEEMM_13_HI+ZEEMM_13_HI+DIGIHI2018PPRECO+RECOHI2018PPRECO+HARVESTHI2018PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/159.1_QCD_Pt_80_120_14_HI_2021+QCD_Pt_80_120_14_HI_2021+DIGIHI2021PPRECO+RECOHI2021PPRECO+HARVESTHI2021PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/159.3_ZMM_14_HI_2021+ZMM_14_HI_2021+DIGIHI2021PPRECO+RECOHI2021PPRECO+HARVESTHI2021PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/159.4_ZEE_14_HI_2021+ZEE_14_HI_2021+DIGIHI2021PPRECO+RECOHI2021PPRECO+HARVESTHI2021PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/300.0_Pyquen_GammaJet_pt20_2760GeV+Pyquen_GammaJet_pt20_2760GeV+DIGIHIMIX+RECOHIMIX+HARVESTHI2018PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/301.0_Pyquen_DiJet_pt80to120_2760GeV+Pyquen_DiJet_pt80to120_2760GeV+DIGIHIMIX+RECOHIMIX+HARVESTHI2018PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/311.0_Pyquen_DiJet_pt80to120_2760GeV_2021+Pyquen_DiJet_pt80to120_2760GeV_2021+DIGIHI2021MIX+RECOHI2021MIX+HARVESTHI2021PPRECO
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e08935/312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2021+Pyquen_ZeemumuJets_pt10_2760GeV_2021+DIGIHI2021MIX+RECOHI2021MIX+HARVESTHI2021PPRECO

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 146 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2542225
DQMHistoTests: Total failures: 1616
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2540587
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

slava77 · 2020-10-06T00:16:41Z

The solution is to update the candidate-based tagInfos to use akCs4PFJets, which was anyway on the to-do list for Run 3, when we plan update to those in the HI b-tagging sequence.

so, this is not really a bigfix, it's a feature change for AOD default b-tagging.
Please update the PR title to reflect this.

I'm still assuming that we stay in 11_X for HI miniAOD.
A more appropriate/targeted bugfix will be needed if there is still intent for a backport.

slava77

it looks like a limited bugfix would be to modify patJetsBDHadron in Validation/RecoB/python/BDHadronTrackValidation_cff.py

Updates in soft lepton taginfos are apparently not related to the crashes.
Are the updates made here going to produce usable results or is it likely that everything will need to be reconfigured in a significant way?

slava77 · 2020-10-06T00:32:40Z

RecoBTag/ImpactParameter/python/pfImpactParameterTagInfos_cfi.py

+
+from Configuration.Eras.Modifier_pp_on_AA_2018_cff import pp_on_AA_2018
+from Configuration.Eras.Modifier_pp_on_PbPb_run3_cff import pp_on_PbPb_run3
+(pp_on_AA_2018 | pp_on_PbPb_run3).toModify(pfImpactParameterTagInfos, jets = "akCs4PFJets")


is this enough to correctly run pfImpactParameterTagInfos for HI?
(similar question applies to the rest).

No, it was not sufficient to modify pfImpactParamterTagInfos. The soft lepton tag infos had to use the consistent jet collection to avoid crashes. I first modified the input to pfImpactParamterTagInfos, but the code was still crashing until I updated the other tag infos. All of the modifications made are necessary to avoid crashes.

I didn't mean the soft lepton taggers.
My question was if the switch of the jet for pfImpactParameterTagInfos expected to provide performance of pfImpactParameterTagInfos acceptable for physics in HI or if more significant changes are needed.

The difference between impactParamterTagInfos and pfImpactParameterTagInfos is basically technical. There should not be any significant difference in performance. I verified this some time ago. We will check this again before actually calculating b-tag discriminators based on the candidate tagInfos. At the moment, they are not really used for anything in the HI workflow besides this validation package, AFAIK.

cmsbuild · 2020-10-06T17:18:12Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa7725/9758/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 142 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2542113
DQMHistoTests: Total failures: 1625
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2540466
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -773.364 KiB( 34 files compared)
DQMHistoSizes: changed ( 10024.0,... ): -48.369 KiB L1T/L1TStage2CaloLayer1
DQMHistoSizes: changed ( 136.731 ): -47.829 KiB L1T/L1TStage2CaloLayer1
Checked 149 log files, 22 edm output root files, 35 DQM output files

silviodonato · 2020-10-06T19:56:58Z

merge
@cms-sw/analysis-l2 @cms-sw/dqm-l2 please let us know if you have comments

Martin-Grunewald · 2020-10-08T08:03:08Z

It seems there is still a problem in an HIon workflow (HLT validation tests suite):

https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_11_2_X_2020-10-07-2300/slc7_amd64_gcc820/RelVal_RECO_HIon_MC.log

----- Begin Fatal Exception 08-Oct-2020 07:51:28 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 12 event: 3302 stream: 3
   [1] Running path 'validation_step'
   [2] Calling method for module B2GHadronicHLTValidation/'b2gDiJetHLTValidatio\
n'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::Jet
Looking for module label: ak8PFJetsPuppi
Looking for productInstanceName:

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exc\
eption,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSe\
t in the configuration.

----- End Fatal Exception -------------------------------------------------

mandrenguyen · 2020-10-08T08:19:34Z

Hi @Martin-Grunewald
So this is different than the addOnTest I checked, right?:
addOnTests.py -t hlt_mc_HIon
Pease give me a command to reproduce this error, and I'll have a look.

Martin-Grunewald · 2020-10-08T08:32:39Z

Yes, it runs some more steps after reco: --step=RAW2DIGI,L1Reco,RECO,EI,PAT,VALIDATION,DQM

In a recent developer area:

...
cd src
cmsenv
git cms-addpkg HLTrigger/Configuration
scram b
cd HLTrigger/Configuration/test
# create job files (takes a few minutes)
./cmsDriver.csh HIon
# prepare input file using cmsRun:
cmsRun RelVal_DigiL1RawHLT_HIon_MC.py >& RelVal_DigiL1RawHLT_HIon_MC.log
# run problematic cmsRunjob:
cmsRun RelVal_RECO_HIon_MC.py >& RelVal_RECO_HIon_MC.log

mandrenguyen · 2020-10-08T11:05:54Z

This error was not picked up in the standard relval wf 159 because the validation is limited to VALIDATION:@standardValidationNoHLT+@miniAODValidation
I verified that no other crashes are present when simply removing b2gHLTriggerValidation from HLTriggerOffline.Common.HLTValidation_cff
I'm not immediately sure why an empty version of ak8PFJetsPuppi is not present. I will see if I can straighten that out before resorting to removing the validation sequence (although I can't imagine it's useful in heavy ions)

slava77 · 2020-10-08T12:14:11Z

I think that eventually the goal should be to cleanup the DQM/Validation for the HI setup.
It may be more practical to patch this up by inserting empty/dummy collections from the reco side for now.

mandrenguyen · 2020-10-08T12:19:26Z

@slava77 Agreed. Is this jet collection dropped from the following commit you made?
d225fa1
It looks like it disables applySubstructure, which is where this jet collection is produced, right?

santocch · 2020-10-08T12:27:12Z

+1

slava77 · 2020-10-08T12:56:31Z

It looks like it disables applySubstructure, which is where this jet collection is produced, right?

right, the creation of all modules is short-circuited

jfernan2 · 2020-10-08T14:33:56Z

+1
DQM Btag folder in wf140.56 has as many as 1617 histograms modified by this PR, I understand they are produced by the change of PF candidates
E.g. https://tinyurl.com/yyu62zpm

cmsbuild · 2020-10-08T14:34:18Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

mandrenguyen added 2 commits October 5, 2020 18:02

rename cleaned genJet collection

153b1af

Use akCs4PFJets for candidate-based tag infos

8474eb7

cmsbuild added this to the CMSSW_11_2_X milestone Oct 5, 2020

cmsbuild added analysis-pending code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 5, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 5, 2020

cmsbuild added tests-started and removed tests-pending labels Oct 5, 2020

cmsbuild added tests-rejected and removed tests-started labels Oct 5, 2020

cmsbuild added comparison-available and removed comparison-pending labels Oct 5, 2020

slava77 reviewed Oct 6, 2020

View reviewed changes

cmsbuild added comparison-pending tests-approved and removed tests-started labels Oct 6, 2020

cmsbuild added comparison-available and removed comparison-pending labels Oct 6, 2020

cmsbuild mentioned this pull request Oct 6, 2020

Legacy csvV2 calibrations for 2018 PbPb data #31660

Merged

cmsbuild added orp-approved and removed orp-pending labels Oct 6, 2020

cmsbuild merged commit 8d89aa3 into cms-sw:master Oct 6, 2020

cmsbuild mentioned this pull request Oct 6, 2020

Add a customisation to run the "GRun" HLT menu on Run 2 data/MC #31655

Merged

smuzaffar mentioned this pull request Oct 7, 2020

Segfault in BDHadronTrackMonitoringAnalyzer #31670

Closed

cmsbuild added analysis-approved and removed analysis-pending labels Oct 8, 2020

slava77 mentioned this pull request Oct 8, 2020

Bad PF candidate filter for 2018 PbPb data #31668

Merged

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Oct 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use akCs4PFJets for candidate-based tagInfos for HI workflows #31674

Use akCs4PFJets for candidate-based tagInfos for HI workflows #31674

mandrenguyen commented Oct 5, 2020 •

edited

Loading

cmsbuild commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

slava77 commented Oct 5, 2020

cmsbuild commented Oct 5, 2020 •

edited

Loading

cmsbuild commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

slava77 commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

slava77 commented Oct 6, 2020

slava77 left a comment

slava77 Oct 6, 2020

mandrenguyen Oct 6, 2020 •

edited

Loading

slava77 Oct 6, 2020

mandrenguyen Oct 6, 2020

cmsbuild commented Oct 6, 2020

silviodonato commented Oct 6, 2020

Martin-Grunewald commented Oct 8, 2020

mandrenguyen commented Oct 8, 2020

Martin-Grunewald commented Oct 8, 2020

mandrenguyen commented Oct 8, 2020

slava77 commented Oct 8, 2020

mandrenguyen commented Oct 8, 2020

santocch commented Oct 8, 2020

slava77 commented Oct 8, 2020

jfernan2 commented Oct 8, 2020

cmsbuild commented Oct 8, 2020

Use akCs4PFJets for candidate-based tagInfos for HI workflows #31674

Use akCs4PFJets for candidate-based tagInfos for HI workflows #31674

Conversation

mandrenguyen commented Oct 5, 2020 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

slava77 commented Oct 5, 2020

cmsbuild commented Oct 5, 2020 • edited Loading

cmsbuild commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

slava77 commented Oct 5, 2020

cmsbuild commented Oct 5, 2020

slava77 commented Oct 6, 2020

slava77 left a comment

Choose a reason for hiding this comment

slava77 Oct 6, 2020

Choose a reason for hiding this comment

mandrenguyen Oct 6, 2020 • edited Loading

Choose a reason for hiding this comment

slava77 Oct 6, 2020

Choose a reason for hiding this comment

mandrenguyen Oct 6, 2020

Choose a reason for hiding this comment

cmsbuild commented Oct 6, 2020

silviodonato commented Oct 6, 2020

Martin-Grunewald commented Oct 8, 2020

mandrenguyen commented Oct 8, 2020

Martin-Grunewald commented Oct 8, 2020

mandrenguyen commented Oct 8, 2020

slava77 commented Oct 8, 2020

mandrenguyen commented Oct 8, 2020

santocch commented Oct 8, 2020

slava77 commented Oct 8, 2020

jfernan2 commented Oct 8, 2020

cmsbuild commented Oct 8, 2020

mandrenguyen commented Oct 5, 2020 •

edited

Loading

cmsbuild commented Oct 5, 2020 •

edited

Loading

mandrenguyen Oct 6, 2020 •

edited

Loading