Adding ability to output PF Candidate information in JME NTuples #31795

laurenhay · 2020-10-14T18:11:14Z

PR description:

Adding ability to output PF Candidate information belonging to a desired jet collection along with a mapping between said jets and candidates to the custom JME workflows.

Related materials:
NanoAODJMAR github: https://github.com/cms-jet/NanoAODJMAR
JME Twiki: https://twiki.cern.ch/twiki/bin/view/CMS/JMECustomNanoAOD
PR for NanoAODJMAR's candidate tables: #29550
Last JMAR meeting slides announcing JME+PF: https://indico.cern.ch/event/962252/contributions/4047997/attachments/2116746/3562198/JMEBTVnanoDiscussion_20201006_updated.pdf

PR validation:

Added runTheMatrix workflows:
25202.16
which adds only AK8 PF Candidates to the JME workflow 25202.15, and:
25202.17
which adds all PF Candidates to 25202.15.

if this PR is a backport please specify the original PR and why you need to backport that PR:

N/A

Tagging @rappoccio

cmsbuild · 2020-10-14T18:11:38Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-14T18:16:26Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31795/19074

This PR adds an extra 80KB to repository
Found files with invalid states:
- PhysicsTools/NanoAOD/python/addBTV.py:
  - Added: 2801813
  - Deleted: d736f42
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Nano gen dqm #31591, Bad PF candidate filter for 2018 PbPb data #31668, fix for displaced susy relval production (TRK and EXO validation of displaced tracks) #31772, Sub-event aware genJet and parton matching for HI miniAOD #31698
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Integrating VectorHits reconstruction for the Phase2 OT - reprise #31314, MTD geometry/reconstruction: define new ETL reco layer geometry, update LGAD pixel structure, activate workflows for scenarios D72 and D73 #31765, MTD geometry: add scenarios D72 (ETL v4) and D73 (ETL v5), remove D53 #31710
- File PhysicsTools/NanoAOD/plugins/BuildFile.xml modified in PR(s): PPS nanoAOD #31531
- File PhysicsTools/NanoAOD/python/custom_jme_cff.py modified in PR(s): fix: adjust jme_nano cff for changes in #31618 #31786, Add ParticleNetAK4 to JME custom NanoAOD #31714

cmsbuild · 2020-10-14T18:16:48Z

A new Pull Request was created by @laurenhay for master.

It involves the following packages:

Configuration/PyReleaseValidation
PhysicsTools/NanoAOD

@gouskos, @jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @fgolf, @mariadalfonso, @santocch can you please review it and eventually sign? Thanks.
@makortel, @fabiocos, @Martin-Grunewald, @gpetruc, @peruzzim, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

mariadalfonso · 2020-10-14T18:59:34Z

-1

There is no consensus on how to flatten the edm objects
We have not discussed how technically proceed with the multiplication of flat tree for Run3 developments
Another reason for not accepting this proposal is the large side of the trees > 20 kb/event: it's more than 10 times larger than central nano (1.5k/ev) and an ideal analysis format of that dimension already exist (miniAOD)

the jmeNano is currently unattended.
when experts updated the central nano, lead to crash in the JMEtree.
#31446 (from Lauren JME)
#31786 (from BTV)

The nanoGen is much more structured and should inspire additional tree for PPD and POGs

This should be first presented in the XPOG meeting (next occasion is on November 4)

cmsbuild · 2020-10-14T18:59:59Z

@laurenhay, CMSSW_11_2_X branch is closed for direct updates. cms-bot is going to move this PR to master branch.
In future, please use cmssw master branch to submit your changes.

laurenhay · 2020-10-15T18:08:07Z

For more context, this is what some sample contents are:
https://lahay.web.cern.ch/lahay/website2/PFJME_JetHT2017UL_size.html
https://lahay.web.cern.ch/lahay/website2/PFJME_JetHT2017UL_description.html
https://lahay.web.cern.ch/lahay/website2/AK8PFJME_JetHT2017UL_size.html
https://lahay.web.cern.ch/lahay/website2/AK8PFJME_JetHT2017UL_description.html
https://lahay.web.cern.ch/lahay/website2/JME_JetHT2017UL_size.html
https://lahay.web.cern.ch/lahay/website2/JME_JetHT2017UL_description.html
html files starting with PFJME are using the custom JME and have all PF cands stored, AK8PFJME is only the fatjet candidates, and JME is with no PF cands.
The configuration files for these samples were created using the following command in 11_2_pre7:

cmsDriver.py step1 --data --eventcontent NANOAOD --datatier NANOAOD --conditions auto:run2_data --step NANO --era Run2_2017,run2_nanoAOD_106Xv1 --python_filename PFJME_Data17UL_cfg.py --no_exec --customise_commands="process.add_(cms.Service('InitRootHandlers', EnableIMT = cms.untracked.bool(False))) \n from PhysicsTools.NanoAOD.custom_jme_cff import PrepJMECustomPFNanoAOD_Data; PrepJMECustomPFNanoAOD_Data(process)\n "

And then directly editing the config file to change the input file to:

/store/data/Run2017B/JetHT/MINIAOD/09Aug2019_UL2017-v1/130000/07017B4E-9794-DB40-BFBE-F8664579C066.root

and the number of events to be 10000.
This will not exclusively be used for a central production, and is also being used by BTV and JMAR for private samples.

chayanit · 2020-10-17T06:07:56Z

Hello @laurenhay, why do we need new workflows for all 3 MC year? Is it supposed to be included in IB test only?

rappoccio · 2020-10-19T17:30:52Z

Hello @laurenhay, why do we need new workflows for all 3 MC year? Is it supposed to be included in IB test only?

Hi @chayanit, do you mean here ?

chayanit · 2020-10-20T09:39:55Z

Hello @laurenhay, why do we need new workflows for all 3 MC year? Is it supposed to be included in IB test only?

Hi @chayanit, do you mean here ?

Hi @rappoccio , I mean https://github.com/cms-sw/cmssw/pull/31795/files#diff-09f871ad2ecabe8c1c5df4ce58cdfe28cb19dc1194361f8b29ebc7278869ca85R43-R44 and https://github.com/cms-sw/cmssw/pull/31795/files#diff-b0dce787f15ea8526a7b5585e797ba4b9e17ba829550c8283b5b5f97cba58b04R3138-R3141. Why do we need to declare new workflows for 2016? Shouldn't it need only IB test which are included here instead? https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/relval_2017.py

silviodonato · 2020-10-29T17:45:41Z

Hello @laurenhay, why do we need new workflows for all 3 MC year? Is it supposed to be included in IB test only?

@laurenhay

laurenhay · 2020-10-29T18:56:58Z

Hello @laurenhay, why do we need new workflows for all 3 MC year? Is it supposed to be included in IB test only?

Hi @chayanit, do you mean here ?

Hi @rappoccio , I mean https://github.com/cms-sw/cmssw/pull/31795/files#diff-09f871ad2ecabe8c1c5df4ce58cdfe28cb19dc1194361f8b29ebc7278869ca85R43-R44 and https://github.com/cms-sw/cmssw/pull/31795/files#diff-b0dce787f15ea8526a7b5585e797ba4b9e17ba829550c8283b5b5f97cba58b04R3138-R3141. Why do we need to declare new workflows for 2016? Shouldn't it need only IB test which are included here instead? https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/relval_2017.py

I still don't know what you're referring to since we don't declare a workflow for 2016. The workflow we are upgrading for 2017 and 2018. The diff links you provided are broken as well.

chayanit · 2020-11-01T13:29:05Z

Configuration/PyReleaseValidation/python/relval_pileup.py

@@ -40,6 +40,8 @@
 workflows[25202.1]=['',['TTbar_13','DIGIUP15APVSimu_PU25','RECOUP15_PU25','HARVESTUP15_PU25']]
 workflows[25202.2]=['',['TTbar_13','DIGIUP15APVSimu_PU25','RECOUP15_PU25_HIPM','HARVESTUP15_PU25']]
 workflows[25202.15]=['',['TTbar_13','DIGIUP15_PU25','RECOUP15_PU25','HARVESTUP15_PU25','NANOUP15MC_PU25_JME']]
+workflows[25202.16]=['',['TTbar_13','DIGIUP15_PU25','RECOUP15_PU25','HARVESTUP15_PU25','NANOUP15MC_PU25_JMEAK8PF']]
+workflows[25202.17]=['',['TTbar_13','DIGIUP15_PU25','RECOUP15_PU25','HARVESTUP15_PU25','NANOUP15MC_PU25_JMEPF']]


what you declared here is for 2016 not 2017 and 2018 @laurenhay. If you don't want, you need to remove them

chayanit · 2020-11-01T13:29:20Z

Configuration/PyReleaseValidation/python/relval_steps.py

+steps['NANOUP15MC_PU25_JMEAK8PF']=merge([{'--customise':'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomAK8PFNanoAOD_MC'},steps['NANOUP15']])
+steps['NANOUP15Data_PU25_JMEAK8PF']=merge([{'--customise':'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomAK8PFNanoAOD_Data','--data':''},steps['NANOUP15']])
+steps['NANOUP15MC_PU25_JMEPF']=merge([{'--customise':'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomPFNanoAOD_MC'},steps['NANOUP15']])
+steps['NANOUP15Data_PU25_JMEPF']=merge([{'--customise':'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomPFNanoAOD_Data','--data':''},steps['NANOUP15']])


as well as these 4 lines here, they are for 2016 condition

chayanit · 2020-11-01T13:32:03Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

+        if 'Nano' in step:
+            stepDict[stepName][k] = merge([{'--customise': 'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomAK8PFNanoAOD_MC'}, stepDict[step][k]])
+    def condition(self, fragment, stepList, key, hasHarvest):
+        return fragment=="TTbar_13" and ('2017' in key or '2018' in key)


This class is asking for 2017 and 2018 but why you need Run2? Shouldn't we target Run3?

chayanit · 2020-11-01T13:32:15Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

+        if 'Nano' in step:
+            stepDict[stepName][k] = merge([{'--customise': 'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomPFNanoAOD_MC'}, stepDict[step][k]])
+    def condition(self, fragment, stepList, key, hasHarvest):
+        return fragment=="TTbar_13" and ('2017' in key or '2018' in key)


and same here

rappoccio · 2020-11-02T15:31:45Z

Thanks @chayanit. This PR is still under discussion of the scope so we will hold the 2016/2017/2018 workflows for now.

silviodonato · 2020-11-09T17:52:55Z

@laurenhay @rappoccio shall we close this PR?

rappoccio · 2020-11-09T18:03:22Z

Hi, Silvio, That's fine. Sal

…

On Mon, Nov 9, 2020 at 12:53 PM Silvio Donato ***@***.***> wrote: @laurenhay <https://github.com/laurenhay> @rappoccio <https://github.com/rappoccio> shall we close this PR? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#31795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAR5OI3YYSL65OSHB46SBDSPAUANANCNFSM4SQ6IHEQ> .

kpedro88 · 2020-11-19T19:29:45Z

please close

laurenhay added 5 commits October 1, 2020 23:02

Adding PF candidates and additional BTV information to JME workflow.

2801813

Updating to 11_2_X syntax.

d736f42

Changing names of candidate collections for clarity.

0495536

Changing description of candidate tables for clarity.

1b813fb

Cand mapping names, descriptions updated. Code checks.

f7e6b07

cmsbuild added this to the CMSSW_11_2_X milestone Oct 14, 2020

cmsbuild added analysis-pending code-checks-pending comparison-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending xpog-pending labels Oct 14, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 14, 2020

laurenhay marked this pull request as ready for review October 14, 2020 18:25

laurenhay changed the base branch from master to CMSSW_11_2_X October 14, 2020 18:26

cmsbuild changed the base branch from CMSSW_11_2_X to master October 14, 2020 18:59

cmsbuild added xpog-rejected and removed xpog-pending labels Oct 14, 2020

This was referenced Oct 14, 2020

Add ParticleNetAK4 to JME custom NanoAOD #31714

Merged

Fix to allow concurrent processing of lumisections with ECAL DQM #31689

Merged

This was referenced Oct 17, 2020

Bad PF candidate filter for 2018 PbPb data #31668

Merged

Phase2-gex25 Add a new scenario 2026D74 with V14 HGCal and GE0 #31842

Merged

Patatrack integration - Pixel workflows (12/N) #31854

Merged

This was referenced Oct 20, 2020

fix stray Global Tags post GeometricDetExtra Migration #31871

Merged

Ecal phase2 new digitization 11 2 x #31726

Merged

This was referenced Oct 27, 2020

adding MWGR4 run with GEM in Run3 data workflow #31955

Merged

DD4hep RelVal for Phase2: update geometry configuration and PyReleaseValidation #31975

Merged

Add record type HeavyIonRPRcd to 2021 HI global tag #31957

Merged

cmsbuild mentioned this pull request Nov 1, 2020

[HGCal] TICL v3 major upgrade #31906

Merged

chayanit reviewed Nov 1, 2020

View reviewed changes

cmsbuild mentioned this pull request Nov 2, 2020

PuppiMET with PUPPIv15 and UL17 PUID #31927

Merged

cmsbuild mentioned this pull request Nov 6, 2020

Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983

Closed

This was referenced Nov 14, 2020

Update GEN-SIM string in runTheMatrix for 11_2_X #32140

Merged

Add DD4hep Run 3 workflows 11624.911 and 11642.91 to PR/IB tests #32096

Merged

MLPF: first integration of EDProducer and training code #32048

Merged

cmsbuild closed this Nov 19, 2020

mariadalfonso mentioned this pull request Nov 19, 2020

[RFC] Outline of HGCAL ML ntuples using NanoAOD #32187

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ability to output PF Candidate information in JME NTuples #31795

Adding ability to output PF Candidate information in JME NTuples #31795

laurenhay commented Oct 14, 2020 •

edited

Loading

cmsbuild commented Oct 14, 2020

cmsbuild commented Oct 14, 2020

cmsbuild commented Oct 14, 2020

mariadalfonso commented Oct 14, 2020 •

edited

Loading

cmsbuild commented Oct 14, 2020

laurenhay commented Oct 15, 2020

chayanit commented Oct 17, 2020

rappoccio commented Oct 19, 2020

chayanit commented Oct 20, 2020

silviodonato commented Oct 29, 2020

laurenhay commented Oct 29, 2020

chayanit Nov 1, 2020

chayanit Nov 1, 2020

chayanit Nov 1, 2020

chayanit Nov 1, 2020

rappoccio commented Nov 2, 2020

silviodonato commented Nov 9, 2020

rappoccio commented Nov 9, 2020 via email

kpedro88 commented Nov 19, 2020

Adding ability to output PF Candidate information in JME NTuples #31795

Adding ability to output PF Candidate information in JME NTuples #31795

Conversation

laurenhay commented Oct 14, 2020 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Oct 14, 2020

cmsbuild commented Oct 14, 2020

cmsbuild commented Oct 14, 2020

mariadalfonso commented Oct 14, 2020 • edited Loading

cmsbuild commented Oct 14, 2020

laurenhay commented Oct 15, 2020

chayanit commented Oct 17, 2020

rappoccio commented Oct 19, 2020

chayanit commented Oct 20, 2020

silviodonato commented Oct 29, 2020

laurenhay commented Oct 29, 2020

chayanit Nov 1, 2020

Choose a reason for hiding this comment

chayanit Nov 1, 2020

Choose a reason for hiding this comment

chayanit Nov 1, 2020

Choose a reason for hiding this comment

chayanit Nov 1, 2020

Choose a reason for hiding this comment

rappoccio commented Nov 2, 2020

silviodonato commented Nov 9, 2020

rappoccio commented Nov 9, 2020 via email

kpedro88 commented Nov 19, 2020

laurenhay commented Oct 14, 2020 •

edited

Loading

mariadalfonso commented Oct 14, 2020 •

edited

Loading