[12.2.X] Refactoring gen weight storage in EDM + Nano integration #38699

sroychow · 2022-07-12T08:54:20Z

PR description:

Backport of #32167

overhaul of code: now works with tinyxml2 instead of regex

Dylan's improvements of weight parsing changed LHEsource so it compiles, running tests now added scientific notation for numbers so values are correct added a few more test statements. Need to think of future needs Add testing suite (will remove/squash commits right before merge) Change pdfSetsInfo to be read in from LHAPDF path Still lingering question of how to decide pdf type This can be resolved in the new function "setupPdfSetsInfo" Add check if orphan weight is pdf and fix xml tag swap (needs review) Allow reading '>' as '>' and fix bug with first weight w/o group As for the first weight error: if the first weight doesn't have a weight group, the code would not add it to the list of weights because of currentGroupName and the first weight both having a trivial "" as their name. Thats the reason for the currentGroupName being set to "None" for initialization Fix space issue in regex and moved unchaning variables outside for Add pdfgroup splitting. To do this, needed LHAPDF requirements clang format Fix parsing issue with pdfweights in certain files those files being "DrellYan_LO_MGMLMv233_2016_weightInfo.txt" and "DrellYan_NLO_MGFXFXv242_2017_weightInfo.txt", Remove PdfSetInfo in favor of LHAPDF library Refactor pdfweight so pdfweights are always split May put flags needed for the splitting. Only impliemented in LHEWeightHelper Simplify ScaleWeight structure in anticipation of dyn weight changes Add dynWeights to scaleWeightInfo. Can improve code (basic refactor) Add orphaned pdf weight into scale weight missing central index not my finest work. Definitely could have improvements, but technically works Add Pdf info to pdfWeights (size, type) & check of size consistency More testing scripts, better error handling Fix orphaned central scale problem. was resizing to 1 (not great) Simplify logic of adding to make less opaque Move weight building logic to WeightHelper so GEN can use it Add basic GenLevel parsing. Works on ZGTo2NuG test file Add PartonShower parsing for GenWeights Change PartonShower filename to match h file More error handling, parsing improvements Add option for raising expection if XML is not valid Fix parsing of PartonShower name in GenWeights Add basic MEParam weight functionality Allow for general adding of params to vector Also fixed formating with clang Simplifications for PS and ME weight groups

adding option to save preferred PS weights Support unassociated LHE weights, simplify PS weights Refactor ScaleWeightGroup to simplify codebase Remove/change lines in WeightHelper to clean Fix naming problems in ScaleWeight Refactor LHWeightHelper (moved to helper functions) Fix errors in ScaleWeight and add print statements for debug

Fix off-by-one error in pdf weight splitting Templatize product adder in WeightHelper Expand emission functionality to PSWeight needs to add more to make sure works correctly clang-format the partonshowerWeightGroupInfo.cc file as well Fix central weight and error from mg26xNew Fix PSWeights in LumiHeader to work w diff pythia format

Further cleanup, nano improvements change way PS weights handled Allow for finding missing weights Fix error with gap fixing code Change around logic to have bool to decide if use guess for PS Amend wellformed logic for scaleweight Cleanup code Nano back to GenWeightsTableProducer file

fix runtime error failIfInvalidXML_ cleanup failIfInvalidXML_ cleanup (2) scram b code-format

Fix problem of confounding issues in xmlParsing code fails to parse correctly when swapped weigthgroup tags and escaped </> characters in header. Also clean up the xmlError parsing and add some debug statements associated with tose errors falling back to vector of flattables Add "fillEmptyIfWeightFails" option and fix an err Error is related to the product size not being updated when an unknown weight is found Add more error states to LHEWEightHelper to handle empty xml

Remove redundant files (renamed or dropped) revert unneeded change Produce new LHEEventProduct with weights zeroed Can also be done for GenEventInfoProduct, but here the number of floats duplicated is way less severe, and the amount of downstream code accessing it is way larger Code format

Code format

Fix case where weight lumi product doesn't exist but event weight does Remove allowUnassociated for LHE, Code format

Treat unassociated weights properly Code checks

code format

cmsbuild · 2022-07-12T08:54:45Z

A new Pull Request was created by @sroychow (Suvankar Roy Chowdhury) for CMSSW_12_2_X.

It involves the following packages:

Configuration/ProcessModifiers (operations)
Configuration/StandardSequences (operations)
DataFormats/NanoAOD (xpog)
GeneratorInterface/Configuration (generators)
GeneratorInterface/Core (generators)
GeneratorInterface/LHEInterface (generators)
PhysicsTools/NanoAOD (xpog)
PhysicsTools/PatAlgos (reconstruction)
SimDataFormats/GeneratorProducts (generators)

@SiewYan, @perrotta, @rappoccio, @gouskos, @mkirsano, @clacaputo, @fabiocos, @cmsbuild, @alberto-sanchez, @fgolf, @Saptaparna, @jpata, @mariadalfonso, @GurpreetSinghChahal, @qliphy, @davidlange6 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @felicepantaleo, @hatakeyamak, @emilbols, @Martin-Grunewald, @bsunanda, @alberto-sanchez, @ahinzmann, @demuller, @mmusich, @seemasharmafnal, @mmarionncern, @apsallid, @makortel, @JanFSchulte, @mbluj, @dgulhan, @jdolen, @azotz, @slomeo, @GiacomoSguazzoni, @rovere, @VinInn, @jdamgov, @missirol, @nhanvtran, @gkasieczka, @schoef, @ebrondol, @mtosi, @fabiocos, @AlexDeMoor, @mkirsano, @swertz, @JyothsnaKomaragiri, @lecriste, @gpetruc, @mariadalfonso, @andrzejnovak this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Refactoring gen weight storage in EDM + Nano integration #32167

sroychow · 2022-07-12T08:55:24Z

please test

perrotta · 2022-07-12T09:46:49Z

backport of #32167

cmsbuild · 2022-07-12T09:50:37Z

-1

Failed Tests: Build ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-63c31d/26159/summary.html
COMMIT: a118207
CMSSW: CMSSW_12_2_X_2022-07-10-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38699/26159/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation error when building:

>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NPUTablesProducer.cc
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NanoAODBaseCrossCleaner.cc
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NanoAODDQM.cc
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NanoAODOutputModule.cc
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NanoAODSimpleCrossCleaner.cc
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NanoAODOutputModule.cc:51:10: fatal error: PhysicsTools/NanoAOD/plugins/LumiOutputBranches.h: No such file or directory
   51 | #include "PhysicsTools/NanoAOD/plugins/LumiOutputBranches.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NativeArrayTableProducer.cc
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/PhysicsTools/NanoAOD/plugins/NanoAODOutputModule.cc:51:10: fatal error: PhysicsTools/NanoAOD/plugins/LumiOutputBranches.h: No such file or directory

Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package GeneratorInterface/LHEInterface
>> Entering Package PhysicsTools/NanoAOD
>> Entering Package PhysicsTools/PatAlgos
>> Entering Package SimDataFormats/GeneratorProducts
>> Compile sequence completed for CMSSW CMSSW_12_2_X_2022-07-10-0000
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
+ eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/tmp/slc7_amd64_gcc900/cache/log/src '||' 'true)'
++ scram build outputlog
>> Entering Package DataFormats/NanoAOD
Entering library rule at DataFormats/NanoAOD
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2022-07-10-0000/src/DataFormats/NanoAOD/src/FlatTable.cc

sroychow · 2022-07-12T13:56:14Z

The file in question was introduced via #34169 . Discussing with @kdlong, we think is useful to backport the changes for lumi branch. Is it okay to be backported through this PR or we need to create another backport for 34169?

perrotta · 2022-07-12T14:03:37Z

The file in question was introduced via #34169 . Discussing with @kdlong, we think is useful to backport the changes for lumi branch. Is it okay to be backported through this PR or we need to create another backport for 34169?

"The file in question" refers to which conversation? I don't see "files" mentioned so far in this thread
In any case, if you plan to backport them both it would be probably easier to make a single backport PR, because the two are interefering in a couplke of files, and they should be merged separately and after rebase if submitted separately

sroychow · 2022-07-12T14:07:02Z

hi @perrotta I meant the file PhysicsTools/NanoAOD/plugins/LumiOutputBranches.h.

Okay I shall proceed with backporting the other PR as well.

perrotta · 2022-08-23T07:38:29Z

As anticipated already several times during the ORP meetings, the 12_2_X and 12_3_X queues must be considered as closed. We consequently close this longstanding and apparently unattended PR, in order to clean up the queue. Please feel free to reopen it (with a valid justification) if you really think that it should be still merged, instead.

kdlong · 2022-08-23T07:54:35Z

This was back ported in anticipation of a test campaign for Nano v10, which was cancelled. So it's ok to close it, but we would still really like to have a similar test in another release if possible (@mariadalfonso )

kdlong and others added 23 commits July 11, 2022 15:00

First implementation of WeightGroupInfo/product classes

4e90d7e

Working on parsing integration with scale/pdf weights

a019e8a

overhaul of code: now works with tinyxml2 instead of regex

adding altset index table

b13136f

attempt to delete extra kets after the last </weightgroup>

37e4dad

fix runtime error failIfInvalidXML_ cleanup failIfInvalidXML_ cleanup (2) scram b code-format

Allow gen products to run at GEN step or Nano

d27b812

Support ignoregroups in nano, fixes for unassociated weights

bc7f352

Convert OwnVector to unique_ptr

543c4ba

Simplify nano producer and weight parsing

56c6a7e

Make producers edm::Global

78aaa8b

Update nano and nanogen configs

871b024

Fix scale weights in case of < 9 entries

15b0b4e

Code format

8689e3b

Code format

Hopefully fixing test errors

c8a9816

Fix case where weight lumi product doesn't exist but event weight does Remove allowUnassociated for LHE, Code format

Attempt to fit cosmics workflow error

cd7c306

Treat unassociated weights properly Code checks

mask genweight addition in procmodifier

5eed38e

macro to permit catch statement with LHAPDF

9e8ec88

code format

Fix event content for gen weights in AOD

a118207

cmsbuild added this to the CMSSW_12_2_X milestone Jul 12, 2022

cmsbuild added generators-pending operations-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 12, 2022

cmsbuild added the xpog-pending label Jul 12, 2022

cmsbuild added tests-started and removed tests-pending labels Jul 12, 2022

cmsbuild added backport tests-rejected and removed tests-started labels Jul 12, 2022

perrotta closed this Aug 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[12.2.X] Refactoring gen weight storage in EDM + Nano integration #38699

[12.2.X] Refactoring gen weight storage in EDM + Nano integration #38699

sroychow commented Jul 12, 2022

cmsbuild commented Jul 12, 2022 •

edited

Loading

sroychow commented Jul 12, 2022

perrotta commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

sroychow commented Jul 12, 2022

perrotta commented Jul 12, 2022

sroychow commented Jul 12, 2022

perrotta commented Aug 23, 2022

kdlong commented Aug 23, 2022

[12.2.X] Refactoring gen weight storage in EDM + Nano integration #38699

[12.2.X] Refactoring gen weight storage in EDM + Nano integration #38699

Conversation

sroychow commented Jul 12, 2022

PR description:

cmsbuild commented Jul 12, 2022 • edited Loading

sroychow commented Jul 12, 2022

perrotta commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

Build

Clang Build

sroychow commented Jul 12, 2022

perrotta commented Jul 12, 2022

sroychow commented Jul 12, 2022

perrotta commented Aug 23, 2022

kdlong commented Aug 23, 2022

cmsbuild commented Jul 12, 2022 •

edited

Loading