Add ParticleNet mass regression

[hqucms]

Requires: cms-data/RecoBTag-Combined#44 (DNN model)

PR description:

The ParticleNet mass regression is a new algorithm that uses PF candidates and the ParticleNet architecture to improve the AK8 jet mass resolution, particularly for 2-prong jets (X->bb/X->cc/X->qq). The model is trained with the flat-m(H) HH4Q samples and QCD samples (same as used in the ParticleNet-MD tagger), and uses the same ParticleNet architecture. The training target is the generated particle mass for the Higgs jets, and gen-jet soft drop mass for the QCD jets. The resulting regressed mass shows a significant improvement in the mass resolution compared to the soft drop mass and also reduces the tails at low/high SD mass values. This new algorithm is being used by the full Run2 VHcc merged-jet topology analysis and the VBF HH4b boosted analysis and brings significant gain in sensitivity. More details of the algorithm can be found in the talks at ML Forum and JMAR.

This algorithm is requested to be included in NanoAODv9 by a few analyses. Therefore, a new training using the UL samples is performed and used in this PR. The performance of the UL training is summarized in this presentation.

PR validation:

Validated with a UL NanoAOD workflow.

Memory/time increase should be the same as pfMassDecorrelatedParticleNetJetTags as the network architecture is the same, i.e., ~20M and ~30ms per high pT AK8 jet.

@selvaggi @gouskos @pmaksim1

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33483/22207

• This PR adds an extra 52KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File PhysicsTools/NanoAOD/python/nano_cff.py modified in PR(s): Refactoring gen weight storage in EDM + Nano integration #32167

[cmsbuild]

A new Pull Request was created by @hqucms (Huilin Qu) for master.

It involves the following packages:

PhysicsTools/NanoAOD
PhysicsTools/PatAlgos
RecoBTag/ONNXRuntime

@perrotta, @gouskos, @cmsbuild, @fgolf, @slava77, @jpata, @mariadalfonso can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @jdolen, @swertz, @JyothsnaKomaragiri, @ahinzmann, @schoef, @emilbols, @swozniewski, @jdamgov, @mbluj, @nhanvtran, @gkasieczka, @clelange, @hatakeyamak, @ferencek, @gpetruc, @andrzejnovak, @mariadalfonso, @seemasharmafnal, @mmarionncern this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[mariadalfonso]

please test

[hqucms]

please test

@mariadalfonso This needs to be tested together with cms-data/RecoBTag-Combined#44

[mariadalfonso]

please abort

[mariadalfonso]

test parameters:

pull_request = cms-data/RecoBTag-Combined#44

[mariadalfonso]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1c4177/14377/summary.html
COMMIT: 52973d8
CMSSW: CMSSW_12_0_X_2021-04-20-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33483/14377/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 3632 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2877046
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2877023
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.292 KiB( 37 files compared)
• DQMHistoSizes: changed ( 1325.81,... ): 0.146 KiB Physics/NanoAODDQM
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[slava77]

For the memory, it's actually ~20M:

How does this part scale with the number of threads?
It looks like 15MB are per thread.

How does this compare to the other taggers?

was there some review in the ML group(s) done to confirm that this can not be somewhat safely reduced?

[slava77]

we have now pfMassDecorrelatedParticleNetJetTags and pfParticleNetMassRegressionJetTags
as "modifiers" of pfParticleNetJetTags.
There is clearly some inconsistency in naming (prefix vs postfix); sticking to the existing pattern is often a way to go.
This better be resolved, unless there is a good justification.

[hqucms]

For the memory, it's actually ~20M:

How does this part scale with the number of threads?
It looks like 15MB are per thread.

How does this compare to the other taggers?

was there some review in the ML group(s) done to confirm that this can not be somewhat safely reduced?

@slava77 The memory usage does not scale with the number of threads as the ONNXRuntime session is shared by all threads.

[hqucms]

we have now pfMassDecorrelatedParticleNetJetTags and pfParticleNetMassRegressionJetTags
as "modifiers" of pfParticleNetJetTags.
There is clearly some inconsistency in naming (prefix vs postfix); sticking to the existing pattern is often a way to go.
This better be resolved, unless there is a good justification.

pfMassDecorrelatedParticleNetJetTags is a jet tagger (i.e., a classifier) while pfParticleNetMassRegressionJetTags is a mass regression algorithm, so I think it's better to make a distinction in the names. In fact, if it's not that the btagging code assumes the name ends with JetTags I would remove that in pfParticleNetMassRegressionJetTags to be more clear.

[slava77]

we have now pfMassDecorrelatedParticleNetJetTags and pfParticleNetMassRegressionJetTags
as "modifiers" of pfParticleNetJetTags.
There is clearly some inconsistency in naming (prefix vs postfix); sticking to the existing pattern is often a way to go.
This better be resolved, unless there is a good justification.

pfMassDecorrelatedParticleNetJetTags is a jet tagger (i.e., a classifier) while pfParticleNetMassRegressionJetTags is a mass regression algorithm, so I think it's better to make a distinction in the names. In fact, if it's not that the btagging code assumes the name ends with JetTags I would remove that in pfParticleNetMassRegressionJetTags to be more clear.

I see, thanks for clarifying that this is semantically different.
Please keep JetTags at the end, as this still follows the TagInfos->JetTags sequence.
So, no action necessary in the code.

On a remotely related subject, is it eventually possible for a single model to compute the mass as well as the probabilities?

[hqucms]

On a remotely related subject, is it eventually possible for a single model to compute the mass as well as the probabilities?

Yes that's an interesting direction to go and we plan to look into that for the next iteration.

[hqucms]

Any comments @mariadalfonso @gouskos ?

[jpata]

Per discussions with Slava, to prevent issues with relvals from externals, I removed the reco signature until cms-sw/cmsdist#6844 is merged.

[jpata]

+reconstruction

• new ParticleNet mass regression in MiniAOD and NanoAOD
• no unexpected reco differences, we observe a new discriminator in the list
• computing performance is under control: Add ParticleNet mass regression #33483 (comment)
• requires Add AK8 ParticleNet mass regression model cms-data/RecoBTag-Combined#44 and Update tag for RecoBTag-Combined to V01-11-00 cmsdist#6844 (both are merged)

[silviodonato]

kind reminder @cms-sw/xpog-l2

[gouskos]

kind reminder @cms-sw/xpog-l2
@silviodonato Soon we will be done with the nanoAOD tests

[mariadalfonso]

so this nanoAOD_addParticleNetMass_switch always True here means that we will re-run always the regression and it's expensive (about 10% of the whole nanoAOD production).

line 347 should be set to False as default
then activate for the past (
i.e. nanoAOD_addParticleNetMass_switch = cms.untracked.bool(True), in line 360 and 367)

and then finally add a block
run2_nanoAOD_106Xv2.toModify(
nanoAOD_addDeepInfoAK8_switch,
nanoAOD_addParticleNetMass_switch = True
)

[hqucms]

@mariadalfonso I believe we need to re-run it until it becomes available in MiniAOD, so setting False as the default does not seem a future-proof way to go.

[mariadalfonso]

the miniAOD as the come out from this master (either Run2 or Run3): will contain or not the particleNetMass ?

[hqucms]

Yes it will contain. I guess maybe I am a bit confused with the many modifiers in NanoAOD. Specifically, does the modifier run2_nanoAOD_106Xv2 correspond specifically to NanoAOD V9, or generally any Nano (V10, V11...) as long as it runs over 106X MiniAODv2? If the former, then it means we will need to modify the code to enable re-calculating the mass regression every time a new NanoAOD version is introduced (as long as we are still running on 106X MiniAODv2), right? If the latter then I agree False is a better default.

[mariadalfonso]

Nano modifiers are used to identify the mini in input.

So for the UL we have in production:
MiniAODv1 --> these have been used as input for the nanov8 and so all recipes need to be called with run2_nanoAOD_106Xv1 --> when nano is being produced one call it v8
MiniAODv2 were started in December '20 -->nano recipes will need to be called with the run2_nanoAOD_106Xv2 --> these nano will be also called v9

if NanoV10 will read MiniAODv2 or a newer mini we do not know.

Master anyway make fresh mini with the particleNetMass included, so we can avoid to recalculate.

I hope it's clearer.

Said that, we will need to do a major cleanup soon to prepare Run3 config, so we can restore the False as default at that time. I leave to you.

[hqucms]

Thanks a lot for the clarification @mariadalfonso !

If you are fine then I would prefer to postpone the change to False for the future cleanup.

[mariadalfonso]

ok, let's do like this

[mariadalfonso]

+xpog

• add one the mass regression variable to complement the ParticleNet classifiers.
• 1 training is used all the nano eras and years

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1