Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GloParT inference utility #45830

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

colizz
Copy link
Contributor

@colizz colizz commented Aug 28, 2024

PR description:

This PR adds the GloParT 3 model's inference utility into CMSSW, prepared for NanoAODv15. The topic was discussed in the recent JMAR meeting [slides]

Global Particle Transformer (GloParT) 3 is an inclusive tagging model for AK8 jets that covers the entire phase space and enables resonance mass regression for each jet class. It functions as both a global tagger and a mass regression model for AK8 jets and can also be utilized as a pre-trained model. The hidden layer neurons (with dimension 256) are stored in MiniAOD, providing the capability to resume all output scores. Further details can be found in the slides.

In summary, the AK8 jet features added to MiniAOD and NanoAOD are illustrated below.

glopart-integration

Please test this PR with cms-data/RecoBTag-Combined#61.

Note: the current model in cms-data/RecoBTag-Combined#61 is a placeholder for incorporating this PR. During our development history, GloParT 2 has already been validated to function properly within CMSSW (using this branch: https://github.com/colizz/cmssw/tree/dev-130X-glopart ), although it has not been officially integrated. We are now upgrading GloParT from version 2 to version 3 (details in slides), and the final GloParT 3 model will be ready within 2-3 weeks. We are suggested to make this infrastructure PR to cmssw first.

Adding @qliphy @cmantill @rkansal47 @jmduarte

PR validation:

The PR passes the unit test test_globalpart_cfg.py. It is also validated under the workflow AODSIM->MINIAOD->NANOAOD.

@cmsbuild
Copy link
Contributor

cmsbuild commented Aug 28, 2024

cms-bot internal usage

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @colizz for master.

It involves the following packages:

  • PhysicsTools/NanoAOD (xpog)
  • PhysicsTools/PatAlgos (reconstruction, xpog)
  • RecoBTag/Configuration (reconstruction)
  • RecoBTag/FeatureTools (reconstruction)
  • RecoBTag/ONNXRuntime (reconstruction)

@cmsbuild, @ftorrresd, @hqucms, @jfernan2, @mandrenguyen, @vlimant can you please review it and eventually sign? Thanks.
@AlexDeMoor, @AnnikaStein, @Ming-Yan, @Senphy, @ahinzmann, @andrzejnovak, @azotz, @castaned, @demuller, @gkasieczka, @gouskos, @gpetruc, @hatakeyamak, @hqucms, @jdamgov, @jdolen, @mariadalfonso, @mbluj, @missirol, @mmarionncern, @nhanvtran, @rappoccio, @schoef, @seemasharmafnal this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@hqucms
Copy link
Contributor

hqucms commented Aug 29, 2024

enable nano

@hqucms
Copy link
Contributor

hqucms commented Aug 29, 2024

test parameters:

@hqucms
Copy link
Contributor

hqucms commented Aug 29, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 60KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41191/summary.html
COMMIT: f185f0c
CMSSW: CMSSW_14_2_X_2024-08-28-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/45830/41191/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41191/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41191/git-merge-result

Comparison Summary

Summary:

  • You potentially added 12 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 2856 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3328202
  • DQMHistoTests: Total failures: 152
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3328030
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 6411.173999999998 KiB( 43 files compared)
  • DQMHistoSizes: changed ( 10224.0,... ): 305.294 KiB L1T/L1TCaloLayer1Summary
  • Checked 193 log files, 163 edm output root files, 44 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • You potentially added 715 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 937 differences found in the comparisons
  • DQMHistoTests: Total files compared: 21
  • DQMHistoTests: Total histograms compared: 54921
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 54921
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 20 files compared)
  • Checked 102 log files, 58 edm output root files, 21 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001 2.800 2.788 0.011 ( +0.4% ) 3.31 3.27 +1.0% 6.057 6.097
2500.002 2.912 2.901 0.011 ( +0.4% ) 2.91 2.92 -0.4% 6.400 6.425
2500.003 2.859 2.848 0.011 ( +0.4% ) 3.06 3.03 +0.8% 6.382 6.393
2500.011 1.457 1.450 0.007 ( +0.5% ) 5.70 5.59 +1.9% 2.431 2.427
2500.012 1.917 1.909 0.007 ( +0.4% ) 3.12 3.10 +0.7% 2.605 2.607
2500.013 1.774 1.765 0.010 ( +0.5% ) 4.54 4.43 +2.4% 2.531 2.409
2500.021 0.022 0.022 0.000 ( +0.0% ) 0.94 0.90 +3.8% 2.254 2.222
2500.022 0.022 0.022 0.000 ( +0.0% ) 0.89 0.87 +2.3% 2.254 2.220
2500.023 0.022 0.022 0.000 ( +0.0% ) 0.90 0.89 +1.2% 2.212 2.167
2500.024 0.022 0.022 0.000 ( +0.0% ) 0.68 0.67 +0.6% 2.367 2.338
2500.031 0.035 0.035 -0.000 ( -0.1% ) 0.84 0.83 +0.5% 2.351 2.306
2500.032 0.036 0.036 0.000 ( +0.0% ) 0.86 0.85 +1.0% 2.288 2.245
2500.033 0.037 0.037 0.000 ( +0.0% ) 0.77 0.76 +1.2% 2.394 2.350
2500.034 0.036 0.036 0.000 ( +0.0% ) 0.76 0.77 -1.4% 2.374 2.344
2500.101 2.657 2.646 0.011 ( +0.4% ) 8.83 8.54 +3.4% 6.910 6.295
2500.111 1.336 1.330 0.006 ( +0.5% ) 20.15 19.17 +5.1% 2.242 2.161
2500.112 1.745 1.735 0.010 ( +0.6% ) 14.28 14.61 -2.3% 2.171 2.107
2500.131 5.194 5.194 0.000 ( +0.0% ) 15.22 15.06 +1.0% 1.470 1.456
2500.201 2.488 2.478 0.010 ( +0.4% ) 7.52 7.27 +3.3% 5.534 5.502
2500.211 1.600 1.592 0.008 ( +0.5% ) 16.56 16.32 +1.5% 2.113 2.087
2500.212 2.044 2.033 0.010 ( +0.5% ) 13.53 13.10 +3.3% 2.160 2.125
2500.221 2.006 2.006 0.000 ( +0.0% ) 7.60 7.46 +1.9% 2.311 2.258
2500.222 3.218 3.206 0.012 ( +0.4% ) 6.90 7.03 -1.9% 2.360 2.310
2500.223 8.900 8.888 0.012 ( +0.1% ) 2.73 2.63 +3.8% 2.378 2.302
2500.224 5.527 5.515 0.012 ( +0.2% ) 1.09 1.07 +1.9% 2.564 2.130
2500.225 5.545 5.533 0.012 ( +0.2% ) 0.99 0.97 +2.6% 2.329 1.977
2500.226 2.985 2.973 0.012 ( +0.4% ) 7.33 7.18 +2.1% 2.322 2.315
2500.227 8.972 8.972 0.000 ( +0.0% ) 10.00 9.71 +3.0% 1.420 1.412
2500.231 1.407 1.407 0.000 ( +0.0% ) 13.22 13.02 +1.5% 1.968 1.971
2500.232 2.247 2.237 0.010 ( +0.5% ) 12.97 12.81 +1.3% 2.058 1.812
2500.233 4.681 4.670 0.010 ( +0.2% ) 4.61 4.50 +2.5% 2.066 1.860
2500.234 3.317 3.307 0.010 ( +0.3% ) 1.46 1.44 +1.6% 1.856 2.071
2500.235 3.328 3.317 0.010 ( +0.3% ) 1.38 1.35 +1.9% 1.839 2.099
2500.236 2.095 2.085 0.010 ( +0.5% ) 13.10 13.05 +0.4% 2.043 2.043
2500.237 7.977 7.977 0.000 ( +0.0% ) 14.41 13.94 +3.3% 1.414 1.418
2500.241 9.405 9.405 0.000 ( +0.0% ) 3.80 3.66 +3.7% 1.903 1.782
2500.242 10.331 10.331 0.000 ( +0.0% ) 0.87 0.86 +2.0% 1.748 1.699
2500.243 2.712 2.712 0.000 ( +0.0% ) 7.85 7.01 +12.0% 1.086 1.084
2500.244 485.976 485.976 0.000 ( +0.0% ) 0.55 0.54 +2.5% 1.638 1.627
2500.245 823.224 823.224 0.000 ( +0.0% ) 0.73 0.71 +2.6% 1.579 1.549
2500.901 1.749 1.749 0.000 ( +0.0% ) 21.12 20.16 +4.8% 1.464 1.817
2500.902 1.598 1.598 0.000 ( +0.0% ) 20.88 21.03 -0.7% 1.783 1.777
2500.911 13.931 13.931 0.000 ( +0.0% ) 2.84 2.48 +14.9% 1.102 1.100
2500.912 0.310 0.199 0.112 ( +56.2% ) 1.12 1.09 +3.0% 0.989 0.996
2500.913 0.110 0.110 0.000 ( +0.0% ) 1.22 1.16 +4.4% 0.996 0.989

@colizz
Copy link
Contributor Author

colizz commented Aug 30, 2024

Hi all, please see below a security check on the inferred tagger scores (using the GloParT 2 model). They are consistent with the score predicted by Weaver.
glopart2-check

@jfernan2
Copy link
Contributor

jfernan2 commented Sep 2, 2024

enable profiling

@jfernan2
Copy link
Contributor

jfernan2 commented Sep 2, 2024

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 2, 2024

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41217/summary.html
COMMIT: f185f0c
CMSSW: CMSSW_14_2_X_2024-09-01-2300/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45830/41217/install.sh to create a dev area with all the needed externals and cmssw changes.

  • DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially removed 164 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 2850 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3328315
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3328295
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
  • Checked 193 log files, 163 edm output root files, 44 DQM output files
  • TriggerResults: no differences found

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Pull request #45830 was updated. @cmsbuild, @ftorrresd, @hqucms, @jfernan2, @mandrenguyen can you please check and sign again.

@colizz
Copy link
Contributor Author

colizz commented Sep 16, 2024

Hi @jfernan2 @tvami, many thanks! This PR requires us to finalize the upload of the model in cms-data/RecoBTag-Combined#61.

XPOG has suggested that we upload our current "beta1" model (instead of a dummy model) so that performance can be tested. We've now done this and updated both PRs accordingly. Thanks very much for your test!

@hqucms
Copy link
Contributor

hqucms commented Sep 16, 2024

please abort

@hqucms
Copy link
Contributor

hqucms commented Sep 16, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41545/summary.html
COMMIT: 867be77
CMSSW: CMSSW_14_2_X_2024-09-16-1100/el8_amd64_gcc12
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45830/41545/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41545/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41545/git-merge-result

Comparison Summary

Summary:

  • You potentially added 1 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 2932 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3331158
  • DQMHistoTests: Total failures: 2853
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3328285
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
  • Checked 193 log files, 163 edm output root files, 44 DQM output files
  • TriggerResults: no differences found

@hqucms
Copy link
Contributor

hqucms commented Sep 17, 2024

enable profiling,nano

@hqucms
Copy link
Contributor

hqucms commented Sep 17, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41571/summary.html
COMMIT: 867be77
CMSSW: CMSSW_14_2_X_2024-09-17-1100/el8_amd64_gcc12
Additional Tests: PROFILING,NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45830/41571/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41571/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-560df3/41571/git-merge-result

Comparison Summary

Summary:

  • You potentially added 57 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 26611 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3331158
  • DQMHistoTests: Total failures: 48956
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3282182
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -1.304 KiB( 43 files compared)
  • DQMHistoSizes: changed ( 250202.181 ): 0.182 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 25202.0 ): -0.127 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 7.3 ): -0.120 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 8.0 ): -1.239 KiB SiStrip/MechanicalView
  • Checked 193 log files, 163 edm output root files, 44 DQM output files
  • TriggerResults: found differences in 6 / 42 workflows

NANO Comparison Summary

Summary:

  • You potentially added 36 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 952 differences found in the comparisons
  • DQMHistoTests: Total files compared: 21
  • DQMHistoTests: Total histograms compared: 55132
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 55132
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 20 files compared)
  • Checked 102 log files, 58 edm output root files, 21 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001 2.850 2.839 0.011 ( +0.4% ) 3.35 3.30 +1.5% 6.084 6.214
2500.002 2.966 2.954 0.012 ( +0.4% ) 2.98 2.96 +0.5% 6.457 6.441
2500.003 2.911 2.900 0.011 ( +0.4% ) 3.10 3.08 +0.8% 6.411 6.377
2500.011 1.497 1.489 0.008 ( +0.5% ) 5.79 5.78 +0.1% 2.339 2.167
2500.012 1.967 1.958 0.009 ( +0.5% ) 3.15 3.17 -0.6% 2.606 2.185
2500.013 1.817 1.809 0.008 ( +0.4% ) 4.58 4.56 +0.2% 2.431 2.208
2500.021 0.022 0.022 0.000 ( +0.0% ) 0.94 0.93 +1.3% 2.365 2.387
2500.022 0.022 0.022 0.000 ( +0.0% ) 0.89 0.93 -3.3% 2.383 2.380
2500.023 0.022 0.022 0.000 ( +0.0% ) 0.91 0.92 -1.7% 2.327 2.246
2500.024 0.022 0.022 0.000 ( +0.0% ) 0.70 0.69 +1.9% 2.474 2.410
2500.031 0.035 0.035 0.000 ( +0.0% ) 0.83 0.89 -6.5% 2.469 2.445
2500.032 0.036 0.036 0.000 ( +0.0% ) 0.84 0.91 -6.9% 2.407 2.403
2500.033 0.037 0.037 0.000 ( +0.1% ) 0.77 0.81 -5.7% 2.508 2.490
2500.034 0.036 0.036 0.000 ( +0.0% ) 0.78 0.79 -2.0% 2.483 2.456
2500.101 2.670 2.660 0.010 ( +0.4% ) 9.04 8.79 +2.9% 6.313 6.325
2500.111 1.349 1.342 0.007 ( +0.5% ) 20.33 19.80 +2.7% 2.243 2.235
2500.112 1.760 1.749 0.010 ( +0.6% ) 15.41 15.66 -1.6% 2.156 2.308
2500.131 0.747 0.747 0.000 ( +0.0% ) 18.51 18.51 -0.0% 1.390 1.412
2500.201 2.501 2.490 0.011 ( +0.4% ) 7.67 7.73 -0.7% 5.553 5.611
2500.211 1.653 1.612 0.041 ( +2.5% ) 17.61 18.23 -3.4% 2.101 1.989
2500.212 2.065 2.055 0.010 ( +0.5% ) 14.03 14.23 -1.4% 2.165 2.081
2500.221 2.003 2.003 0.000 ( +0.0% ) 7.83 7.84 -0.2% 2.306 2.041
2500.222 3.234 3.222 0.012 ( +0.4% ) 7.30 7.65 -4.6% 2.365 2.094
2500.223 8.914 8.902 0.012 ( +0.1% ) 2.79 2.84 -1.6% 2.023 2.050
2500.224 5.582 5.570 0.012 ( +0.2% ) 1.11 1.10 +0.7% 2.081 2.268
2500.225 5.602 5.590 0.012 ( +0.2% ) 1.01 1.03 -1.7% 1.844 2.270
2500.226 3.000 2.988 0.012 ( +0.4% ) 7.38 7.79 -5.2% 2.074 2.528
2500.227 1.437 1.437 0.000 ( +0.0% ) 11.84 11.96 -1.0% 1.252 1.420
2500.231 1.403 1.403 0.000 ( +0.0% ) 13.60 14.65 -7.2% 1.967 2.175
2500.232 2.270 2.260 0.010 ( +0.5% ) 13.62 14.19 -4.0% 2.017 2.273
2500.233 4.702 4.692 0.010 ( +0.2% ) 4.77 4.98 -4.2% 2.038 2.019
2500.234 3.358 3.348 0.010 ( +0.3% ) 1.52 1.52 -0.4% 1.808 2.068
2500.235 3.369 3.359 0.010 ( +0.3% ) 1.42 1.40 +1.4% 1.843 1.716
2500.236 2.117 2.107 0.010 ( +0.5% ) 13.83 14.56 -5.1% 1.774 2.260
2500.237 1.016 1.016 0.000 ( +0.0% ) 17.16 17.59 -2.4% 1.209 1.423
2500.241 9.404 9.404 0.000 ( +0.0% ) 3.79 3.81 -0.5% 1.576 1.945
2500.242 10.331 10.331 0.000 ( +0.0% ) 0.93 0.88 +4.7% 1.661 1.734
2500.243 2.712 2.712 0.000 ( +0.0% ) 8.65 8.54 +1.4% 1.083 1.079
2500.244 485.976 485.976 0.000 ( +0.0% ) 0.58 0.57 +0.4% 1.707 1.683
2500.245 823.202 823.202 0.000 ( +0.0% ) 0.76 0.75 +1.3% 1.689 1.657
2500.901 1.777 1.777 0.000 ( +0.0% ) 21.86 21.59 +1.3% 1.783 1.840
2500.902 1.626 1.626 0.000 ( +0.0% ) 21.48 20.87 +2.9% 1.746 1.771
2500.911 13.995 13.995 0.000 ( +0.0% ) 4.29 1.77 +142.4% 1.106 1.094
2500.912 0.199 0.199 0.000 ( +0.0% ) 1.22 1.31 -6.5% 0.986 0.977
2500.913 0.110 0.110 0.000 ( +0.0% ) 1.21 1.19 +1.3% 0.987 0.976

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants