DeepSC algorithm for ECAL SuperClusters

[valsdav]

PR description:

The PR introduces the implementation of the DeepSC algorithm in the ECAL SuperCluster reconstruction sequence.
The algorithm is based on graph NNs and it is evaluated through the vanilla CMSSW Tensorflow interface.

Three modules have been implemented to handle the ECAL PFClusters as a graph and to encapsulate the evaluation of the DeepSC algo.

• EcalClustersGraph: utility class exposing the DeepSC algorithm to the PFECALSuperClusterAlgo
• DeepSCGraphEvalution: encapsulates the tensorflow interface and inputs handling
• GraphMap: utility class to extract sub-graphs as the final SuperClusters candidates after the model evaluation

A separate PR (cms-data/RecoEcal-EgammaClusterProducers#2) has been created for the model data.

A summary of the features implemented in the PR and the relative validation can be found at:
https://indico.cern.ch/event/1129736/#5-dnn-superclustering

The algorithm is intended to replace the Mustache (not in the short term) and it is OFF by default in the PR. It can be switched ON by the ecal_deepsc process modifier.

The DeepSC algorithm has been presented in ECAL, PF and ML group meetings. More documentation and details can be find in the links below.

Recent talks and documentation:

• ACAT2021 proceeding: https://cms.cern.ch/iCMS/user/noteinfo?cmsnoteid=CMS%20CR-2022/031
• DPS note approved for ACAT2021: https://cds.cern.ch/record/2792321
• PF meeting: https://indico.cern.ch/event/1102164/#1-ml-based-ecal-clustering
• ML Townhall meeting 2021: https://indico.cern.ch/event/1045606/timetable/#20210726.detailed

PR validation:

No changes are expected in any standard workflow. The DeepSC model is only activated by the run3_ecalclustering process modifier. A summary of the features implemented in the PR and the relative validation can be found at:
https://indico.cern.ch/event/1129736/#5-dnn-superclustering

Added runTheMatrix workflow 11846.19 on Zee_14TeV.

Profiling

11846 workflow: 2021PU+ZEE_14TeV_TuneCP5_GenSim+DigiPU+RecoNanoPU+HARVESTNanoPU

11846.0 standard reco:

• CPU https://dvalsecc.web.cern.ch/cgi-bin/igprof-navigator/DeepSC/step3_igprofCPU_w11846.0
• MEM https://dvalsecc.web.cern.ch/cgi-bin/igprof-navigator/DeepSC/step3_igprofMEM99_w11846.0

comment: I am not able to find a reference to the PFECALSuperCluster producer in the profiler output, probably given the small footprint of the standard Mustache.

11846.19 DeepSC SuperClusters

• CPU https://dvalsecc.web.cern.ch/cgi-bin/igprof-navigator/DeepSC/step3_igprofCPU_w11846.19
• MEM https://dvalsecc.web.cern.ch/cgi-bin/igprof-navigator/DeepSC/step3_igprofMEM99_w11846.19

comment: evaluate the DeepSC takes 3.8% of the CPU time (https://dvalsecc.web.cern.ch/cgi-bin/igprof-navigator/DeepSC/step3_igprofCPU_w11846.19/100) , whereas the memory footprint is rather small 0.46% (https://dvalsecc.web.cern.ch/cgi-bin/igprof-navigator/DeepSC/step3_igprofMEM99_w11846.19/809).

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37115/28616

• This PR adds an extra 60KB to repository

• Found files with invalid states:

  • RecoEcal/EgammaCoreTools/interface/GraphMatrix.h:
    • Added: 9d00a35
    • Deleted: 26a3b96

[cmsbuild]

A new Pull Request was created by @valsdav (Davide Valsecchi) for master.

It involves the following packages:

• RecoEcal/EgammaClusterAlgos (reconstruction)
• RecoEcal/EgammaClusterProducers (reconstruction)
• RecoEcal/EgammaCoreTools (reconstruction)

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@Sam-Harper, @lecriste, @rchatter, @jainshilpi, @rovere, @lgray, @sobhatta, @thomreis, @afiqaize, @simonepigazzini, @wrtabb, @argiro, @varuns23, @ram1123 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37115/28618

• This PR adds an extra 56KB to repository

[cmsbuild]

Pull request #37115 was updated. @jpata, @cmsbuild, @clacaputo, @slava77 can you please check and sign again.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37115/28619

• This PR adds an extra 56KB to repository

[cmsbuild]

Pull request #37115 was updated. @jpata, @cmsbuild, @clacaputo, @slava77 can you please check and sign again.

[jpata]

test parameters:

• pull_requests = Adding models files for DeepSC algorithm cms-data/RecoEcal-EgammaClusterProducers#2

[jpata]

could you add a new test workflow with the modifier?

[valsdav]

could you add a new test workflow with the modifier?

Which number should be used for the test WF?

[jpata]

AFAIk you can pick any number that is not already used by something else.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-80034b/25522/summary.html
COMMIT: 2aec0fb
CMSSW: CMSSW_12_5_X_2022-06-14-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37115/25522/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-80034b/11846.19_ZEE_14+2021PU_ecalDeepSC+ZEE_14TeV_TuneCP5_GenSim+DigiPU+RecoNanoPU+HARVESTNanoPU

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3659074
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3659044
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[valsdav]

Dear @reconstruction-l2, @pdmv-l2 @operations-l2, @upgrade-l2 after the latest small improvement, on our side we are satisfied about the PR status. If you also are, can you please sign at your convenience? Thanks!

[srimanob]

+Upgrade

[jpata]

+reconstruction

• since the last sig in DeepSC algorithm for ECAL SuperClusters #37115 (comment), fix kBOX handling
• does not affect standard reco
• requires Adding models files for DeepSC algorithm cms-data/RecoEcal-EgammaClusterProducers#2

[perrotta]

@cms-sw/pdmv-l2 changes since your last signature are minimal, and all included in this commit

[perrotta]

@cms-sw/pdmv-l2 changes since your last signature are minimal, and all included in this commit

@cms-sw/pdmv-l2 if you have any remaing doubt, could you please report it here, so that we know we must wait for them being solved? Otherwise, could you please re-sign before the tests expire?

[kskovpen]

+pdmv

[perrotta]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.