PyReleaseValidation: move DD4hep test workflow into upgrade special wf section

[fabiocos]

PR description:

This PR is a follow-up of the discussion in #29975 about the possible workflow to test DD4hep in a standard environment. As now a 2021 full geometry is available, a list of possible workflows fully based on DD4hep can be easily added as suggested by @kpedro88. The workflows themselves are failing, and for this reason they are not made available to the IB or short matrix. But they can be used for routine checks.

As offset ".911" has been chosen, it does not overlaps with any other use to my knowledge.

PR validation:

17:41 cmsdev25 1039> runTheMatrix.py --what upgrade -l 11634.911 --dryRun 

ignoring non-requested file relval_standard
ignoring non-requested file relval_highstats
ignoring non-requested file relval_pileup
ignoring non-requested file relval_generator
ignoring non-requested file relval_extendedgen
ignoring non-requested file relval_production
ignoring non-requested file relval_ged
processing relval_upgrade
ignoring non-requested file relval_2017
ignoring non-requested file relval_2026
ignoring non-requested file relval_identity
ignoring non-requested file relval_machine
ignoring non-requested file relval_unsch
ignoring non-requested file relval_premix
Running in 4 thread(s)

Preparing to run 11634.911 TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021

# in: /build/fabiocos/112X/dd4hep-test/CMSSW_11_2_X_2020-05-26-2300/work dryRun for 'cd 11634.911_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021
 cmsDriver.py TTbar_14TeV_TuneCP5_cfi  --conditions auto:phase1_2021_realistic -n 10 --era Run3_dd4hep --eventcontent FEVTDEBUG --relval 9000,100 -s GEN,SIM --datatier GEN-SIM --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --geometry DD4hepExtended2021 --fileout file:step1.root  > step1_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021.log  2>&1
 

# in: /build/fabiocos/112X/dd4hep-test/CMSSW_11_2_X_2020-05-26-2300/work dryRun for 'cd 11634.911_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021
 cmsDriver.py step2  --conditions auto:phase1_2021_realistic -s DIGI:pdigi_valid,L1,DIGI2RAW,HLT:@relval2021 --datatier GEN-SIM-DIGI-RAW -n 10 --geometry DD4hepExtended2021 --era Run3_dd4hep --eventcontent FEVTDEBUGHLT --filein  file:step1.root  --fileout file:step2.root  > step2_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021.log  2>&1
 

# in: /build/fabiocos/112X/dd4hep-test/CMSSW_11_2_X_2020-05-26-2300/work dryRun for 'cd 11634.911_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021
 cmsDriver.py step3  --conditions auto:phase1_2021_realistic -n 10 --era Run3_dd4hep --eventcontent RECOSIM,MINIAODSIM,DQM --runUnscheduled  -s RAW2DIGI,L1Reco,RECO,RECOSIM,EI,PAT,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO --geometry DD4hepExtended2021 --filein  file:step2.root  --fileout file:step3.root  > step3_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021.log  2>&1
 

# in: /build/fabiocos/112X/dd4hep-test/CMSSW_11_2_X_2020-05-26-2300/work dryRun for 'cd 11634.911_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021
 cmsDriver.py step4  --conditions auto:phase1_2021_realistic -s HARVESTING:@standardValidation+@standardDQM+@ExtraHLT+@miniAODValidation+@miniAODDQM --scenario pp --filetype DQM --geometry DD4hepExtended2021 --era Run3_dd4hep --mc  -n 100  --filein file:step3_inDQM.root --fileout file:step4.root  > step4_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021.log  2>&1

# in: /build/fabiocos/112X/dd4hep-test/CMSSW_11_2_X_2020-05-26-2300/work dryRun for 'cd 11634.911_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021
 cmsDriver.py step5  --conditions auto:phase1_2021_realistic -n 10 --era Run3_dd4hep --eventcontent ALCARECO --filein file:step3.root -s ALCA:SiPixelCalSingleMuon+TkAlMuonIsolated+TkAlMinBias+MuAlOverlaps+EcalESAlign+TkAlZMuMu+HcalCalHBHEMuonFilter+TkAlUpsilonMuMu+TkAlJpsiMuMu+SiStripCalMinBias --datatier ALCARECO --geometry DD4hepExtended2021 --fileout file:step5.root  > step5_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021.log  2>&1
 
11634.911_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_DD4hep+DigiFull_DD4hep_2021+RecoFull_DD4hep_2021+HARVESTFull_DD4hep_2021+ALCAFull_DD4hep_2021 Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED  - time date Wed May 27 17:40:48 2020-date Wed May 27 17:40:48 2020; exit: 0 0 0 0 0
1 1 1 1 1 tests passed, 0 0 0 0 0 failed

The list of workflows is just enlarged by the desired addition.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30003/15681

• This PR adds an extra 20KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): New met filter #29996

[cmsbuild]

A new Pull Request was created by @fabiocos (Fabio Cossutti) for master.

It involves the following packages:

Configuration/PyReleaseValidation
Geometry/CMSCommonData

@pgunnell, @civanch, @Dr15Jones, @chayanit, @cvuosalo, @wajidalikhan, @ianna, @mdhildreth, @cmsbuild, @makortel, @kpedro88 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

[fabiocos]

@bsunanda this might be of interest for you

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6621/console Started: 2020/05/27 18:02

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+1
Tested at: c153dd1
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339979/6621/summary.html
CMSSW: CMSSW_11_2_X_2020-05-26-2300
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

+1
Tested at: 753cf5c
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339979/6633/summary.html
CMSSW: CMSSW_11_2_X_2020-05-27-2300
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339979/6633/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 36
• DQMHistoTests: Total histograms compared: 2780792
• DQMHistoTests: Total failures: 16
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2780726
• DQMHistoTests: Total skipped: 50
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
• Checked 152 log files, 16 edm output root files, 36 DQM output files

[fabiocos]

failures are clearly unrelated to this PR (which has no effect in this test, it requires --what upgrade to become active)

[cvuosalo]

+1

[kpedro88]

+upgrade

[fabiocos]

@chayanit any comment?

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1