[DD4hep] Add DD4hep Run 3 workflow that reads geometry from DB

[cvuosalo]

Up to now, DD4hep workflows have read geometry from XML files. There is now a DD4hep ideal geometry description in the Conditions DB, so a DD4hep workflow that reads geometry from the DB can be created.

Addition to the PR/IB tests for this workflow is commented out for now until the Cond DB issue with creating payloads is resolved.

This PR depends on PR #34344 to run.

PR validation:

runTheMatrix.py correctly creates the configuration for running this workflow. The workflow can run after the Cond DB issue with creating DB payloads is resolved.

No backport will be done.

[cvuosalo]

When I try to run the new 11634.912 workflow, the following error occurs. I don't know what it means.

n exception of category 'ConditionDatabase' occurred while
   [0] Processing global begin Run 
Exception Message:
Payload of type FileBlob with id 2fdfa84a08623caa48cec3d71726922f938d2cd5 could not be loaded. An exception of category 'ConditionDatabase' occurred.
Exception Message:
De-serialization failed: invalid signature Payload serialization info:  {
"CMSSW_version": "CMSSW_12_0_X_2021-07-03-1100",
"architecture": "slc7_amd64_gcc900",
"technology": "boost/serialization",
"tech_version": "1_75"
 }
 from default_deserialize 
 from Session::fetchPayload 

[cvuosalo]

@civanch FYI: Here's the DD4hep DB workflow PR.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34343/23715

• This PR adds an extra 28KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/README.md modified in PR(s): Add run3 tracking low pu era #33532
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Run 2, Run3, and Phase2 tracker: adding T27 (geometry with bricked pixels) and support for storing bricked pixel information #34120, Add run3 tracking low pu era #33532, Bianca pinolini ecal reco phase2 commit unimib #34212
  • File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): Add GPUParams to runTheMatrix workflows #33538, Bianca pinolini ecal reco phase2 commit unimib #34212

[cmsbuild]

A new Pull Request was created by @cvuosalo (Carl Vuosalo) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @srimanob can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[srimanob]

When I try to run the new 11634.912 workflow, the following error occurs. I don't know what it means.

n exception of category 'ConditionDatabase' occurred while
   [0] Processing global begin Run 
Exception Message:
Payload of type FileBlob with id 2fdfa84a08623caa48cec3d71726922f938d2cd5 could not be loaded. An exception of category 'ConditionDatabase' occurred.
Exception Message:
De-serialization failed: invalid signature Payload serialization info:  {
"CMSSW_version": "CMSSW_12_0_X_2021-07-03-1100",
"architecture": "slc7_amd64_gcc900",
"technology": "boost/serialization",
"tech_version": "1_75"
 }
 from default_deserialize 
 from Session::fetchPayload 

@cms-sw/alca-l2 Could you please advise? Thanks.

[cvuosalo]

It would be good to get this PR into 12_0_0_pre4.

[srimanob]

Please test

[srimanob]

Hi @civanch
The workflow defined here is to make short matrix runs. All run-3 workflows with DB DD4hep can be done with upgrades option of runTheMatrix, or with relvals machine of PdmV. No need to define them here.

[srimanob]

Urgent

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1112a1/16497/summary.html
COMMIT: 545522e
CMSSW: CMSSW_12_0_X_2021-07-06-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34343/16497/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 11634.912

----- Begin Fatal Exception 06-Jul-2021 22:07:16 CEST-----------------------
An exception of category 'ConditionDatabase' occurred while
   [0] Processing global begin Run 
Exception Message:
Payload of type FileBlob with id 2fdfa84a08623caa48cec3d71726922f938d2cd5 could not be loaded. An exception of category 'ConditionDatabase' occurred.
Exception Message:
De-serialization failed: invalid signature Payload serialization info:  {
"CMSSW_version": "CMSSW_12_0_X_2021-07-03-1100",
"architecture": "slc7_amd64_gcc900",
"technology": "boost/serialization",
"tech_version": "1_75"
 }
 from default_deserialize 
 from Session::fetchPayload 
----- End Fatal Exception -------------------------------------------------

[srimanob]

@cvuosalo @civanch @cms-sw/alca-l2
Could you please clarify if the issue is on GT or something else? I assume when @cvuosalo develops this PR, the test is done with the candidate GT, or this is the last thing to solve before we can run until the end?

Since this is the workflow level, if we are not ready (e.g. can't pass the PR test), we do not need to be merge. This is not a showstopper to validate DD4hep DB because PdmV can pick the cmsDriver to submit the relvals anyways.

What do you think? @cms-sw/pdmv-l2 @silviodonato @qliphy @perrotta

[srimanob]

test parameters:

• workflow = 11634.911

[srimanob]

Please test

[srimanob]

test parameters:

Ah, OK. Thanks @perrotta

[srimanob]

Please abort

[srimanob]

test parameters:

[srimanob]

Please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1112a1/16611/summary.html
COMMIT: 05beb78
CMSSW: CMSSW_12_0_X_2021-07-07-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34343/16611/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 5159 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2786260
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2786225
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.004 KiB( 37 files compared)
• DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[srimanob]

+Upgrade

This PR is to add the workflow for DD4hep with DB geometry, however, the workflow is disable for now.

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1

Reco comparison results: 5159 differences found in the comparisons

this is not related to this PR, see #34409.