Lock more methods of JobReport [12_1] #38018

Dr15Jones · 2022-05-19T14:32:40Z

PR description:

Production jobs were seeing scrambled job reports caused by pileup file entries.

PR validation:

Code compiles

if this PR is a backport please specify the original PR and why you need to backport that PR:

backport of #37946

Needed for production.

Production jobs were seeing scrambled job reports caused by pileup file entries.

cmsbuild · 2022-05-19T14:32:58Z

A new Pull Request was created by @Dr15Jones (Chris Jones) for CMSSW_12_1_X.

It involves the following packages:

FWCore/MessageLogger (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@makortel, @wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Dr15Jones · 2022-05-19T14:37:27Z

please test

makortel · 2022-05-23T13:09:14Z

@cmsbuild, please abort

makortel · 2022-05-23T13:09:23Z

@cmsbuild, please test

cmsbuild · 2022-05-23T17:18:15Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66cb7e/24911/summary.html
COMMIT: 2347b3d
CMSSW: CMSSW_12_1_X_2022-05-22-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38018/24911/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

138.4138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3/step2_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3.log
250203.18250203.18_H125GGgluonfusion_13UP18+H125GGgluonfusion_13UP18INPUT+DIGIPRMXUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25/step2_H125GGgluonfusion_13UP18+H125GGgluonfusion_13UP18INPUT+DIGIPRMXUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25.log
250203.17250203.17_H125GGgluonfusion_13UP17+H125GGgluonfusion_13UP17INPUT+DIGIPRMXUP17_PU25+RECOPRMXUP17_PU25+HARVESTUP17_PU25/step2_H125GGgluonfusion_13UP17+H125GGgluonfusion_13UP17INPUT+DIGIPRMXUP17_PU25+RECOPRMXUP17_PU25+HARVESTUP17_PU25.log

Expand to see more relval errors ...

250203.0
250202.5
250202.4
250204.0
250204.17
250204.18
250205.0
250205.17
250205.18
250206.0
250206.17
250206.18
250206.181
250207.0
250207.17
250207.18
250208.17
250208.18
500200.0
500202.0
500203.0
500204.0
500205.0
500206.0
500207.0

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2901406
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 41 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

makortel · 2022-05-23T17:30:22Z

138.4 fails already in the IB with

----- Begin Fatal Exception 23-May-2022 14:49:44 UTC-----------------------
An exception of category 'ConditionDatabase' occurred while
   [0] Processing Event run: 346512 lumi: 250 event: 243042266
   [1] Running path 'dqmoffline_step'
Exception Message:
Payload of type LHCInfo with id 33134a07bf5346992723195b6b8ab9e21778b71d could not be loaded. An exception of category 'ConditionDatabase' occurred.
Exception Message:
De-serialization failed: the current boost version (1_75) is unable to read the payload. Data might have been serialized with an incompatible version. Payload serialization info:  {
"CMSSW_version": "CMSSW_12_3_0",
"architecture": "slc7_amd64_gcc10",
"technology": "boost/serialization",
"tech_version": "1_78"
 }
 from default_deserialize 
 from Session::fetchPayload 
----- End Fatal Exception -------------------------------------------------

(@cms-sw/db-l2 @cms-sw/alca-l2)

250203.18 failed with

----- Begin Fatal Exception 23-May-2022 17:10:49 UTC-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_DIGI_DATAMIX_L1_DIGI2RAW_HLT.py
Exception Message:
 unknown python problem occurred.
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/RecoJets/JetProducers/ak4PFJets_cfi.py'

At:
  <frozen importlib._bootstrap_external>(1039): get_data
  <frozen importlib._bootstrap_external>(982): get_code
  <frozen importlib._bootstrap_external>(846): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/CommonTools/ParticleFlow/Tools/jetTools.py(11): jetAlgo
  /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/CommonTools/ParticleFlow/PFBRECO_cff.py(120): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/PhysicsTools/PatAlgos/slimming/packedPFCandidates_cff.py(5): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/PhysicsTools/PatAlgos/slimming/slimming_cff.py(3): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/Configuration/EventContent/EventContent_cff.py(824): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/python/FWCore/ParameterSet/Config.py(695): load
  step2_DIGI_DATAMIX_L1_DIGI2RAW_HLT.py(17): <module>

----- End Fatal Exception -------------------------------------------------

250203.17 failed with

----- Begin Fatal Exception 23-May-2022 17:10:51 UTC-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=PreMixingModule label='mixData'
   [2] While attempting to load plugin PreMixingEcalWorker
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/lib/slc7_amd64_gcc900/pluginSimCalorimetryEcalSimProducersPlugins.so because /cvmfs/cms-ib.cern.ch/nweek-02734/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2022-05-22-0000/lib/slc7_amd64_gcc900/pluginSimCalorimetryEcalSimProducersPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

The other failures than 138.4 look transient.

makortel · 2022-05-23T17:30:41Z

+1

cmsbuild · 2022-05-23T17:30:47Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_1_X IBs (but tests are reportedly failing) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-05-23T19:26:36Z

please test

cmsbuild · 2022-05-23T22:35:14Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66cb7e/24926/summary.html
COMMIT: 2347b3d
CMSSW: CMSSW_12_1_X_2022-05-22-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38018/24926/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

138.4138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3/step2_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2901405
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 41 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

francescobrivio · 2022-05-23T22:43:12Z

-1

RelVals-INPUT

* [138.4](https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66cb7e/24926/runTheMatrixINPUT-results/138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3)`138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3/step2_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3.log`

138.4 fails due to change in the boost version between 12_2 and 12_3.
We had the same issue in 12_2_X (#37653 (comment)) and we fixed it by changing the GTs in #37670. We also introduced a new policy to not have open-ended (with snapshot set to infinity) GTs in the release, and we have enforced it starting from 12_2 onwards.

Do we need fixing the 12_1_X IB?
(I thought they were already closed tbh)

perrotta · 2022-05-24T04:40:22Z

+1

Just in case it is still needed in 12_1...

perrotta · 2022-05-24T04:41:01Z

merge

Lock more methods of JobReport

2347b3d

Production jobs were seeing scrambled job reports caused by pileup file entries.

cmsbuild added this to the CMSSW_12_1_X milestone May 19, 2022

cmsbuild added core-pending orp-pending pending-signatures tests-pending labels May 19, 2022

makortel mentioned this pull request May 19, 2022

Fix race condition between JobReport::inputFileClosed() and JobReport::reportReadBranches() cms-sw/framework-team#400

Closed

5 tasks

cmsbuild added tests-started and removed tests-pending labels May 19, 2022

cmsbuild added tests-rejected and removed tests-started labels May 23, 2022

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels May 23, 2022

cmsbuild added tests-started and removed tests-rejected labels May 23, 2022

cmsbuild added tests-rejected and removed tests-started labels May 23, 2022

cmsbuild added orp-approved and removed orp-pending labels May 24, 2022

cmsbuild merged commit a8f3e62 into cms-sw:CMSSW_12_1_X May 24, 2022

Dr15Jones deleted the moreLocksJobReport_12_1 branch May 24, 2022 14:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lock more methods of JobReport [12_1] #38018

Lock more methods of JobReport [12_1] #38018

Dr15Jones commented May 19, 2022

cmsbuild commented May 19, 2022

Dr15Jones commented May 19, 2022

makortel commented May 23, 2022

makortel commented May 23, 2022

cmsbuild commented May 23, 2022

makortel commented May 23, 2022

makortel commented May 23, 2022

cmsbuild commented May 23, 2022

perrotta commented May 23, 2022

cmsbuild commented May 23, 2022

francescobrivio commented May 23, 2022

RelVals-INPUT

perrotta commented May 24, 2022

perrotta commented May 24, 2022

Lock more methods of JobReport [12_1] #38018

Lock more methods of JobReport [12_1] #38018

Conversation

Dr15Jones commented May 19, 2022

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented May 19, 2022

Dr15Jones commented May 19, 2022

makortel commented May 23, 2022

makortel commented May 23, 2022

cmsbuild commented May 23, 2022

RelVals-INPUT

Comparison Summary

makortel commented May 23, 2022

makortel commented May 23, 2022

cmsbuild commented May 23, 2022

perrotta commented May 23, 2022

cmsbuild commented May 23, 2022

RelVals-INPUT

Comparison Summary

francescobrivio commented May 23, 2022

RelVals-INPUT

perrotta commented May 24, 2022

perrotta commented May 24, 2022