Enable concurrent lumisection processing in ECAL #30187

tanmaymudholkar · 2020-06-11T06:24:45Z

Changed ECAL Monitor Tasks from DQMOneLumiEDAnalyzer to DQMOneEDAnalyzer. All by-lumisection plots are disabled for the time being.

PR description:

There has been a long-standing request to enable concurrent processing for the ECAL DQM workflows. The problematic functions in DQMOneLumiEDAnalyzer are the ones that act at the beginning and end of the lumisection transitions. This PR migrates to DQMOneEDAnalyzer and disables all by-lumisection plots for now.

PR validation:

We have run the recommended suite of tests: runTheMatrix.py -l limited -i all --ibeos

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport.

…nalyzer. All by-lumisection plots are disabled for the time being.

cmsbuild · 2020-06-11T06:25:07Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-11T06:33:34Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16021

This PR adds an extra 76KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/EcalMonitorClient/plugins/EcalDQMonitorClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorClient/src/DQWorkerClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935
- File DQM/EcalMonitorClient/src/TimingClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/plugins/EcalDQMonitorTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/EnergyTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/IntegrityTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/LaserTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/LedTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/OccupancyTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/RawDataTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/TimingTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/TrigPrimTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16021/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16021/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-06-11T19:55:56Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-11T20:00:33Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16048

This PR adds an extra 76KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/EcalMonitorClient/plugins/EcalDQMonitorClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorClient/src/DQWorkerClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935
- File DQM/EcalMonitorClient/src/TimingClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/plugins/EcalDQMonitorTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/EnergyTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/IntegrityTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/LaserTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/LedTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/OccupancyTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/RawDataTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/TimingTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/TrigPrimTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936

cmsbuild · 2020-06-11T20:00:57Z

A new Pull Request was created by @tanmaymudholkar (Tanmay Mudholkar) for master.

It involves the following packages:

DQM/EcalCommon
DQM/EcalMonitorClient
DQM/EcalMonitorTasks

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@rchatter, @argiro this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2020-06-11T20:55:01Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-11T21:01:08Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16049

This PR adds an extra 120KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/EcalMonitorClient/plugins/EcalDQMonitorClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorClient/src/DQWorkerClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935
- File DQM/EcalMonitorClient/src/TimingClient.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/plugins/EcalDQMonitorTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/EnergyTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/IntegrityTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/LaserTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/LedTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/OccupancyTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/RawDataTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/TimingTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936
- File DQM/EcalMonitorTasks/src/TrigPrimTask.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-auto #29936

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16049/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30187/16049/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-07-10T11:50:53Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-07-09-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7856/console Started: 2020/07/10 14:57

cmsbuild · 2020-07-10T14:44:34Z

+1
Tested at: 14e8052
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1c6a6c/7856/summary.html
CMSSW: CMSSW_11_2_X_2020-07-09-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-07-10T14:44:36Z

Comparison job queued.

jfernan2 · 2020-07-10T14:45:53Z

@silviodonato it depends on @tanmaymudholkar : the PR as it is now, only removes the conflicting code but this causes troubles in other parts of ECAL dqm

cmsbuild · 2020-07-10T17:23:44Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1c6a6c/7856/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2783367
DQMHistoTests: Total failures: 308
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2783009
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -81295.177 KiB( 36 files compared)
DQMHistoSizes: changed ( 1000.0,... ): -540.738 KiB EcalBarrel/EBPedestalOnlineTask
DQMHistoSizes: changed ( 1000.0,... ): -540.000 KiB EcalBarrel/EBTimingTask
DQMHistoSizes: changed ( 1000.0,... ): -285.020 KiB EcalBarrel/EBSummaryClient
DQMHistoSizes: changed ( 1000.0,... ): -251.787 KiB EcalEndcap/EESummaryClient
DQMHistoSizes: changed ( 1000.0,... ): -243.489 KiB EcalBarrel/EBIntegrityTask
DQMHistoSizes: changed ( 1000.0,... ): -243.270 KiB EcalBarrel/EBIntegrityClient
DQMHistoSizes: changed ( 1000.0,... ): -229.588 KiB EcalEndcap/EEPedestalOnlineTask
DQMHistoSizes: changed ( 1000.0,... ): -229.219 KiB EcalEndcap/EETimingTask
DQMHistoSizes: changed ( 1000.0,... ): -162.824 KiB EcalEndcap/EETriggerTowerTask
DQMHistoSizes: changed ( 1000.0,... ): -89.217 KiB EcalEndcap/EEOccupancyTask
DQMHistoSizes: changed ( 1000.0 ): ...
Checked 154 log files, 17 edm output root files, 37 DQM output files

silviodonato · 2020-07-21T14:08:18Z

@tanmaymudholkar do you have any news?

tanmaymudholkar · 2020-07-22T06:16:12Z

@silviodonato sorry again for the delay, I was waiting for confirmation about the timeline for our next fix, but in any case I will post an update in a few hours.

silviodonato · 2020-07-27T14:14:39Z

FYI CMSSW_11_2_0_pre3 will be cut tomorrow evening

tanmaymudholkar · 2020-07-28T13:12:33Z

Hi @silviodonato I tried to track down the remaining bad quality plots, but I think it would be best to proceed with the current PR. Some plots might be temporarily bad, but we are reintroducing the by-LS plots anyway.

silviodonato · 2020-08-05T08:49:21Z

@tanmaymudholkar does @cms-sw/dqm-l2 agree to go with the current PR?

jfernan2 · 2020-08-05T15:06:09Z

I don't think this PR makes any help: the code is removed but a lot of plots are broken as mentioned before

silviodonato · 2020-08-25T09:57:51Z

@tanmaymudholkar I moved this PR as a "draft". Please open it back as soon as you have an update to be reviewed (please let me know if you do not have the permissions to do it)

silviodonato · 2020-10-07T15:19:00Z

moved to #31689

Changed all ECAL Monitor Tasks from DQMOneLumiEDAnalyzer to DQMOneEDA…

71eca1e

…nalyzer. All by-lumisection plots are disabled for the time being.

cmsbuild added this to the CMSSW_11_2_X milestone Jun 11, 2020

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Jun 11, 2020

tanmaymudholkar mentioned this pull request Jun 11, 2020

DQM: Remove concurrent-lumi blocking DQM Modules. #30111

Closed

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jun 11, 2020

Minor style changes to make code pass CMSSW code style checks.

c7f0841

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jun 11, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 11, 2020

Removed all comments introduced in this PR to keep code clean.

cbdce40

cmsbuild added code-checks-pending and removed code-checks-approved labels Jun 11, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jun 11, 2020

Further minor style changes to make code pass CMSSW code style checks.

f7373e5

cmsbuild removed the code-checks-rejected label Jun 11, 2020

cmsbuild added comparison-pending tests-pending labels Jul 10, 2020

cmsbuild added tests-started and removed tests-pending labels Jul 10, 2020

cmsbuild added tests-approved and removed tests-started labels Jul 10, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jul 10, 2020

cmsbuild mentioned this pull request Jul 15, 2020

HLTTauDQM: conditional plotting #30243

Closed

cmsbuild mentioned this pull request Jul 28, 2020

HDF5 interface for LHE-HDF5 event files #30312

Merged

schneiml mentioned this pull request Jul 29, 2020

Modules prohibiting concurrent lumis in RelVal workflows #25090

Closed

silviodonato marked this pull request as draft August 25, 2020 09:55

silviodonato marked this pull request as ready for review September 1, 2020 13:42

cmsbuild mentioned this pull request Sep 5, 2020

Remove legacy dqm #31369

Closed

tanmaymudholkar mentioned this pull request Oct 7, 2020

Fix to allow concurrent processing of lumisections with ECAL DQM #31689

Merged

silviodonato closed this Oct 7, 2020

tanmaymudholkar deleted the ConcurrentLumisFix_112X branch October 20, 2020 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable concurrent lumisection processing in ECAL #30187

Enable concurrent lumisection processing in ECAL #30187

tanmaymudholkar commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jul 10, 2020 •

edited

Loading

cmsbuild commented Jul 10, 2020

cmsbuild commented Jul 10, 2020

jfernan2 commented Jul 10, 2020

cmsbuild commented Jul 10, 2020

silviodonato commented Jul 21, 2020

tanmaymudholkar commented Jul 22, 2020

silviodonato commented Jul 27, 2020

tanmaymudholkar commented Jul 28, 2020

silviodonato commented Aug 5, 2020

jfernan2 commented Aug 5, 2020

silviodonato commented Aug 25, 2020

silviodonato commented Oct 7, 2020

Enable concurrent lumisection processing in ECAL #30187

Enable concurrent lumisection processing in ECAL #30187

Conversation

tanmaymudholkar commented Jun 11, 2020

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jun 11, 2020

cmsbuild commented Jul 10, 2020 • edited Loading

cmsbuild commented Jul 10, 2020

cmsbuild commented Jul 10, 2020

jfernan2 commented Jul 10, 2020

cmsbuild commented Jul 10, 2020

silviodonato commented Jul 21, 2020

tanmaymudholkar commented Jul 22, 2020

silviodonato commented Jul 27, 2020

tanmaymudholkar commented Jul 28, 2020

silviodonato commented Aug 5, 2020

jfernan2 commented Aug 5, 2020

silviodonato commented Aug 25, 2020

silviodonato commented Oct 7, 2020

cmsbuild commented Jul 10, 2020 •

edited

Loading