ECAL Logs Cleanup #28672

tanmaymudholkar · 2019-12-21T00:50:37Z

Disabled a few plots from the offline workflows (that were not meant to be filled), stopping them from filling up the logs with warnings. These changes have no effect on DQM other than a significant reduction in these warnings.

PR description:

Due to diverse reasons, offline DQM logs are currently filled up with false alarms from ECAL DQM. Some plots that are only meant to be filled online (HLT emulation) produce a warning when an offline DQM process tries to fill them. Also, the collections passed to ECAL DQM are slightly different in different running conditions (offline, HI, pp collisions), and this is expected because of the slightly different rechit algos run with each process. Plots that are not meaningful for cosmics runs also produce similar warnings.

This PR fixes most problems with unnecessary warnings in the logs due to ECAL DQM. This is done by introducing a new flag that is set differently in the offline cfg file or by reusing already existing flags, and also by including the correct collection names in the cfg files.

PR validation:

We ran the standard suite of tests: runTheMatrix.py --nproc=4 --nThreads=10 -l limited -i all --ibeos, and noted that all warnings related to ECAL DQM disappear from the logs.

if this PR is a backport please specify the original PR:

Not a backport.

…o be filled. These changes have no effect on DQM other than a significant reduction in the ECAL DQM warnings published on the logs.

cmsbuild · 2019-12-21T00:51:08Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-12-21T00:56:19Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28672/13240

This PR adds an extra 20KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28672/13240/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28672/13240/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2019-12-21T01:06:32Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-12-21T01:11:07Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28672/13241

This PR adds an extra 20KB to repository

cmsbuild · 2019-12-21T01:11:37Z

A new Pull Request was created by @tanmaymudholkar (Tanmay Mudholkar) for master.

It involves the following packages:

DQM/EcalMonitorTasks
DQMOffline/Ecal

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@argiro, @rociovilar this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

jfernan2 · 2019-12-21T15:10:48Z

please test
Thanks a lot @tanmaymudholkar

cmsbuild · 2019-12-21T15:11:06Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4085/console Started: 2019/12/21 16:11

cmsbuild · 2019-12-21T16:44:22Z

+1
Tested at: 88c6fed
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e133d3/4085/summary.html
CMSSW: CMSSW_11_1_X_2019-12-21-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2019-12-21T16:44:26Z

Comparison job queued.

cmsbuild · 2019-12-21T18:12:55Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e133d3/4085/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2809739
DQMHistoTests: Total failures: 172
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2809226
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -9614.856 KiB( 33 files compared)
DQMHistoSizes: changed ( 1000.0,... ): -283.273 KiB EcalEndcap/EETriggerTowerTask
DQMHistoSizes: changed ( 1000.0,... ): -117.316 KiB EcalBarrel/EBTriggerTowerTask
DQMHistoSizes: changed ( 4.22,... ): -0.116 KiB EcalBarrel/EBRecoSummary
DQMHistoSizes: changed ( 4.22,... ): -0.116 KiB EcalEndcap/EERecoSummary
Checked 147 log files, 16 edm output root files, 34 DQM output files

jfernan2 · 2019-12-22T10:57:34Z

Thanks a lot for the logs cleaning @tanmaymudholkar and the empty histograms from emulator removed.
There are still some plots TrigPrim left emtpy like
ECal/Trends/TrigPrimClient E* number of TTs with
Ecal*/E* SummaryClient/
Ecal*/E* TriggerTowerTask
Not sure if this could be cleaned further

silviodonato · 2020-01-09T17:13:47Z

@jfernan2 @andrius-k @schneiml @fioriNTU @kmaeshima please review/sign

jfernan2 · 2020-01-09T17:37:22Z

@silviodonato we already reviewed, signing was pending upon an answer for last question above to @tanmaymudholkar to see if further cleaning could be achieved

silviodonato · 2020-01-09T17:49:44Z

@silviodonato we already reviewed, signing was pending upon an answer for last question above to @tanmaymudholkar to see if further cleaning could be achieved

I'm sorry, I missed your very last comment.

silviodonato · 2020-01-15T10:09:20Z

@tanmaymudholkar

silviodonato · 2020-01-21T08:22:06Z

@tanmaymudholkar today we are going to make CMSSW_11_1_0_pre2. If you can address @jfernan2 ' s comment in time, we might include this cleanup in CMSSW_11_1_0_pre2

silviodonato · 2020-01-28T15:10:11Z

@tanmaymudholkar, could you address to @jfernan2's comments? thanks

silviodonato · 2020-01-30T09:52:38Z

@tanmaymudholkar could you reply to this comment #28672 (comment) ?

silviodonato · 2020-02-04T09:34:49Z

Weekly reminder of this PR

silviodonato · 2020-02-10T08:45:49Z

@tanmaymudholkar I'm closing this PR because there was no activity in the last month. Please feel free to open it back as soon as you have any news.

mmusich · 2020-02-10T08:52:02Z

@silviodonato @jfernan2
some time ago, when working to consolidate the beam halo and cosmics relvals in cmssw and I stumbled upon the fact that the there seem to be some misconfiguration in the ECAL part.
When I run e.g workflow 7.2 I get swamped by messages of the type:

%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 20-Dec-2019 15:15:03 CET  Run: 1 Event: 77
Ecal Monitor Source::runOnCollection: EBUncalibRecHit does not exist
%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 20-Dec-2019 15:15:03 CET  Run: 1 Event: 77
Ecal Monitor Source::runOnCollection: EEUncalibRecHit does not exist
%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 20-Dec-2019 15:15:03 CET  Run: 1 Event: 77
Ecal Monitor Source::runOnCollection: EBReducedRecHit does not exist
%MSG
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 20-Dec-2019 15:15:03 CET  Run: 1 Event: 77
Ecal Monitor Source::runOnCollection: EEReducedRecHit does not exist
%MSG

The large amount of I/O makes it cumbersome to test it.
I am wondering if this PR could be considered for merging even if the clean-up is not complete as it would at least get rid of the deluge of warnings.

jfernan2 · 2020-02-10T08:54:51Z

I am in favour of @mmusich proposal, however I feel rather annoyed by the delay in answer from the ECAL contact, I have contacted the conveners several times about this.
I am doing it again once more

silviodonato · 2020-02-10T09:24:11Z

merge
@jfernan2 @mmusich ok, I think #28672 (comment) can be done later with a small PR.

jfernan2 · 2020-02-11T09:42:01Z

+1

cmsbuild · 2020-02-11T09:42:21Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

Disabled a few plots from the offline workflows that were not meant t…

2ed8215

…o be filled. These changes have no effect on DQM other than a significant reduction in the ECAL DQM warnings published on the logs.

cmsbuild added this to the CMSSW_11_1_X milestone Dec 21, 2019

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Dec 21, 2019

cmsbuild added code-checks-rejected and removed code-checks-pending labels Dec 21, 2019

Enforced by automated style checks.

88c6fed

cmsbuild added code-checks-pending and removed code-checks-rejected labels Dec 21, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 21, 2019

cmsbuild added tests-started and removed tests-pending labels Dec 21, 2019

cmsbuild added tests-approved and removed tests-started labels Dec 21, 2019

cmsbuild removed the comparison-pending label Dec 21, 2019

cmsbuild added the comparison-available label Dec 21, 2019

cmsbuild added orp-approved and removed orp-pending labels Feb 10, 2020

cmsbuild merged commit f7d8f42 into cms-sw:master Feb 10, 2020

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Feb 11, 2020

tanmaymudholkar deleted the OfflineLogsCleanup_11_1_X branch February 26, 2020 22:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECAL Logs Cleanup #28672

ECAL Logs Cleanup #28672

tanmaymudholkar commented Dec 21, 2019 •

edited

Loading

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

jfernan2 commented Dec 21, 2019

cmsbuild commented Dec 21, 2019 •

edited

Loading

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

jfernan2 commented Dec 22, 2019 •

edited

Loading

silviodonato commented Jan 9, 2020

jfernan2 commented Jan 9, 2020

silviodonato commented Jan 9, 2020

silviodonato commented Jan 15, 2020

silviodonato commented Jan 21, 2020

silviodonato commented Jan 28, 2020

silviodonato commented Jan 30, 2020

silviodonato commented Feb 4, 2020

silviodonato commented Feb 10, 2020

mmusich commented Feb 10, 2020

jfernan2 commented Feb 10, 2020 •

edited

Loading

silviodonato commented Feb 10, 2020

jfernan2 commented Feb 11, 2020

cmsbuild commented Feb 11, 2020

ECAL Logs Cleanup #28672

ECAL Logs Cleanup #28672

Conversation

tanmaymudholkar commented Dec 21, 2019 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR:

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

jfernan2 commented Dec 21, 2019

cmsbuild commented Dec 21, 2019 • edited Loading

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

cmsbuild commented Dec 21, 2019

jfernan2 commented Dec 22, 2019 • edited Loading

silviodonato commented Jan 9, 2020

jfernan2 commented Jan 9, 2020

silviodonato commented Jan 9, 2020

silviodonato commented Jan 15, 2020

silviodonato commented Jan 21, 2020

silviodonato commented Jan 28, 2020

silviodonato commented Jan 30, 2020

silviodonato commented Feb 4, 2020

silviodonato commented Feb 10, 2020

mmusich commented Feb 10, 2020

jfernan2 commented Feb 10, 2020 • edited Loading

silviodonato commented Feb 10, 2020

jfernan2 commented Feb 11, 2020

cmsbuild commented Feb 11, 2020

tanmaymudholkar commented Dec 21, 2019 •

edited

Loading

cmsbuild commented Dec 21, 2019 •

edited

Loading

jfernan2 commented Dec 22, 2019 •

edited

Loading

jfernan2 commented Feb 10, 2020 •

edited

Loading