Lumi-based status plots on GEM onlineDQM

[quark2]

PR description:

A new type of status plots are added; lumi-based status plots. It displays the change of statuses of chambers as a function of lumi-blocks (=time).

PR validation:

Test are done and one can check again by runTheMatrix workflows

@jshlee @watson-ij

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37135/28668

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @quark2 for master.

It involves the following packages:

• DQM/GEM (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please review it and eventually sign? Thanks.
@watson-ij this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[jfernan2]

test parameters:

• addpkg = DQM/Integration

[jfernan2]

@quark2 we need a backport to 12_2_X to test this PR at P5 Online machines

[jfernan2]

BTW: DQM shifter is complaining that GEM plots are not linked to the Shift workspace in Online DQM GUI.

Could you make them available? A shift_gem_layout.py and a shift_gem_T0_layout.py file for Online and Offline resp. are needed in:

https://github.com/dmwm/deployment/tree/master/dqmgui/layouts

Thanks,

[quark2]

Hi @jfernan2,

Sorry for my terribly late... I've made a PR for the shift plots for GEM on the DQM GUI software. Please see [1].

Best regards,
Byeonghak Ko

[1] dmwm/deployment#1131

[quark2]

Hi @jfernan2,

I think you forgot to write "please test". Could you check it?

Best regards,
Byeonghak Ko

[perrotta]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6bff3c/22950/summary.html
COMMIT: 5bcea1f
CMSSW: CMSSW_12_3_X_2022-03-08-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37135/22950/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• Reco comparison had 14 failed jobs
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3985573
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3985548
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 49 DQM output files

[jfernan2]

@quark2 we need a backport to 12_2_X to test this PR at P5 Online machines

[quark2]

Hi @jfernan2,

Sorry for my late. Here is the backport, #37204.

Best regards,
Byeonghak Ko

[perrotta]

Now the master branch has moved to 12_4_X
Since you have already a backport to 12_2, please also prepare a backport for the intermediate 12_3 cycle, which will be used online after 12_2?

[quark2]

Hi @perrotta,

Yep, sure. The backport for 12_3_X is now ready -- #37213.

Best regards,
Byeonghak Ko

[jfernan2]

+1
Backport PR in 12_2_X Tested successfully in Online DQM at P5

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

@quark2 Would you please confirm the followings are ok?

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_3_X_2022-03-08-1100+6bff3c/48804/validateJR/all_OldVSNew_TTbar14TeV2026D49wf23234p0/all_OldVSNew_TTbar14TeV2026D49wf23234p0.log

plotting variable: recoRecoEcalCandidates_hfRecoEcalCandidate__RECO.obj.pt()
Error in TH1F::KolmogorovTest: Histogram1 recoRecoEcalCandidates_hfRecoEcalCandidate__RECO_obj_pt_refplot integral is zero

plotting variable: recoRecoEcalCandidates_hfRecoEcalCandidate__RECO.obj.eta()
Error in TH1F::KolmogorovTest: Histogram1 recoRecoEcalCandidates_hfRecoEcalCandidate__RECO_obj_eta_refplot integral is zero

plotting variable: recoRecoEcalCandidates_hfRecoEcalCandidate__RECO.obj.phi()
Error in TH1F::KolmogorovTest: Histogram1 recoRecoEcalCandidates_hfRecoEcalCandidate__RECO_obj_phi_refplot integral is zero

Branch LumiDetails_lumiProducer__RECO.obj is not found in either of the inputs. Skip.
Branch .obj is not found in either of the inputs. Skip.
DONE calling validate

[quark2]

Hi @qliphy,

I think these messages are not related to this PR. It's only about GEM onlineDQM, and it doesn't have any effect on ECAL and luminosity and so on. Other messages in the log file related to GEM are not related to GEM onlineDQM. I have no idea why these messages come out (lack of statistics? or missed under/overflow bins?), but these are not in the coverage of GEM onlineDQM, I think.

Best regards,
Byeonghak Ko

[qliphy]

please test
to check again for the differences as mentioned above

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6bff3c/23081/summary.html
COMMIT: 5bcea1f
CMSSW: CMSSW_12_4_X_2022-03-13-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37135/23081/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3695377
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3695353
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 204 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[qliphy]

+1
the unrelated differences seem to be clear now