Introduce fnmatch in a DQM script #28808

fioriNTU · 2020-01-28T11:15:24Z

PR description:

This PR add the possibility to use wildcards in the script which extract DQMIO data in the form of SQMLITE tables

PR validation:

This PR has been validated in a private setup successfully. The standard PR test will not run this code, which is a standalone script, so I expect no changes at all in whatever comparison due to this PR

fioriNTU · 2020-01-28T11:15:42Z

please test

cmsbuild · 2020-01-28T11:15:50Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-28T11:20:42Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28808/13511

This PR adds an extra 12KB to repository

cmsbuild · 2020-01-28T11:22:23Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4380/console Started: 2020/01/28 12:30

cmsbuild · 2020-01-28T11:22:24Z

A new Pull Request was created by @fioriNTU for master.

It involves the following packages:

DQMServices/Components

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2020-01-28T12:39:44Z

+1
Tested at: d7a5b0f
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-210f06/4380/summary.html
CMSSW: CMSSW_11_1_X_2020-01-27-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-01-28T12:39:47Z

Comparison job queued.

cmsbuild · 2020-01-28T14:39:09Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-210f06/4380/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2697090
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2696743
DQMHistoTests: Total skipped: 346
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

fioriNTU · 2020-01-28T14:48:54Z

+1

cmsbuild · 2020-01-28T14:49:22Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-01-29T17:15:53Z

DQMServices/Components/scripts/dqmiodatasetharvest.py

-interesting_mes = {
-  "PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_1",
-  "PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_2",
-  "PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_3",
-  "PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_4",
-  "PixelPhase1/Phase1_MechanicalView/PXForward/digi_occupancy_per_SignedDiskCoord_per_SignedBladePanelCoord_PXRing_1",
-  "PixelPhase1/Phase1_MechanicalView/PXForward/digi_occupancy_per_SignedDiskCoord_per_SignedBladePanelCoord_PXRing_2",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_1/TkHMap_NumberValidHits_TECM_W1",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_2/TkHMap_NumberValidHits_TECM_W2",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_3/TkHMap_NumberValidHits_TECM_W3",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_4/TkHMap_NumberValidHits_TECM_W4",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_5/TkHMap_NumberValidHits_TECM_W5",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_6/TkHMap_NumberValidHits_TECM_W6",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_7/TkHMap_NumberValidHits_TECM_W7",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_8/TkHMap_NumberValidHits_TECM_W8",
-  "SiStrip/MechanicalView/TEC/MINUS/wheel_9/TkHMap_NumberValidHits_TECM_W9",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_1/TkHMap_NumberValidHits_TECP_W1",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_2/TkHMap_NumberValidHits_TECP_W2",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_3/TkHMap_NumberValidHits_TECP_W3",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_4/TkHMap_NumberValidHits_TECP_W4",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_5/TkHMap_NumberValidHits_TECP_W5",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_6/TkHMap_NumberValidHits_TECP_W6",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_7/TkHMap_NumberValidHits_TECP_W7",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_8/TkHMap_NumberValidHits_TECP_W8",
-  "SiStrip/MechanicalView/TEC/PLUS/wheel_9/TkHMap_NumberValidHits_TECP_W9",
-  "SiStrip/MechanicalView/TIB/layer_1/TkHMap_NumberValidHits_TIB_L1",
-  "SiStrip/MechanicalView/TIB/layer_2/TkHMap_NumberValidHits_TIB_L2",
-  "SiStrip/MechanicalView/TIB/layer_3/TkHMap_NumberValidHits_TIB_L3",
-  "SiStrip/MechanicalView/TIB/layer_4/TkHMap_NumberValidHits_TIB_L4",
-  "SiStrip/MechanicalView/TID/MINUS/wheel_1/TkHMap_NumberValidHits_TIDM_D1",
-  "SiStrip/MechanicalView/TID/MINUS/wheel_2/TkHMap_NumberValidHits_TIDM_D2",
-  "SiStrip/MechanicalView/TID/MINUS/wheel_3/TkHMap_NumberValidHits_TIDM_D3",
-  "SiStrip/MechanicalView/TID/PLUS/wheel_1/TkHMap_NumberValidHits_TIDP_D1",
-  "SiStrip/MechanicalView/TID/PLUS/wheel_2/TkHMap_NumberValidHits_TIDP_D2",
-  "SiStrip/MechanicalView/TID/PLUS/wheel_3/TkHMap_NumberValidHits_TIDP_D3",
-  "SiStrip/MechanicalView/TOB/layer_1/TkHMap_NumberValidHits_TOB_L1",
-  "SiStrip/MechanicalView/TOB/layer_2/TkHMap_NumberValidHits_TOB_L2",
-  "SiStrip/MechanicalView/TOB/layer_3/TkHMap_NumberValidHits_TOB_L3",
-  "SiStrip/MechanicalView/TOB/layer_4/TkHMap_NumberValidHits_TOB_L4",
-  "SiStrip/MechanicalView/TOB/layer_5/TkHMap_NumberValidHits_TOB_L5",
-  "SiStrip/MechanicalView/TOB/layer_6/TkHMap_NumberValidHits_TOB_L6",
-  "EcalBarrel/EBOccupancyTask/EBOT digi occupancy",
-  "EcalEndcap/EEOccupancyTask/EEOT digi occupancy EE -",
-  "EcalEndcap/EEOccupancyTask/EEOT digi occupancy EE +",
-  "EcalPreshower/ESOccupancyTask/ES Energy Density Z -1 P 1",
-  "EcalPreshower/ESOccupancyTask/ES Energy Density Z -1 P 2",
-  "EcalPreshower/ESOccupancyTask/ES Energy Density Z 1 P 1",
-  "EcalPreshower/ESOccupancyTask/ES Energy Density Z 1 P 2",
-  "Hcal/DigiRunHarvesting/Occupancy/depth/depth1",
-  "Hcal/DigiRunHarvesting/Occupancy/depth/depth2",
-  "Hcal/DigiRunHarvesting/Occupancy/depth/depth3",
-  "Hcal/DigiRunHarvesting/Occupancy/depth/depth4",
-  "CSC/CSCOfflineMonitor/Occupancy/hOStripsAndWiresAndCLCT",
-  "RPC/AllHits/SummaryHistograms/Occupancy_for_Barrel",
-  "RPC/AllHits/SummaryHistograms/Occupancy_for_Endcap",
-  "DT/02-Segments/Wheel-1/numberOfSegments_W-1",
-  "DT/02-Segments/Wheel-2/numberOfSegments_W-2",
-  "DT/02-Segments/Wheel0/numberOfSegments_W0",
-  "DT/02-Segments/Wheel1/numberOfSegments_W1",
-  "DT/02-Segments/Wheel2/numberOfSegments_W2",
-
-  "L1T/L1TObjects/L1TEGamma/timing/egamma_eta_phi_bx_0",
-  "L1T/L1TObjects/L1TJet/timing/jet_eta_phi_bx_0",
-  "L1T/L1TObjects/L1TMuon/timing/muons_eta_phi_bx_0",
-  "L1T/L1TObjects/L1TTau/timing/tau_eta_phi_bx_0",
-  "L1T/L1TObjects/L1TEGamma/timing/denominator_egamma",
-  "L1T/L1TObjects/L1TJet/timing/denominator_jet",
-  "L1T/L1TObjects/L1TMuon/timing/denominator_muons",
-  "L1T/L1TObjects/L1TTau/timing/denominator_tau",
-}


These names do not match "PixelPhase1/Phase1_MechanicalView/PXBarrel/adc_PXLayer*".
Are you removing them on purpurse? If yes, could update the PR description specifying what you are changing?

silviodonato · 2020-01-29T17:22:48Z

DQMServices/Components/scripts/dqmiodatasetharvest.py

+
+            if mename.find("AlCaReco") != -1: 
+              continue
+
+            if mename.find("Isolated") != -1:
+              continue
+
+            if mename.find("HLT") != -1:
+              continue
+
+            if not ((mename.find("SiStrip") >= 0) or (mename.find("OfflinePV") >= 0) or (mename.find("PixelPhase1") >= 0) or (mename.find("Tracking") >= 0 )):    
+              continue
+
+            if check_interesting(mename):


Do you have any special reasons to apply a veto on "AlCaReco", "Isolated", "HLT" and requiring at least one match among "SiStrip", "OfflinePV", "PixelPhase1", "Tracking"? I expected you specify them directly with "interesting_mes"

well, no, I couldn't do it in interesting_mes once added the fnmatch. The point is that AlCaReco, IsolatedBunches and HLT workspaces all contains a SiStrip folder, which is a copy with different selections of the original SIStrip folder in the GUI. Not adding those lines I would collect also the histograms with the same name in the other folders. This could be removed for the general case, I left these on purpose to remind people, there can be cases such this one.

silviodonato · 2020-01-30T08:57:27Z

Thanks @fioriNTU. Could you also confirm that you are removing many "interesting_mes" on purpose (see the other comment above)?

fioriNTU · 2020-01-30T09:03:37Z

@silviodonato , yes I am removing a lot of interesting_mes on purpose, the reason is that the choice of the particular histogram to be extracted will mostly depend by the specific use case.

silviodonato · 2020-01-30T09:18:26Z

+1

fioriNTU added 2 commits January 28, 2020 12:03

adding fnmatch

b3dd69b

deleting an extra line

d7a5b0f

cmsbuild added this to the CMSSW_11_1_X milestone Jan 28, 2020

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Jan 28, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 28, 2020

cmsbuild added tests-started and removed tests-pending labels Jan 28, 2020

fioriNTU changed the title ~~Sqlit edump~~ Introduce fnmatch in a DQM script Jan 28, 2020

cmsbuild added tests-approved and removed tests-started labels Jan 28, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jan 28, 2020

cmsbuild added dqm-approved and removed dqm-pending pending-signatures labels Jan 28, 2020

cmsbuild added the fully-signed label Jan 28, 2020

silviodonato reviewed Jan 29, 2020

View reviewed changes

cmsbuild added orp-approved and removed orp-pending labels Jan 30, 2020

cmsbuild merged commit 23e8336 into cms-sw:master Jan 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce fnmatch in a DQM script #28808

Introduce fnmatch in a DQM script #28808

fioriNTU commented Jan 28, 2020

fioriNTU commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020 •

edited

Loading

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

fioriNTU commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

silviodonato Jan 29, 2020

silviodonato Jan 29, 2020

fioriNTU Jan 29, 2020

silviodonato commented Jan 30, 2020

fioriNTU commented Jan 30, 2020

silviodonato commented Jan 30, 2020

Introduce fnmatch in a DQM script #28808

Introduce fnmatch in a DQM script #28808

Conversation

fioriNTU commented Jan 28, 2020

PR description:

PR validation:

fioriNTU commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020 • edited Loading

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

fioriNTU commented Jan 28, 2020

cmsbuild commented Jan 28, 2020

silviodonato Jan 29, 2020

Choose a reason for hiding this comment

silviodonato Jan 29, 2020

Choose a reason for hiding this comment

fioriNTU Jan 29, 2020

Choose a reason for hiding this comment

silviodonato commented Jan 30, 2020

fioriNTU commented Jan 30, 2020

silviodonato commented Jan 30, 2020

cmsbuild commented Jan 28, 2020 •

edited

Loading