Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce fnmatch in a DQM script #28808

Merged
merged 2 commits into from
Jan 30, 2020
Merged

Introduce fnmatch in a DQM script #28808

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b3dd69b
adding fnmatch
fioriNTU Jan 28, 2020
d7a5b0f
deleting an extra line
fioriNTU Jan 28, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 32 additions & 75 deletions DQMServices/Components/scripts/dqmiodatasetharvest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import argparse
import subprocess
import multiprocessing

import fnmatch

ROOTPREFIX = "root://cms-xrd-global.cern.ch/"
#ROOTPREFIX = "root://eoscms//eos/cms" # for more local files
Expand All @@ -23,83 +23,27 @@

# we can save a lot of time by only scanning some types, if we know all interesting MEs are of these types.
interesting_types = {
"TH2Fs",
"TH1Fs",
"TH1Ds",
"TH2Fs"
}

interesting_mes = {
"PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_1",
"PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_2",
"PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_3",
"PixelPhase1/Phase1_MechanicalView/PXBarrel/digi_occupancy_per_SignedModuleCoord_per_SignedLadderCoord_PXLayer_4",
"PixelPhase1/Phase1_MechanicalView/PXForward/digi_occupancy_per_SignedDiskCoord_per_SignedBladePanelCoord_PXRing_1",
"PixelPhase1/Phase1_MechanicalView/PXForward/digi_occupancy_per_SignedDiskCoord_per_SignedBladePanelCoord_PXRing_2",
"SiStrip/MechanicalView/TEC/MINUS/wheel_1/TkHMap_NumberValidHits_TECM_W1",
"SiStrip/MechanicalView/TEC/MINUS/wheel_2/TkHMap_NumberValidHits_TECM_W2",
"SiStrip/MechanicalView/TEC/MINUS/wheel_3/TkHMap_NumberValidHits_TECM_W3",
"SiStrip/MechanicalView/TEC/MINUS/wheel_4/TkHMap_NumberValidHits_TECM_W4",
"SiStrip/MechanicalView/TEC/MINUS/wheel_5/TkHMap_NumberValidHits_TECM_W5",
"SiStrip/MechanicalView/TEC/MINUS/wheel_6/TkHMap_NumberValidHits_TECM_W6",
"SiStrip/MechanicalView/TEC/MINUS/wheel_7/TkHMap_NumberValidHits_TECM_W7",
"SiStrip/MechanicalView/TEC/MINUS/wheel_8/TkHMap_NumberValidHits_TECM_W8",
"SiStrip/MechanicalView/TEC/MINUS/wheel_9/TkHMap_NumberValidHits_TECM_W9",
"SiStrip/MechanicalView/TEC/PLUS/wheel_1/TkHMap_NumberValidHits_TECP_W1",
"SiStrip/MechanicalView/TEC/PLUS/wheel_2/TkHMap_NumberValidHits_TECP_W2",
"SiStrip/MechanicalView/TEC/PLUS/wheel_3/TkHMap_NumberValidHits_TECP_W3",
"SiStrip/MechanicalView/TEC/PLUS/wheel_4/TkHMap_NumberValidHits_TECP_W4",
"SiStrip/MechanicalView/TEC/PLUS/wheel_5/TkHMap_NumberValidHits_TECP_W5",
"SiStrip/MechanicalView/TEC/PLUS/wheel_6/TkHMap_NumberValidHits_TECP_W6",
"SiStrip/MechanicalView/TEC/PLUS/wheel_7/TkHMap_NumberValidHits_TECP_W7",
"SiStrip/MechanicalView/TEC/PLUS/wheel_8/TkHMap_NumberValidHits_TECP_W8",
"SiStrip/MechanicalView/TEC/PLUS/wheel_9/TkHMap_NumberValidHits_TECP_W9",
"SiStrip/MechanicalView/TIB/layer_1/TkHMap_NumberValidHits_TIB_L1",
"SiStrip/MechanicalView/TIB/layer_2/TkHMap_NumberValidHits_TIB_L2",
"SiStrip/MechanicalView/TIB/layer_3/TkHMap_NumberValidHits_TIB_L3",
"SiStrip/MechanicalView/TIB/layer_4/TkHMap_NumberValidHits_TIB_L4",
"SiStrip/MechanicalView/TID/MINUS/wheel_1/TkHMap_NumberValidHits_TIDM_D1",
"SiStrip/MechanicalView/TID/MINUS/wheel_2/TkHMap_NumberValidHits_TIDM_D2",
"SiStrip/MechanicalView/TID/MINUS/wheel_3/TkHMap_NumberValidHits_TIDM_D3",
"SiStrip/MechanicalView/TID/PLUS/wheel_1/TkHMap_NumberValidHits_TIDP_D1",
"SiStrip/MechanicalView/TID/PLUS/wheel_2/TkHMap_NumberValidHits_TIDP_D2",
"SiStrip/MechanicalView/TID/PLUS/wheel_3/TkHMap_NumberValidHits_TIDP_D3",
"SiStrip/MechanicalView/TOB/layer_1/TkHMap_NumberValidHits_TOB_L1",
"SiStrip/MechanicalView/TOB/layer_2/TkHMap_NumberValidHits_TOB_L2",
"SiStrip/MechanicalView/TOB/layer_3/TkHMap_NumberValidHits_TOB_L3",
"SiStrip/MechanicalView/TOB/layer_4/TkHMap_NumberValidHits_TOB_L4",
"SiStrip/MechanicalView/TOB/layer_5/TkHMap_NumberValidHits_TOB_L5",
"SiStrip/MechanicalView/TOB/layer_6/TkHMap_NumberValidHits_TOB_L6",
"EcalBarrel/EBOccupancyTask/EBOT digi occupancy",
"EcalEndcap/EEOccupancyTask/EEOT digi occupancy EE -",
"EcalEndcap/EEOccupancyTask/EEOT digi occupancy EE +",
"EcalPreshower/ESOccupancyTask/ES Energy Density Z -1 P 1",
"EcalPreshower/ESOccupancyTask/ES Energy Density Z -1 P 2",
"EcalPreshower/ESOccupancyTask/ES Energy Density Z 1 P 1",
"EcalPreshower/ESOccupancyTask/ES Energy Density Z 1 P 2",
"Hcal/DigiRunHarvesting/Occupancy/depth/depth1",
"Hcal/DigiRunHarvesting/Occupancy/depth/depth2",
"Hcal/DigiRunHarvesting/Occupancy/depth/depth3",
"Hcal/DigiRunHarvesting/Occupancy/depth/depth4",
"CSC/CSCOfflineMonitor/Occupancy/hOStripsAndWiresAndCLCT",
"RPC/AllHits/SummaryHistograms/Occupancy_for_Barrel",
"RPC/AllHits/SummaryHistograms/Occupancy_for_Endcap",
"DT/02-Segments/Wheel-1/numberOfSegments_W-1",
"DT/02-Segments/Wheel-2/numberOfSegments_W-2",
"DT/02-Segments/Wheel0/numberOfSegments_W0",
"DT/02-Segments/Wheel1/numberOfSegments_W1",
"DT/02-Segments/Wheel2/numberOfSegments_W2",

"L1T/L1TObjects/L1TEGamma/timing/egamma_eta_phi_bx_0",
"L1T/L1TObjects/L1TJet/timing/jet_eta_phi_bx_0",
"L1T/L1TObjects/L1TMuon/timing/muons_eta_phi_bx_0",
"L1T/L1TObjects/L1TTau/timing/tau_eta_phi_bx_0",
"L1T/L1TObjects/L1TEGamma/timing/denominator_egamma",
"L1T/L1TObjects/L1TJet/timing/denominator_jet",
"L1T/L1TObjects/L1TMuon/timing/denominator_muons",
"L1T/L1TObjects/L1TTau/timing/denominator_tau",
}
Comment on lines -29 to -98
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These names do not match "PixelPhase1/Phase1_MechanicalView/PXBarrel/adc_PXLayer*".
Are you removing them on purpurse? If yes, could update the PR description specifying what you are changing?

# insert the list of needed histograms below, wild cards are usable
interesting_mes = [

"PixelPhase1/Phase1_MechanicalView/PXBarrel/adc_PXLayer*",

]

inf = re.compile("([- \[])inf([,}\]])")
nan = re.compile("([- \[])nan([,}\]])")

def check_interesting(mename):
for pattern in interesting_mes:
if fnmatch.fnmatch(mename,pattern):
return True
return False

def tosqlite(x):
if isinstance(x, ROOT.string):
try:
Expand Down Expand Up @@ -195,8 +139,7 @@ def harvestfile(fname):
knownlumis.add(lumi)

if not treenames[metype] in interesting_types:
continue

continue

endrun = run # assume no multi-run files for now
if lumi == 0: # per-job ME
Expand All @@ -208,13 +151,27 @@ def harvestfile(fname):
# inclusive range -- for 0 entries, row is left out
firstidx, lastidx = idxtree.FirstIndex, idxtree.LastIndex
metree = getattr(f, treenames[metype])
metree.GetEntry(0)
metree.SetBranchStatus("*",0)
metree.SetBranchStatus("FullName",1)

for x in range(firstidx, lastidx+1):
metree.GetEntry(x)
mename = str(metree.FullName)
if mename in interesting_mes:

if mename.find("AlCaReco") != -1:
continue

if mename.find("Isolated") != -1:
continue

if mename.find("HLT") != -1:
continue

if not ((mename.find("SiStrip") >= 0) or (mename.find("OfflinePV") >= 0) or (mename.find("PixelPhase1") >= 0) or (mename.find("Tracking") >= 0 )):
continue

if check_interesting(mename):
Comment on lines +161 to +174
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any special reasons to apply a veto on "AlCaReco", "Isolated", "HLT" and requiring at least one match among "SiStrip", "OfflinePV", "PixelPhase1", "Tracking"? I expected you specify them directly with "interesting_mes"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, no, I couldn't do it in interesting_mes once added the fnmatch. The point is that AlCaReco, IsolatedBunches and HLT workspaces all contains a SiStrip folder, which is a copy with different selections of the original SIStrip folder in the GUI. Not adding those lines I would collect also the histograms with the same name in the other folders. This could be removed for the general case, I left these on purpose to remind people, there can be cases such this one.

metree.GetEntry(x, 1)
value = metree.Value

Expand Down