Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple bug fixes and improvements #7

Merged
merged 3 commits into from
Mar 4, 2020
Merged

Multiple bug fixes and improvements #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
915e9ac
Fixed oms extractor
Mar 3, 2020
50465ab
Fixed OMS and RR extraction
Mar 3, 2020
d5cebf5
Fixed a bug where with from/to_run runs were not filtered
Mar 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 17 additions & 9 deletions backend/api/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,15 +80,21 @@ def data():
processing_string = rows[0]['processing_string']
series = rows[0]['name']

if (from_run == None or to_run == None) and runs == None:
if latest == None:
latest = 50
if runs == None:
# Will need filtering
runs_filter = ''
latest_filter = ''
if from_run != None and to_run != None:
runs_filter = 'AND run >= %s AND run <= %s' % (from_run, to_run)
else:
if latest == None:
latest = 50
latest_filter = 'LIMIT %s' % latest

run_class_like = '%%collision%%'
if 'cosmic' in pd.lower():
run_class_like = '%%cosmic%%'

# Get latest runs for specific user selection

sql = '''
SELECT run FROM oms_data_cache
WHERE run IN
Expand All @@ -100,15 +106,17 @@ def data():
)
AND oms_data_cache.run_class %s :run_class
AND oms_data_cache.significant=%s
AND oms_data_cache.is_dcs=%s
%s
ORDER BY run DESC
LIMIT :latest
%s
;
''' % (db_access.ilike_crossdb(), db_access.true_crossdb())
''' % (db_access.ilike_crossdb(), db_access.true_crossdb(), db_access.true_crossdb(), runs_filter, latest_filter)

print('Getting the list of runs...')
start = timeit.default_timer()

rows = execute_with_retry(session, sql, { 'subsystem': subsystem, 'pd': pd, 'processing_string': processing_string, 'run_class': run_class_like, 'latest': latest })
rows = execute_with_retry(session, sql, { 'subsystem': subsystem, 'pd': pd, 'processing_string': processing_string, 'run_class': run_class_like })
rows = list(rows)

stop = timeit.default_timer()
Expand Down Expand Up @@ -293,9 +301,9 @@ def expand_url():

rows = list(execute_with_retry(session, sql, {'id': data_point_id}))
url = rows[0][url_type]
url = url.replace('+', '%2B')

if url:
url = url.replace('+', '%2B')
return redirect(url, code=302)
else:
return jsonify({'message': 'Requested URL type is not found.'}), 404
Expand Down
4 changes: 2 additions & 2 deletions backend/db_access.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,9 @@ class OMSDataCache(base):
l1_rate = Column(Float)
hlt_physics_rate = Column(Float)
duration = Column(Integer, nullable=False)
fill_number = Column(Integer, nullable=False)
fill_number = Column(Integer, default=0)
injection_scheme = Column(String)
era = Column(String, nullable=False)
era = Column(String)

run_class = Column(String)
significant = Column(Boolean, nullable=False, default=False)
Expand Down
105 changes: 85 additions & 20 deletions backend/extractor/oms_extractor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ def fetch(update, nproc):
db_access.dispose_engine()
pool = Pool(nproc)
pool.map(fetch_run, diff)
print('Done.')


def fetch_run(run):
Expand All @@ -62,11 +63,65 @@ def fetch_run(run):

fills_url = 'https://cmsoms.cern.ch/agg/api/v1/fills?filter[fill_number][eq]=%s&fields=injection_scheme,era' % oms_runs_json['data'][0]['attributes']['fill_number']
oms_fills_json = json.loads(requests.get(fills_url, cookies=cookies, verify=CACERT).text)

dcs_lumisections_url = 'https://cmsoms.cern.ch/agg/api/v1/lumisections?filter[run_number][eq]=%s&filter[cms_active][eq]=true&filter[bpix_ready][eq]=true&filter[fpix_ready][eq]=true&filter[tibtid_ready][eq]=true&filter[tecm_ready][eq]=true&filter[tecp_ready][eq]=true&filter[castor_ready][eq]=true&filter[tob_ready][eq]=true&filter[ebm_ready][eq]=true&filter[ebp_ready][eq]=true&filter[eem_ready][eq]=true&filter[eep_ready][eq]=true&filter[esm_ready][eq]=true&filter[esp_ready][eq]=true&filter[hbhea_ready][eq]=true&filter[hbheb_ready][eq]=true&filter[hbhec_ready][eq]=true&filter[hf_ready][eq]=true&filter[ho_ready][eq]=true&filter[dtm_ready][eq]=true&filter[dtp_ready][eq]=true&filter[dt0_ready][eq]=true&filter[cscm_ready][eq]=true&filter[cscp_ready][eq]=true&filter[rpc_ready][eq]=true&fields=run_number&page[limit]=1' % run

# TODO: Change to prod url, add cookies and certificate
dcs_collisions_lumis_url = '''https://vocms0183.cern.ch/agg/api/v1/lumisections?filter[run_number][eq]=%s
&filter[cms_active][eq]=true
&filter[bpix_ready][eq]=true
&filter[fpix_ready][eq]=true
&filter[tibtid_ready][eq]=true
&filter[tecm_ready][eq]=true
&filter[tecp_ready][eq]=true
&filter[tob_ready][eq]=true
&filter[ebm_ready][eq]=true
&filter[ebp_ready][eq]=true
&filter[eem_ready][eq]=true
&filter[eep_ready][eq]=true
&filter[esm_ready][eq]=true
&filter[esp_ready][eq]=true
&filter[hbhea_ready][eq]=true
&filter[hbheb_ready][eq]=true
&filter[hbhec_ready][eq]=true
&filter[hf_ready][eq]=true
&filter[ho_ready][eq]=true
&filter[dtm_ready][eq]=true
&filter[dtp_ready][eq]=true
&filter[dt0_ready][eq]=true
&filter[cscm_ready][eq]=true
&filter[cscp_ready][eq]=true
&fields=run_number
&page[limit]=1''' % run

dcs_cosmics_lumis_url1 = '''https://vocms0183.cern.ch/agg/api/v1/lumisections
?filter[run_number][eq]=%s
&filter[dtm_ready][eq]=true
&filter[dtp_ready][eq]=true
&filter[dt0_ready][eq]=true
&fields=run_number
&page[limit]=1''' % run

dcs_cosmics_lumis_url2 = '''https://vocms0183.cern.ch/agg/api/v1/lumisections
?filter[run_number][eq]=%s
&filter[cscm_ready][eq]=true
&filter[cscp_ready][eq]=true
&fields=run_number
&page[limit]=1''' % run

is_dcs = False

# For cosmics, we need to OR two terms. If first query if false, make a second one.
if 'cosmic' in oms_runs_json['data'][0]['attributes']['hlt_key']:
dcs_lumisections_url = 'https://cmsoms.cern.ch/agg/api/v1/lumisections?filter[run_number][eq]=%s&filter[tibtid_ready][eq]=true&filter[tecm_ready][eq]=true&filter[tecp_ready][eq]=true&filter[tob_ready][eq]=true&filter[dtm_ready][eq]=true&filter[dtp_ready][eq]=true&filter[dt0_ready][eq]=true&fields=run_number&page[limit]=1' % run
dcs_lumisections_json = json.loads(requests.get(dcs_lumisections_url, cookies=cookies, verify=CACERT).text)
dcs_lumisections_json = json.loads(requests.get(dcs_cosmics_lumis_url1.replace('\n', ''), verify=False).text)
if len(dcs_lumisections_json['data']):
is_dcs = True
else:
dcs_lumisections_json = json.loads(requests.get(dcs_cosmics_lumis_url2.replace('\n', ''), verify=False).text)
if len(dcs_lumisections_json['data']):
is_dcs = True
else:
dcs_lumisections_json = json.loads(requests.get(dcs_collisions_lumis_url.replace('\n', ''), verify=False).text)
if len(dcs_lumisections_json['data']):
is_dcs = True

# Add to cache
session = db_access.get_session()
Expand All @@ -87,7 +142,7 @@ def fetch_run(run):
hlt_physics_rate = oms_runs_json['data'][0]['attributes']['hlt_physics_rate'],
duration = oms_runs_json['data'][0]['attributes']['duration'],
fill_number = oms_runs_json['data'][0]['attributes']['fill_number'],
is_dcs = True if len(dcs_lumisections_json['data']) > 0 else False
is_dcs = is_dcs
)

try:
Expand All @@ -103,28 +158,38 @@ def fetch_run(run):
print('IntegrityError inserting OMS item: %s' % e)
session.rollback()
print('Updating...')

try:
oms_item_existing = session.query(db_access.OMSDataCache).filter(
db_access.OMSDataCache.run == oms_item.run,
db_access.OMSDataCache.lumi == oms_item.lumi
).one_or_none()

if oms_item_existing:
oms_item_existing.start_time = datetime.datetime.strptime(oms_item.start_time, "%Y-%m-%dT%H:%M:%SZ"),
oms_item_existing.end_time = datetime.datetime.strptime(oms_item.end_time, "%Y-%m-%dT%H:%M:%SZ"),
oms_item_existing.b_field = oms_item.b_field,
oms_item_existing.energy = oms_item.energy,
oms_item_existing.delivered_lumi = oms_item.delivered_lumi,
oms_item_existing.end_lumi = oms_item.end_lumi,
oms_item_existing.recorded_lumi = oms_item.recorded_lumi,
oms_item_existing.l1_key = oms_item.l1_key,
oms_item_existing.l1_rate = oms_item.l1_rate,
oms_item_existing.hlt_key = oms_item.hlt_key,
oms_item_existing.hlt_physics_rate = oms_item.hlt_physics_rate,
oms_item_existing.duration = oms_item.duration,
oms_item_existing.fill_number = oms_item.fill_number,
oms_item_existing.injection_scheme = oms_item.injection_scheme,
oms_item_existing.era = oms_item.era,
if isinstance(oms_item.start_time, datetime.datetime):
oms_item_existing.start_time = oms_item.start_time
else:
oms_item_existing.start_time = datetime.datetime.strptime(oms_item.start_time, "%Y-%m-%dT%H:%M:%SZ")

if isinstance(oms_item.end_time, datetime.datetime):
oms_item_existing.end_time = oms_item.end_time
else:
oms_item_existing.end_time = datetime.datetime.strptime(oms_item.end_time, "%Y-%m-%dT%H:%M:%SZ")

oms_item_existing.b_field = oms_item.b_field
oms_item_existing.energy = oms_item.energy
oms_item_existing.delivered_lumi = oms_item.delivered_lumi
oms_item_existing.end_lumi = oms_item.end_lumi
oms_item_existing.recorded_lumi = oms_item.recorded_lumi
oms_item_existing.l1_key = oms_item.l1_key
oms_item_existing.l1_rate = oms_item.l1_rate
oms_item_existing.hlt_key = oms_item.hlt_key
oms_item_existing.hlt_physics_rate = oms_item.hlt_physics_rate
oms_item_existing.duration = oms_item.duration
oms_item_existing.fill_number = oms_item.fill_number
oms_item_existing.injection_scheme = oms_item.injection_scheme
oms_item_existing.era = oms_item.era
oms_item_existing.is_dcs = oms_item.is_dcs
session.commit()
print('Updated.')
except Exception as e:
Expand Down
18 changes: 11 additions & 7 deletions backend/extractor/rr_extractor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
CACERT='../api/etc/cern_cacert.pem'
PREMADE_COOKIE='../api/etc/rr_sso_cookie.txt'

MIN_LUMI_FOR_COLLISIONS = 100
MIN_LUMI_FOR_COLLISIONS = 0.1
MIN_DURATION_FOR_COSMICS = 3600

def fetch(update):
Expand Down Expand Up @@ -70,17 +70,21 @@ def fetch_runs(min_run, max_run):

for run in result_json['runs']:
# Logic to determine if run is significant for HDQM:
# 1. For collision runs integrated luminosity has to be greater than 100
# 2. For cosmic runs duration has to be longer that 1 hour
# 1. If HLT key contains string "special", run is not significant
# 2. For collision runs integrated luminosity has to be greater than 0.1 1/pb
# 3. For cosmic runs duration has to be longer that 1 hour
significant = db_access.false_crossdb()

if 'special' in run['oms_attributes']['hlt_key'].lower():
significant = db_access.false_crossdb()

# For collision runs:
if 'collisions' in run['rr_attributes']['class'].lower():
if run['oms_attributes']['delivered_lumi'] >= MIN_LUMI_FOR_COLLISIONS:
elif 'collision' in run['rr_attributes']['class'].lower():
if run['oms_attributes']['recorded_lumi'] >= MIN_LUMI_FOR_COLLISIONS:
significant = db_access.true_crossdb()

# For cosmic runs:
elif 'cosmics' in run['rr_attributes']['class'].lower():
elif 'cosmic' in run['rr_attributes']['class'].lower():
if run['oms_attributes']['duration'] >= MIN_DURATION_FOR_COSMICS:
significant = db_access.true_crossdb()

Expand Down Expand Up @@ -122,7 +126,7 @@ def get_sso_cookie(url):

if __name__ == '__main__':
parser = argparse.ArgumentParser(description='DCS data extraction from the RR API.')
parser.add_argument('-u', dest='update', type=bool, default=False, help='If True, all runs will be updated. Otherwise only missing runs will be fetched.')
parser.add_argument('-u', dest='update', action='store_true', help='If True, all runs will be updated. Otherwise only missing runs will be fetched. Default is False.')
args = parser.parse_args()

update = args.update
Expand Down
35 changes: 18 additions & 17 deletions frontend/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,22 +110,22 @@
<input class="form-check-input" type="checkbox" id="option-show-run-duration" value="option3" onchange="optionsController.optionToggled(this)" checked>
<label class="form-check-label" for="option-show-run-duration">Show run duration</label>
</div>
<div class="form-check form-check-inline">
<input class="form-check-input" type="checkbox" id="option-show-regression-lines" value="option4" onchange="optionsController.optionToggled(this)" checked>
<label class="form-check-label" for="option-show-regression-lines">Show regression lines</label>
</div>
<div class="form-check form-check-inline">
<input class="form-check-input" type="checkbox" id="option-show-xrange" value="option5" onchange="optionsController.optionToggled(this)">
<label class="form-check-label" for="option-show-xrange">Bin width proportional to run duration</label>
</div>
<div class="form-check form-check-inline">
<input class="form-check-input" type="checkbox" id="option-show-int-lumi" value="option6" onchange="optionsController.optionToggled(this)">
<label class="form-check-label" for="option-show-int-lumi">Bin width proportional to integrated luminosity</label>
<label class="form-check-label" for="option-show-int-lumi">Bin width proportional to delivered luminosity</label>
</div>
<div class="form-check form-check-inline">
<input class="form-check-input" type="checkbox" id="option-show-datetime" value="option7" onchange="optionsController.optionToggled(this)">
<label class="form-check-label" for="option-show-datetime">Show datetime plot</label>
</div>
<div class="form-check form-check-inline">
<input class="form-check-input" type="checkbox" id="option-show-regression-lines" value="option4" onchange="optionsController.optionToggled(this)" checked>
<label class="form-check-label" for="option-show-regression-lines">Show regression lines (only for correlation plots)</label>
</div>
</div>
</div>

Expand Down Expand Up @@ -361,9 +361,8 @@
<div class="row">
<div class="col-auto ml-2 mt-2">
<input type="checkbox" class="form-check-input" id="fs-correlation-checkbox">
<label class="form-check-label" for="fs-correlation-checkbox">Correlation plot</label>
<label class="form-check-label" for="fs-correlation-checkbox">Correlation plot (only for 2 trends)</label>
<div class="invalid-feedback" id="fs-add-series-error">
Please provide a valid city.
</div>
</div>
</div>
Expand Down Expand Up @@ -418,17 +417,21 @@
<td id="fs-value-duration"></td>
</tr>
<tr>
<td>Delivered luminosity</td>
<td>Delivered lumi</td>
<td id="fs-value-delivered-lumi"></td>
</tr>
<tr>
<td>B field</td>
<td id="fs-value-b-field"></td>
<td>Recorded lumi</td>
<td id="fs-value-recorded-lumi"></td>
</tr>
<tr>
<td>End luminosity</td>
<td>End lumi</td>
<td id="fs-value-end-lumi"></td>
</tr>
<tr>
<td>B field</td>
<td id="fs-value-b-field"></td>
</tr>
<tr>
<td>Start time</td>
<td id="fs-value-start-time"></td>
Expand Down Expand Up @@ -465,10 +468,6 @@
<td>L1T rate</td>
<td id="fs-value-l1t-rate"></td>
</tr>
<tr>
<td>Recorded lumi</td>
<td id="fs-value-recorded-lumi"></td>
</tr>
</tbody>
</table>
</div>
Expand Down Expand Up @@ -529,7 +528,7 @@ <h5 class="modal-title">How to use the Historic DQM</h5>
<li><b>Show run duration: </b> the run duration in seconds will be visualized as an histogram, each bin corresponding to one run</li>
<li><b>Show regression lines: </b> In case of correlation plot can show or hide a linear regression line</li>
<li><b>Show run duration proportional plot: </b> the bin width of the trends will be proportional to the run duration</li>
<li><b>Show proportional to integrated luminosity: </b> the bin width of the trends will be proportional to the integrated luminosity of the run</li>
<li><b>Show proportional to delivered luminosity: </b> the bin width of the trends will be proportional to the delivered luminosity of the run</li>
<li><b>Show datetime plot: </b> on the x-axis it will show the date and time, each run represented with a bin going from the start time to the end time (note: this trend may contain white spaces in periods where no run is available)</li>
</ul>
hit Submit button again to apply the changes.
Expand All @@ -544,7 +543,9 @@ <h5 class="modal-title">How to use the Historic DQM</h5>
<li>There is a square button in the upper right place of each plot where you can left click and locally save an image of the plot</li>
<li>By clicking <b>Change Ranges</b> you can change both x and y axis ranges of the plot, and see the updated plot immediately</li>
<li><b>NOTE 1: All plots are "linkable"</b>, copy the url to share a specific trend by mail (as an example)</li>
<li><b>NOTE 2:</b> plots are not available for all the Primary Datasets, as an example Muon trend are only produced for SingleMuon and DoubleMuon PDs, while Tracker plots are only produced for StreamExpress and ZeroBias</li>
<li><b>NOTE 2:</b> plots are not available for all the Primary Datasets, as an example Muon trend are only produced for SingleMuon and DoubleMuon PDs, while Tracker plots are only produced for StreamExpress and ZeroBias</li>
<li>The <b>documentation</b> of this tool is available here: <a href="https://github.com/cms-DQM/CentralHDQM#table-of-contents" target="_blank">https://github.com/cms-DQM/CentralHDQM#table-of-contents</a></li>
<li>The <b>twiki page</b> of this tool is available here: <a href="https://twiki.cern.ch/twiki/bin/viewauth/CMS/HDQM" target="_blank">https://twiki.cern.ch/twiki/bin/viewauth/CMS/HDQM</a></li>
</ul>
</div>
</div>
Expand Down
Loading