Enable show raw data and Job Viewer for the Jobs realm #730

jpwhite4 · 2018-11-16T16:20:46Z

This pull request enables the Job Viewer functionality for the Jobs realm. This one is quite big, but mostly new code. I decided to include a minor internal API change (there will be a corresponding pull request for SUPREMM to update to use the new API). The API change is in classes/Rest/Controllers/WarehouseControllerProvider.php and in the JobSearchPanel.js

EDIT - actually two minor API changes - the job viewer is now enabled if a module defines a realm config file in rawstatistics.d/. This is needed because multiple modules can now enable the job viewer

integration tests have been added to test the new code.

Couple of things to think about:

what access controls are appropriate for the jobs realm raw data? My implementation uses the same access controls as the SUPReMM realm based on the rationale that while aggregated jobs data is public, the details are not.
In the current ETLv2 infrastucture forces me to create a seperate definition files for the _by_day stuff even though there is really not much differences. For example:

--- configuration/etl/etl_action_defs.d/jobs/hpc-aggregation-day.json	2018-11-16 11:08:43.000000000 -0500
+++ configuration/etl/etl_action_defs.d/jobs/hpc-aggregation.json	2018-11-16 10:42:34.000000000 -0500
@@ -1,7 +1,7 @@
 {
     "#": "Aggregation of HPC job records and tasks ingested from the XDCDB",
     "table_definition": {
-        "$ref": "${table_definition_dir}/jobs/xdw/jobfact_by_day.json#/table_definition"
+        "$ref": "${table_definition_dir}/jobs/xdw/jobfact_by_.json#/table_definition"
     },

     "#": "The aggregation period query determines which periods need to be aggregated based on added or modified",
@@ -77,8 +77,7 @@
             "node_time": "COALESCE(SUM(task.node_count * ${wallduration_case_statement}), 0)",
             "sum_node_time_squared": "COALESCE(SUM( CAST(POW(task.node_count * ${wallduration_case_statement}, 2) AS DECIMAL(36,4)) ), 0)",
             "sum_weighted_expansion_factor": "SUM( ((task.wallduration + task.waitduration) / task.wallduration) * task.node_count * COALESCE(${wallduration_case_statement}, 0))",
-            "sum_job_weights": "SUM(task.node_count * COALESCE(${wallduration_case_statement}, 0))",
-            "job_id_list": "GROUP_CONCAT(task.job_id)"
+            "sum_job_weights": "SUM(task.node_count * COALESCE(${wallduration_case_statement}, 0))"
         },
         "groupby": [
             "${AGGREGATION_UNIT}_id",

the $ref does not really improve the situation much because the bulk of the file is the column definintions.

open_xdmod/modules/xdmod/integration_tests/scripts/bootstrap.sh

smgallo · 2018-11-16T16:38:52Z

For the duplication of the "by-day" file, do you mean that the bulk is in the records section of the source_query? Something to think about is perhaps being able to reference a JSON object and then add/override additional items.

jpwhite4 · 2018-11-16T17:00:02Z

Yes. I'd like to be able to reference the base file and then just insert an extra couple of columns to the table definition and add one line to the aggregation query.

smgallo · 2018-11-20T13:59:58Z

classes/DataWarehouse/Query/Jobs/JobMetadata.php

@@ -0,0 +1,66 @@
+<?php
+


This class needs a block at the top describing what it does and what data it encapsulates.

smgallo · 2018-11-20T14:01:22Z

classes/DataWarehouse/Query/Jobs/JobMetadata.php

+        );
+    }
+
+    public function getJobSummary($user, $jobid)


Why do all of these methods return empty arrays? If there is a reason it should be documented in this class.

Because there is no job summary data for the Jobs realm.

Ditto for job timeseries data.

OK, lets document that in the class.

smgallo · 2018-11-20T14:03:21Z

classes/DataWarehouse/Query/Query.php

@@ -725,6 +725,15 @@ public function getCountQueryString()
        $data_query .= ") as a WHERE a.total IS NOT NULL";
        return $data_query;
    }
+
+    protected function nextPdoIndex($value)


Need documentation here as well.

smgallo · 2018-11-20T14:08:00Z

classes/OpenXdmod/Migration/DatabasesMigration.php

+     * @param array $scriptOptions the options to pass to the etlv2 class
+     */
+    protected function runEtlv2(array $scriptOptions) {
+        if (empty($scriptOptions['chunk-size-days'])) {


I've created a task in Asana to work on putting code like this into a central location https://app.asana.com/0/806625283031961/917407032604343

smgallo · 2018-11-20T14:21:16Z

open_xdmod/modules/xdmod/build.json

@@ -71,6 +71,7 @@
            "configuration/processor_buckets.json",
            "configuration/rawstatistics.json",
            "configuration/rawstatistics.d",
+            {"configuration/rawstatistics.d/z_jobs.json": "rawstatistics.d/z_jobs.json"},


What does "z_jobs" stand for?

Config files are processed in alphabetical order. If the SUPREMM realm is installed this ensures that the jobs realm is listed after the SUPREMM realm in the realm list in the job viewer. This enforces the principle of least confusion since the default behavour of the job viewer will be to search supremm data (as it always was).

not that it matters, but the convention i usually see for this is\d+_\w

lets encrypt uses {0000 - 9999}_name
puppet apache modules uses {00 - 99}_name

jpwhite4 · 2018-11-27T19:24:51Z

Splitting this into two to make it easier to review. Please look at #733 first.

jpwhite4 requested review from smgallo and plessbd and removed request for smgallo November 16, 2018 16:20

jpwhite4 commented Nov 16, 2018

View reviewed changes

open_xdmod/modules/xdmod/integration_tests/scripts/bootstrap.sh Outdated Show resolved Hide resolved

jpwhite4 force-pushed the jobs_realm_show_raw_data branch from 85e618f to 90b9146 Compare November 16, 2018 16:28

Enable show raw data for the Jobs realm

fcfcff4

jpwhite4 force-pushed the jobs_realm_show_raw_data branch from 77dcd09 to fcfcff4 Compare November 20, 2018 14:21

smgallo suggested changes Nov 20, 2018

View reviewed changes

jpwhite4 closed this Nov 27, 2018

This was referenced Nov 28, 2018

Add show raw data implementation for jobs realm #734

Closed

Show raw data jobs #735

Closed

Add raw data support for the Jobs realm #736

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable show raw data and Job Viewer for the Jobs realm #730

Enable show raw data and Job Viewer for the Jobs realm #730

jpwhite4 commented Nov 16, 2018 •

edited

Loading

smgallo commented Nov 16, 2018

jpwhite4 commented Nov 16, 2018

smgallo Nov 20, 2018

smgallo Nov 20, 2018

jpwhite4 Nov 20, 2018

jpwhite4 Nov 20, 2018

smgallo Nov 20, 2018

smgallo Nov 20, 2018

smgallo Nov 20, 2018

smgallo Nov 20, 2018

jpwhite4 Nov 20, 2018

plessbd Nov 26, 2018 •

edited

Loading

jpwhite4 commented Nov 27, 2018

Enable show raw data and Job Viewer for the Jobs realm #730

Enable show raw data and Job Viewer for the Jobs realm #730

Conversation

jpwhite4 commented Nov 16, 2018 • edited Loading

smgallo commented Nov 16, 2018

jpwhite4 commented Nov 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plessbd Nov 26, 2018 • edited Loading

Choose a reason for hiding this comment

jpwhite4 commented Nov 27, 2018

jpwhite4 commented Nov 16, 2018 •

edited

Loading

plessbd Nov 26, 2018 •

edited

Loading