The datastore-web-app is a React JavaScript web app for navigating data managed by the MPEX datastore Java server applications. These applications offer APIs for storing and retrieving data captured from a running particle accelerator (or other large experimental facility), providing a platform for building machine learning and other analytics tools for use within control systems at those facilities. The datastore application provides an ingestion service for capturing incoming data, and the datastore-service application offers a query service.
The primary objective for the initial version is to expose the main features of the datastore server API in a web application. The app uses completely unstyled HTML components. The look and feel will be addressed (soon) as a follow on task by incorporating a component library like React Bootstrap or MUI.
The server captures data "snapshots" generated by instruments within the research facility. These are essentially tables of data, where the cells can hold a variety of data types ranging from simple scalar values to images. The snapshot ingestion server utilizes InfluxDB to manage time series data and MongoDB to manage metadata about the snapshots.
The "columns" in each snapshot correspond to Process Variables (PVs) in the Experimental Physics and Industrial Control System (EPICS) environment . Each "row" in the snapshot contains data captured at some timestamp across the set of PVs. The sampling frequency depends on the instrument capturing the data but can be up to 1000 samples per second.
The datastore server API provides features for storing snapshots, tagging them with metadata, and retrieving snapshot and PV information by time range and metadata attributes. The web application reflects the server's query interface, and provides the following pages:
- HomePage - Landing page for the application with navigation controls for the various pages.
- SnapshotListPage - Provides controls for creating a query to retrieve a list of snapshots by time range, metadata attributes, or PV names (e.g., find snapshots containing a particular PV name or PV name pattern), submitting the query, and viewing its results in a table.
- SnapshotPage - Provides controls for navigating an individual snapshot, including its metadata attributes, and filtering the snapshot time series data by time range or PV name. Provides mechanism for paging through filtered snapshot data.
- PvListPage - Allows user to build query by PV name regex pattern and displays the result in a table.
- PvPage - Displays information about an individual PV including metadata attributes and a list of snapshots that include data for that PV.
- AnnotationListPage - Allows a user to query tags used for categorizing snapshots and view the result in a table. Annotations are used, for example, to identify the data for an exceptional event in the accelerator, such as a power outage.
- AnnotationPage - Displays details for an individual annotation, including its name and how it is mapped to timeseries/PV data.
The React web application runs entirely in the browser, but I wanted to support browser URL navigation, history, and bookmarks like in a server application so I used React Router to update the browser URL as the user navigates the web application, and to allow the user to navigate the application by URL bookmarks.
The datastore Java server applications use the gRPC high performance remote procedure call (RPC) framework for communication. There are gRPC bindings for practically any programming language that you can imagine, but getting this to work from JavaScript running in the browser was a bit messy. I used the grpc-webproject's protoc compiler to generate JavaScript stubs from the Java server application gRPC API protocol buffer files.
TODO: I need to incorporate automation for the protoc compilation step into the Javascript project's build environment, or at least add documentation here for how I did this. It was a bit challenging to make this work as I had to compile protoc from source files etc…
The web application can be deployed as a standalone React web application, or via a Node.js server application. Each is discussed in more detail below.
It was created using create-react-app (CRA) so the standard utilities for running a CRA app can be used to build and run the application:
- npm start - Runs the app in development mode using the webpack development server. Open http://localhost:3000 to view it in your browser. The nice thing about this sort of deployment is that changes to the app are detected automatically, and application instances running in the browser are updated.
- npm test - Launches the test runner in interactive watch mode. I need to add test coverage so there isn't much going on here, yet.
- npm run build - This initiates a "production" build of the application, bundled and optimized for the best performance. This can be deployed by a simple static file server like npm serve, or by a Node.js server. The latter option is discussed in more detail below.
The package.json file includes a development proxy configuration, ("proxy": "http://localhost:9000"). This is needed when the web app is running under the dev server (on port 3000) to avoid "Cross-Origin Resource Sharing" (CORS) issues because the server is running on a different network port. This configuration tells the development server to proxy any unknown requests to the API server, avoiding the need to set up an explicit redirect for use during development.
The production build of the application can be deployed via a Node.js server, such as the simple server in the datastore-server-app repo. I'm using this approach so that the web app can be deployed without a static Apache web server (and IT help), and so that we can augment the datastore Java server APIs with additional functionality. Currently the server only provides a config API for retrieving the gRPC host/port connection details and a placeholder authentication API.
The server passes any unhandled route to the web application e.g.,:
// auth routes
this.app.use(this.paths.auth, require("../routes/auth"));
// config routes
this.app.use(this.paths.config, require("../routes/config"));
// catch all requests that don't match other routes
this.app.get("*", (req, res) => {
res.sendFile(
path.join(__dirname, '../client/build/index.html')
);
});
When the Node.js server is running, the server can be accessed via http://localhost:9000 . See the server repo for additional details.
All source files are contained in the datastore-web-app/src directory. There are two subdirectories there, one for page components and one for supporting domain classes.
In the datastore-web-app/src/pages directory are subdirectories for each page listed above. Each subdirectory includes files for the panels included in that page. The "common" subdirectory includes components that are used in multiple pages.
The datastore-web-app/src/domain directory includes subdirectories to organize various types of domain classes used within the application, including:
- filter - Contains classes to build a QueryFilter object for use in the API interface that contains various types of criteria (time range, attribute, PV, annotation, etc).
- grpc-client - "DatastoreApi" is the primary model used by the web application, and contains functions that invoke various API methods to fulfill queries generated by the app's pages. The "grpc-proto" subdirectory contains the protoc-generated gRPC JavaScript stubs for calling the server API.
- models - Contains classes to represent various domain objects in the API, including Snapshot, PV, and Annotation in addition to some special classes for paging through Snapshot data time series.
- node-api - Includes functions for calling APIs provided by the Node.js server. Currently there is only a config API, for the web app to get gRPC host/port connection details for creating a new connection. This is really just a placeholder for now, as I expect will be adding additional APIs to the Node.js server to augment the datastore Javva server APIs.
- utils - Includes "browser-utils" with some convenience methods for accessing browser storage from React, and "timestamp-utils" to manipulate the Unix epoch timestamps contained in the snapshot data.