The time series domain focuses primarily on receiving, validating, storing and distributing time series data to relevant market participants to be used for billing, etc.
A time series is a message containing a collection of measurements for a given Metering Point and it documents the amount of electricity being consumed, produced or exchanged. The measurements received can come in different resolutions e.g. hourly or 15 minutes values, although the domain is built with the intend for higher resolutions like 5 minutes or higher.
Performance is essential for this time series engine as the expected through-put will be high. Current performance target is processing 16 million time series values per hour, but this figure is expected to be raised significantly.
The domain is also in charge of responding to time series data requests from market participants, and it is a key input data provider for the calculations performed by the wholesale domain in order to settle the electricity market.
These are the business processes maintained by this domain.
| Processes |
|---|
| Submission of time series data |
| Request for time series data |
| Request for historical time series data |
| ... |
Upon receipt of a time series message, the data gets processed within Databricks. In Databricks, jobs and libraries implemented in Python takes care of validating the data before storing the data in a Delta Lake. Finally, the validated time series data are published to the wholesale domain for use in their work.
This sections documents the data flows to and from the time series domain.
In the current program increment (PI) the planned work is captured by the stated PI goal:
- When a Grid Access Provider submits a time series for a metering point, the Time Series domain stores it, so that the data is available for calculations done by the wholesale domain.
Learn how to get started with Green Energy Hub here.
Getting started with Databricks development
Required or Recommended tools:
- Azure CLI
- Terraform for infrastructure as code
RiderorVisual Studiofor .NET development- .NET 5 SDK
- Visual Studio Code for PySpark streaming development
- Docker
Read about the community for Green Energy Hub here and learn about how to get involved and get help.
Please note that we have provided a Dictionary to help understand many of the terms used throughout the repository.