This project converts OpenTelemetry (OTel) data into PlantUML activity diagrams. These diagrams are subsequently ingested by plus2json, a tool that transforms them into JSON format. The resulting JSON 'job definitions' are specifically tailored for use with the Protocol Verifier (PV), a tool that monitors and verifies the behaviour of another system, to ensure that it is behaving as expected.
Table of Contents
Although this tool was designed for use with the Protocol Verifier, as a standalone application it offers several benefits for developers, system architects, and DevOps professionals:
-
Visualisation of Complex Systems: Convert abstract OpenTelemetry data into clear, visual PlantUML diagrams, making it easier to understand system interactions and dependencies.
-
Debugging and Troubleshooting: Quickly identify bottlenecks, errors, and unexpected behaviors in your distributed systems by visualising trace data.
-
Documentation: Generate up-to-date system diagrams automatically from actual runtime data, ensuring your documentation accurately reflects the current state of your system.
-
Communication: Use the generated diagrams to facilitate discussions between technical and non-technical stakeholders, improving overall project understanding.
-
Performance Optimisation: Visualise request flows and timing information to identify areas for performance improvements.
-
Microservices Architecture Analysis: Gain insights into how your microservices interact, helping with architectural decisions and optimisations.
-
Integration with Existing Tools: As it works with OpenTelemetry data, it integrates seamlessly with your existing observability stack.
-
Time-Saving: Automate the process of creating system diagrams, saving valuable development time.
-
Training and Onboarding: Use generated diagrams to help new team members understand system architecture and workflows quickly.
By converting OpenTelemetry data to PlantUML diagrams, this tool bridges the gap between raw observability data and easily understandable visual representations, enhancing your ability to develop, maintain, and optimise complex distributed systems.
- Convert OpenTelemetry (OTel) JSON data into detailed PlantUML activity diagrams with flexible output options and multiple subcommands for various processing needs.
- Easily customize and extend the tool to support new data formats or processing requirements through a modular design.
- Identify and extract unique event sequences from OTel data
- Produce clear and comprehensive PlantUML activity diagrams that accurately depict the system.
To quickly convert a PV event sequence JSON to a PlantUML activity diagram:
- Install the tool (see Installation)
- Run the following command:
python -m pv2puml pv_file.json
- Open output_diagram.puml with a PlantUML viewer to see your diagram.
[
{
"jobId": "8077a248-95e9-4687-8d96-e2ca2638cfab",
"jobName": "simple_sequence",
"eventType": "A",
"eventId": "31185642-eee0-4ab4-8aac-43f94a0bb1b7",
"timestamp": "2023-09-25T10:58:06.059959Z",
"applicationName": "test_file_simple_sequence"
},
{
"jobId": "8077a248-95e9-4687-8d96-e2ca2638cfab",
"jobName": "simple_sequence",
"eventType": "B",
"eventId": "721897bf-48f4-499d-a7a3-0a8bff783b66",
"timestamp": "2023-09-25T10:58:06.059993Z",
"applicationName": "test_file_simple_sequence",
"previousEventIds": [
"31185642-eee0-4ab4-8aac-43f94a0bb1b7"
]
},
{
"jobId": "8077a248-95e9-4687-8d96-e2ca2638cfab",
"jobName": "simple_sequence",
"eventType": "C",
"eventId": "91943012-02d4-478c-bcfa-e12c5d6dc880",
"timestamp": "2023-09-25T10:58:06.060022Z",
"applicationName": "test_file_simple_sequence",
"previousEventIds": [
"721897bf-48f4-499d-a7a3-0a8bff783b66"
]
}
]
python -m tel2puml pv2puml example_above.json -jn "output_diagram"
You can run otel2puml using the provided Docker image. This is especially useful if you want to run the tool in an isolated environment without manually managing dependencies. Here’s how to do it:
Example Command:
docker run \
-v /path/to/job_json_files:/job_json_files \
-v /path/to/config.yaml:/config.yaml \
-v /path/to/puml_output:/puml_output \
ghcr.io/xtuml/otel2puml:<version> \
-o /puml_output otel2puml -c /config.yaml
Explanation:
-v /path/to/job_json_files:/job_json_files
: Mounts a local folder containing your OTel data in JSON format to the Docker container's /job_json_files directory.-v /path/to/config.yaml:/config.yaml
: Mounts the configuration file forotel2puml
to the Docker container's /config.yaml.-v /path/to/puml_output:/puml_output
: Mounts a local folder where the output PlantUML diagrams will be saved.ghcr.io/xtuml/otel2puml
:latest: Specifies the Docker image to run.-o /puml_output
: Tellsotel2puml
where to save the generated PlantUML files inside the container (linked to the localpuml_output
folder).otel2puml -c /config.yaml
: Runs theotel2puml
command using the provided configuration file.
Replace /path/to/
with the actual paths on your local machine.
On each new release of otel2puml, both an amd and arm python executable are attached as artifacts. These are downloadable and can be run from the terminal (assuming python is installed). Below is an example of usage using the arm version:
python tel2puml-arm.pyz otel2puml -o /path/to/output -c /path/to/config.yaml
There are two ways to set up this project: manual installation or using a devcontainer (recommended).
The project can be installed manually in a few different ways. Make sure you have the following prerequisites for all installation methods:
git
- Python version 3.11.9
bash
pip
(Python package manager)
(OPTIONAL) If you want to use the following functions:
tel2puml.events.save_vis_logic_gate_tree
tel2puml.utils.get_graphviz_plot
you will also need:- Java runtime environment (can be installed via
apt install default-jre
if using debian but will vary depending on your OS) - Graphviz installed
Before proceeding with the manual installation, ensure you have the following prerequisites:
- Anaconda installed and managing the Python installation
To install the project manually:
- Set up a Python virtual environment (recommended)
- Navigate to the project root directory
- Run the following commands:
conda install -c conda-forge cvxopt
conda install -c conda-forge pygraphviz
./scripts/install_repositories.sh
python3.11 -m pip install -r requirements.txt
If you don't have Anaconda installed, you can still install the project manually on both linux distributions and MacOS.
Before proceeding, ensure you have the following prerequisites that are required (since there is no build wheel for cvxopt1.3.2 for linux):
- lapack and blas libraries installed (can be installed via
apt install liblapack-dev libblas-dev
if using debian but will vary depending on your OS) - suiteparse library installed (can be installed via
apt install libsuitesparse-dev
if using debian but will vary depending on your OS) - glpk library installed (can be installed via
apt install libglpk-dev
if using debian but will vary depending on your OS)
To install the project manually:
- Set up a Python virtual environment (recommended)
- Navigate to the project root directory
- Run the following commands:
./scripts/install_repositories.sh
export CPPFLAGS="-I/usr/include/suitesparse"
export CVXOPT_BUILD_GLPK=1
python3.11 -m pip install -r requirements.txt
To install the project manually on MacOS:
- Set up a Python virtual environment (recommended)
- Navigate to the project root directory
- Run the following commands:
./scripts/install_repositories.sh
python3.11 -m pip install -r requirements.txt
Alternatively, you can use the provided devcontainer, which manages all dependencies automatically. To use the devcontainer:
- Ensure you have Docker installed on your system
- Install the Dev Containers extension for Visual Studio Code
- Open the project folder in VS Code
- When prompted, click "Reopen in Container" or use the command palette (F1) and select "Remote-Containers: Reopen in Container"
The devcontainer will automatically set up the environment with all necessary dependencies.
tel2puml
is a command-line tool designed to convert OpenTelemetry into PlantUML sequence diagrams.
The tel2puml
CLI provides several subcommands to handle different data processing scenarios:
otel2puml
: Convert OpenTelemetry data directly into PlantUML sequence diagrams.otel2pv
: Convert OpenTelemetry data into an intermediate PV format.pv2puml
: Convert PV data into PlantUML sequence diagrams.
python -m tel2puml [-o output_directory] [subcommand] [options]
** Global Options**
-o
,--output-dir
: Output directory path (default is the current directory). Nested folder creation is not supported.
Converts OpenTelemetry data directly into a PlantUML sequence diagram.
Usage:
python -m tel2puml -o OUTPUT_DIR otel2puml -c CONFIG_FILE [options]
Options:
-c
,--config
: (Required) Path to the configuration YAML file. Usage-ni
,--no-ingest
: Do not load data into the data holder.-ug
,--unique-graphs
: Find unique graphs within the data holder.
Example:
python -m tel2puml -o /output/path/ otel2puml -c /path/to/config.yaml
Converts OpenTelemetry data into an intermediate PV format.
Usage:
python -m tel2puml otel2pv -c CONFIG_FILE [options]
Options:
-c
,--config
: (Required) Path to the configuration YAML file. Usage-ni
,--no-ingest
: Do not load data into the data holder.-ug
,--unique-graphs
: Find unique graphs within the data holder.-se
,--save-events
: Save PVEvents in intermediate format.
Example:
python -m tel2puml -o /output/path/ otel2pv -c /path/to/config.yaml -se
Converts PV data into PlantUML sequence diagrams.
Usage:
python -m tel2puml pv2puml [options] [FILE_PATHS...]
Options:
-fp
,--folder-path
: Path to a folder containing job JSON files. Cannot be used withFILE_PATHS
.FILE_PATHS
: One or more files containing job data. Cannot be used with-fp
.-jn
,--job-name
: Name for the PlantUML sequence diagram and output file prefix (default is"default_name"
).-group-by-job
: Group events by job ID.
Notes:
- You must provide either
-fp
orFILE_PATHS
, but not both. - The
-group-by-job
option is useful when you have events from multiple jobs and want to separate them in the diagram.
Examples:
-
Convert a folder of job files into a PlantUML sequence diagram:
python -m tel2puml -o /path/to/output/ pv2puml -fp /path/to/folder
-
Convert specific job json files into a PlantUML sequence diagram:
python -m tel2puml -o /path/to/output/ pv2puml file1.json file2.json
-
Convert a folder of job JSON files with a custom job name:
python -m tel2puml -o /path/to/output/ pv2puml -fp /path/to/folder -jn "MySequenceDiagram"
-
Convert a folder of job JSON files and group events by job ID:
python -m tel2puml -o /path/to/output/ pv2puml -fp /path/to/folder -group-by-job
For detailed help on each subcommand, use the -h
or --help
option:
python -m tel2puml [subcommand] -h
Example:
python -m tel2puml pv2puml -h
To gain a better technical understanding of the project it's recommended to read the technical implementation overview.
Documentation for tel2puml can be found in the docs folder. This contains design notes, end to end test information and the technical implementation overview.
TEL2PUML depends on one other repository:
-
Janus ingests PUML activity diagram files and generates event sequences from them.
This dependency is automatically managed when using the devcontainer setup or when running the install_repositories.sh
script during manual installation.
We welcome contributions to improve tel2puml! Here's how you can contribute:
- Fork the repository
- Create a new branch (git checkout -b feature/amazing-feature)
- Make your changes
- Commit your changes (git commit -m 'Add some amazing feature')
- Push to the branch (git push origin feature/amazing-feature)
- Open a Pull Request