DEV-INFO.md

Debugging commands

The following commands may be useful, if you would like to reproduce the Scoreboard locally for testing.

Configuration file

Configuration in the config.json file contains a list of frameworks included in ONNX Backend Scoreboard. This is a place for base information like results paths and core packages names. Each new runtime has to be added to this file.

Example of config.json file:

{
    "stable": {
        "onnxruntime": {
            "name": "ONNX-Runtime",
            "results_dir": "./results/onnx-runtime/stable",
            "core_packages": ["onnxruntime"]
        },
        "ngraph": {
            "name": "nGraph",
            "results_dir": "./results/ngraph/stable",
            "core_packages": ["ngraph-onnx", "ngraph-core"]
        },
        "onnxtf": {
            "name": "ONNX-TF",
            "results_dir": "./results/tensorflow/stable",
            "core_packages": ["onnx-tf", "tensorflow"]
        }
    },
    "development": {
        "onnxruntime": {
            "name": "ONNX-Runtime",
            "results_dir": "./results/onnxruntime/development"
        },
        "ngraph": {
            "name": "nGraph",
            "results_dir": "./results/ngraph/development"
        }
    },
    "deploy_paths": {
        "index": "./docs",
        "subpages": "./docs",
        "resources": "./docs/resources"
    }
}

Building Docker images

Use these commands from the main directory of this repository to build Docker images which run backend tests.

Stable versions

• ONNX-Runtime:
  docker build -t scoreboard-onnx -f runtimes/onnx-runtime/stable/Dockerfile .

• nGraph:
  docker build -t scoreboard-ngraph -f runtimes/ngraph/stable/Dockerfile .

• ONNX-TF:
  docker build -t scoreboard-tensorflow -f runtimes/tensorflow/stable/Dockerfile .

Development versions (built from source)

• ONNX-Runtime
  docker build -t scoreboard-onnx -f runtimes/onnx-runtime/development/Dockerfile .

• nGraph
  docker build -t scoreboard-ngraph -f runtimes/ngraph/development/Dockerfile .

Proxy settings

If you're working behind a firewall use --build-args to set http_proxy and https_proxy

docker build -t scoreboard-<backend> \
             --build-arg http_proxy=your-http-proxy.com \
             --build-arg https_proxy=your-https-proxy.com \
             -f <path_to_dockerfile>/Dockerfile .

Run Docker containers

Running a Docker container based on the previously prepared image, will run ONNX tests and store results in the directory specified as results_dir in config.json.

Stable

• ONNX-Runtime
  docker run --name onnx-runtime --env-file setups/env.list -v ~/backend-scoreboard/results/onnx-runtime/stable:/root/results scoreboard/onnx

• nGraph
  docker run --name ngraph --env-file setups/env.list -v ~/backend-scoreboard/results/ngraph/stable:/root/results scoreboard/ngraph

• ONNX-TF
  docker run --name tensorflow --env-file setups/env.list -v ~/backend-scoreboard/results/tensorflow/stable:/root/results scoreboard/tensorflow

Development (build from source)

• ONNX-Runtime
  docker run --name onnx-runtime --env-file setups/env.list -v ~/backend-scoreboard/results/onnx-runtime/stable:/root/results scoreboard/onnx

• nGraph
  docker run --name ngraph --env-file setups/env.list -v ~/backend-scoreboard/results/ngraph/development:/root/results scoreboard/ngraph

Generation of the Scoreboard pages

From the main directory of the repository, issue the following command:

python3 website-generator/generator.py --config ./setup/config.json

where --config parameter is the path to config.json file.

This will generate an HTML version of the Scoreboard in the docs directory (or another specified in deploy_paths in config.json).