How can you run this server with more than one schema version?

[tom-leahy]

The schema server hosted at https://schema.ocsf.io/ allows you to pick from multiple schema versions in the upper left corner, however I'm not following how to set this up in the same way when I run this locally in docker. I'm just seeing v1.3.0-dev as the only option.

I also noticed there is an environment property SCHEMA_HOME - wasn't sure whether that's what is needed or not, couldn't find examples of it being used online or in the docs here. Any suggestions?

[tom-leahy]

@rmouritzen-splunk - sorry to @ you here, but I see you've pushed several commits lately and I'm unsure how else to reach out to anyone about this.

[rmouritzen-splunk]

Right. We haven't documented this. It's really ugly. The web app doesn't support multiple versions. Instead, the web app has a very limited ability to switch to a different version at a different URL through some tricky environment variables. (This is something I want to change as this gets more an more burdensome as new OCSF schema versions are released.)

See below for how this works.

[rmouritzen-splunk]

You may want to join the OCSF Slack workspace. Instructions are on the ocsf.io page.

[tom-leahy]

Thanks for the context and detailed answer @rmouritzen-splunk - follow on question: does the validation API require this same setup to support validation of values across different metadata versions or is this specific to the UI? (needing nginx, etc...)

[rmouritzen-splunk]

If I understand your question correctly, yes.

You should validate against the version of the schema for your event. You can use a later version of the schema, though the validation will return a message about mismatched schema version.

[rmouritzen-splunk]

The public server supports multiple version by fronting all traffic through Nginx. Requests are proxied to Docker images based on the request URL, with each Docker image configured to use a specific schema, as well as a base directory containing all of the schemas, where each schema is a git clone of different branches of the ocsf/ocsf-schema repo.

Here is a snippet of the Nginx configuration showing how the redirection is working, with 1.4.0 and 1.5.0-dev mapped:

server {
  .
  .
  .

   # The root path redirects to the latest released schema version, currently 1.4.0
  location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /schema/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /1.4.0/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /1.4.0/schema/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /api/1.4.0/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006/api/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /sample/1.4.0/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006/sample/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /schema/1.4.0/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8006/schema/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /1.5.0-dev/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8007/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /1.5.0-dev/schema/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8007/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /api/1.5.0-dev/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8007/api/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /sample/1.5.0-dev/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8007/sample/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /schema/1.5.0-dev/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://localhost:8007/schema/;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

Here's an example how how to clone the repos:

mkdir -p /opt/ocsf/schemas
cd /opt/ocsf/schemas

branch=v1.4.0
git clone --single-branch --branch $branch https://github.com/ocsf/ocsf-schema.git $branch

branch=main
git clone --single-branch --branch $branch https://github.com/ocsf/ocsf-schema.git $branch

Here a script that launches the Docker images. I regularly push new images to my own work account's Docker hub profile. This script is assumes the schemas are in /opt/ocsf/schemas.

start-server.sh:

#!/bin/bash

# Note: this is a bash-specific test for running as root or sudo
if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root" 
    exit 1
fi

set -x

# Stop existing containers
docker ps -aq | xargs docker stop | xargs docker rm

# The above stop / cleanup fails when there are no containers running,
# so we wait until after that step to force an exit on error.
set -e

# Start new containers
container=rmouritzen388/ocsf-server:2.77.3

function run_server() {
    name=$1
    schema_dir=$2
    web_schema_path=$3
    localhost_port=$4

    docker run --name "${name}" -d --restart unless-stopped \
        -v /opt/ocsf/schemas:/app/schemas \
        -e SCHEMA_HOME=/app/schemas \
        -e SCHEMA_DIR=/app/schemas/"${schema_dir}" \
        -e SCHEMA_PATH=/"${web_schema_path}" \
        -e HOST=schema.ocsf.io \
        -p 127.0.0.1:"${localhost_port}":8080/tcp \
        $container
}

# run_server rc2 v1.0.0-rc.2 1.0.0-rc.2 8000
# run_server rc3 v1.0.0-rc.3 1.0.0-rc.3 8001
# run_server 1.0 v1.0.0 1.0.0 8002
# run_server 1.1 v1.1.0 1.1.0 8003
# run_server 1.2 v1.2.0 1.2.0 8004
# run_server 1.3 v1.3.0 1.3.0 8005
run_server 1.4 v1.4.0 1.4.0 8006
run_server dev main 1.5.0-dev 8007

# Cleanup Docker
# https://docs.docker.com/config/pruning/#prune-everything
docker system prune --volumes -a -f

Note that these aren't instructions to build this up a Linux server from scratch.

• You'll need to install Nginx and Docker,
• You'll need to make sure that Nginx can read the directory where the schemas are located.
• The Nginx configuration isn't complete -- I'm only showing the redirection, not the configuration for listening and SSL and such.

[tom-leahy]

Thanks @rmouritzen-splunk !