Skip to content

Deployment Guide for Runtime Bridge 2

Jump to bottom
Krhougs edited this page Feb 1, 2022 · 1 revision

Understanding Runtime Bridge

Following services make a Runtime Bridge setup work:

  • Data Provider(a.k.a data_provider):
    • Fetches and analyses block data from Substrate,
    • Makes pre-encoded blobs that pRuntime consumes,
    • Serves pre-encoded blobs for lifecycle managers;
  • Lifecycle Manager(a.k.a lifecycle):
    • Sends block data blobs to workers(pRuntime);
    • Registers worker on the chain;
    • Starts mining with configured stake amount;
    • Provides API to query workers and pools.
  • Trader(a.k.a trade):
    • Sends transactions from queue to the chain.

Every service above uses https://github.com/Phala-Network/runtime-bridge-walkie as a communication method built on libp2p and Protocol Buffer. On the first startup of each service, an RSA key pair will be generated(by default saved in /var/data/keys/id) to be used as the identity in the libp2p protocol. The current identity public key can be found in the stdout of running service by searching Got my peer id in the base58 format. And for lifecycle managers, the key pair is used to encrypt critical information. DO BACK THE KEYS UP AND KEEP THE KEYS SAFE.

When the lifecycle manager has finished the initial blob synchronization, it will attempt to sync messages in the message queue between the pRuntime and the blockchain. After that, it will try to register workers and do the “start mining” operation on the blockchain.

Extrinsics (transactions) shall be sent to the blockchain for the operation mentioned above. To achieve this, the lifecycle manager shall push the extrinsics data to the trader queue(currently maintained with Redis). The trader will grab jobs from the queue and report the extrinsic results back.

Understanding the startup process and internal dependencies

Data providers and lifecycle managers are designed to discover each other using libp2p. While starting the service, it connects to the blockchain then analyses the chainIdentity from the parachain, libp2p will be initialized with the identity key and the chainIdentity. After the initialization, the service should begin its work and discover other peers via mDNS as well as the bootstrap node configured.

A data provider fetches data from the parachain and the relaychain(called parent chain in Substrate). It starts a standalone TCP server for blob synching due to the performance issue of libp2p, the port of which will be included in the peer handshake process.

While starting the lifecycle manager, it connects to the Redis server for the trade queue. There should be only 1 lifecycle manager and 1 trader accessing the same Redis server. The trader paired with the lifecycle manager should use the same identity key to ensure that the trader can decrypt the private key of saved pools. While the lifecycle requires a specified Redis server to start, it doesn’t require any data provider to start the service. It will wait until any valid data providers are discovered, which means it’s safe to stop the data provider when the lifecycle manager running, and you can build an HA setup for data providers.

Data in the Redis server should never be persisted since the lifecycle manager always builds states from the blockchain and pRuntime. Restart the lifecycle manager, the trader, and the Redis server together when any error occurs.

Configuring Runtime Bridge

Services with a Runtime Bridges setup are configured with environment variables.

Shared items

Name Description
NODE_ENV Application environment, set to development to put the app in development mode.
PHALA_MODULE The module to start.
PHALA_LOGGER_LEVEL Logger level, defaults to info. See https://github.com/trentm/node-bunyan for more information.
PHALA_CHAIN_ENDPOINT The WebSocket endpoint to the Substrate RPC of parachain.
PHALA_PARENT_CHAIN_ENDPOINT The WebSocket endpoint to the Substrate RPC of relaychain(e.g. for Khala it’s Kusama).
PHALA_PEER_ID_PREFIX The path storing identity keys, defaults to '/var/data/keys/id'.
PHALA_WALKIE_LISTEN_ADDRESSES The multiaddr(https://github.com/libp2p/specs/tree/master/addressing) of listen address for libp2p, defaults to '/ip4/0.0.0.0/tcp/0,/ip6/::/tcp/0' which means listen to a random port on every interface. Only TCP protocol is supported. Use a comma between addresses.
PHALA_WALKIE_BOOT_NODES The multiaddr list of bootstrap nodes for peer discovery, defaults to '/ip4/0.0.0.0/tcp/18888,/ip6/::/tcp/28889' which means no bootstrap node. Only TCP protocol is supported. Use a comma between addresses.

Items for data providers

Name Description
PHALA_LOCAL_DB_PATH The path to database, defaults to '/var/data/0'.
PHALA_DATA_PROVIDER_LOCAL_SERVER_PORT The listen port for the blob server, defaults to 8012.

Items for lifecycle managers

Name Description
PHALA_Q_REDIS_ENDPOINT The endpoint to the Redis for trader task queue.
PHALA_RUNNER_MAX_WORKER_NUMBER The maximum worker number of one runner, defaults to 150.
PHALA_LIFECYCLE_CONFIG_MODE Whether the lifecycle manager should enter config mode, where it will start only the API to add/modify saved pools/workers. Set to true to enable.
PHALA_LRU_CACHE_SIZE The size of LRU cache, defaults to 5000.
PHALA_LRU_CACHE_MAX_AGE The maximum age of items in the LRU cache in milliseconds, defaults to 30 minutes.
PHALA_ENFORCE_MIN_BENCH_SCORE Whether the lifecycle manager should re-try the worker registration on the chain if the on-chain benchmark score to too low. Set to true to enable.
PHALA_MIN_BENCH_SCORE Desired minimum benchmark score.

Items for trader

Name Description
PHALA_Q_REDIS_ENDPOINT The endpoint to the Redis for task queue.

Configuring Node.js memory usage

Only do this when you are suffering from OOM issues.

Change the docker entrypoint to node -trace-warnings -experimental-json-modules -es-module-specifier-resolution=node -harmony-top-level-await -max-old-space-size=$MAX_OLD_SPACE_SIZE dist/index. Change the $MAX_OLD_SPACE_SIZE to your desired size in MB.

https://nodejs.org/dist/latest-v16.x/docs/api/cli.html#--max-old-space-sizesize-in-megabytes

Using the monitor

The monitor(https://github.com/Phala-Network/runtime-bridge/tree/next) is an example of the Walkie usage. It provides a simple management ability to play with Runtime Bridge.

It also implements a JSON proxy to the Walkie API.

POST /ptp/discover returns the list of discovered peers.

POST /ptp/proxy/:peer_id/:method sends the API request to the specified peer.

The API definition can be found in https://github.com/Phala-Network/runtime-bridge-walkie/blob/master/src/proto/message.proto#L78.

Import pools and workers

To import pools: https://github.com/Phala-Network/runtime-bridge-walkie/blob/master/src/proto/message.proto#L95

With monitor:

curl --location --request POST 'http://path.to.monitor/ptp/proxy/Qmbz...RjpwY/CreatePool' \
--header 'Content-Type: application/json' \
--data-raw '{
    "pools": [
        {
            "pid": 2,
            "name": "test2",
            "owner": {
                "mnemonic": "boss...chase"
            },
            "enabled": true,
            "realPhalaSs58": "3zieG9...1z5g"
        }
    ]
}'

To import workers: https://github.com/Phala-Network/runtime-bridge-walkie/blob/master/src/proto/message.proto#L98

With monitor:

curl --location --request POST 'http://path.to.monitor/ptp/proxy/Qmbz...RjpwY/CreateWorker' \
--header 'Content-Type: application/json' \
--data-raw '{
    "workers": [
        {
            "pid": 2,
            "name": "test-node-1",
            "endpoint": "http://path.to.worker1:8000",
            "enabled": true,
            "stake": "4000000000000000"
        },
        {
            "pid": 2,
            "name": "test-node-2",
            "endpoint": "http://path.to.worker2:8000",
            "enabled": true,
            "stake": "4000000000000000"
        }
    ]
}'

Restart the lifecycle manager after modified pools/workers.

Docker Compose example

(WIP)

Clone this wiki locally