How to start the node

[plafer]

In our weekly meeting, we discussed different ways to start the node. Specifically, given that we currently need at least 3 processes (store, rpc, and block-producer), and there are currently dependencies between these (i.e. store must be started before rpc), how does a user start everything?

I looked at how Reth and Erigon do it:

• Reth: you can simply build a binary reth with cargo build --release, and then execute it to run the full node.
• Erigon: Similarly, you run make erigon, which generates a single binary ./build/bin/erigon which you can execute to get the full node. Even though Erigon can be run on multiple machines or in docker containers, they specifically recommend only to do that if necessary.

I think we should have the same approach. Very simply, we generate an executable which spawns 3 child processes which open their gRPC endpoints at configurable ports (with sensible defaults that just work). This is easy, and it would work just fine for our initial testnet. In the future, we can offer Docker images, prebuilt binaries for each component, etc.

What do people think?

[bobbinth]

I'm fine with this approach. Couple of questions:

1. I'm assuming this executable will be a different crate and it will use the other 3 crates as dependencies? Or is there some other way to do it?
   a. If different crate, we should confirm that it'll work as the other 3 crates might require some refactoring to be used a libraries.
   b. We also should come up with a name for this crate (and the binary file).
2. How would we configure logging here? I'm assuming all 3 processes will print to stdout and it could get quite noisy. Is there a better approach?

[plafer]

I'm assuming this executable will be a different crate and it will use the other 3 crates as dependencies? Or is there some other way to do it?

I was thinking to do it like that yes. In my opinion, every component should expose a constructor of some kind, which all executables use (whether it's all started on the same machine, different machines, etc). If that's not how they all work today, I think it's a desirable refactor anyway.

We also should come up with a name for this crate (and the binary file).

miden-node?

How would we configure logging here? I'm assuming all 3 processes will print to stdout and it could get quite noisy. Is there a better approach?

This is solved by using tokio's tracing library, which @hackaugusto already started using in the store. This talk gives a great overview of exactly the problem of a bunch of async processes spewing out logs, and how tracing solves it.

[hackaugusto]

IMO we should always make crates libraries and have a thing binary created from it. Ideally the binary that is mentioned here would started all the services in the same process, this prevents tons of issues.

[plafer]

Another consideration: we can either support Unix only, or Unix and Windows.

• Unix only: we can use fork()/exec(), which in my opinion is the easiest and cleanest approach
• Unix and Windows: we need to use the cross-platform std::process::Command API, which only supports executing named binaries in the PATH. This is a bit more tricky since we need to make sure that cargo install for example moves all binaries properly such that it doesn't break the expected relative paths. It might be easy to do that, I'm just not sure at the moment.

[bobbinth]

I'm fine with sticking to Unix for now. We can expand this in the future versions.

[hackaugusto]

we generate an executable which spawns 3 child processes which open their gRPC endpoints at configurable ports (with sensible defaults that just work). This is easy, and it would work just fine for our initial testnet.

I'm against having a special process that spawn 3 children processes. There are many reasons why, here are some:

• If the parent dies, it is effectively a double fork and the children become daemons, probably not what we want
• stdout/stderr redirection will become messy. if the children write to their stdout, it will need to be read by the parent, adding complexity and possibly becoming a bottle neck (when the pipe is full it blocks). the alternative is to write to separate files which is also not ideal
• the process will need to monitor the children and restart them if necessary, which is the job of a service monitor and we probably don't want to go there
• this doesn't eliminate the need for synchronization, since the ordering is dependent on the socket being available, and not the process being spawned
• it makes it harder to debug some scenarios (often I want to start a process from the debugger, and this will require configuration from the top-level process to not start something, and that becomes tricky if the something is a dependency of something else)

[bobbinth]

What would be the alternative? Have a script that starts 3 separate processes?

[hackaugusto]

my suggestion is to go with a single process: #81 (reply in thread)

if for some reason that doesn't work, and we have 3 processes, the issues above don't go away, and then I would go for the simplest thing possible:

#!/usr/bin/env sh

miden-store&;

sleep 5;
miden-block-producer&;

sleep 5;
miden-rpc&;

wait;

or maybe

#!/usr/bin/env sh

miden-store&;

false;
while [[ $? -ne 0 ]]; do
    miden-block-producer&;
done

false;
while [[ $? -ne 0 ]]; do
    miden-rpc&;
done

wait;

Note: The above has the downside of the stdout being mangled, one could use 2>&1 miden-store.log or something like that to fix that. We also lose access to stdin, but I guess we don't need it here.

If the above doesn't work well enough, then IMO we should use a process supervision, there are tons out there. From the OS (like systemd, upstart, launchd, and so on), or tools for the task (monit, daemontools, god), they offer way more capabilities than we need but fix the issue. Or we can just simple go with docker-compose

[hackaugusto]

One thing that I'd like to mention, this discussion skipped the conversation of why docker is better to directly go for a start script. Here are the reasons why I recommend docker:

• isolate the network space.
  • prevents issues of port collision with the host system and the test environment
  • this also means that running multiple testcasts concurrently is possible, otherwise it gets messy to configure the ports on the host for multiple runs
• child process management
  • there is literally no way of having a parent process guaranteeing they children will exit without cgroups, at least as far as I'm aware (and I would love to be proven wrong)
  • the above means that it is super easy to have "rouge" processes from a previous run on the system, with dirty state, that can cause frustration during testing.
• log management
  • it is nice to have log lines that are not intermixed, and are accessible via a paging interface, without having to do much configuration in the server process
• devex for downstream projects
  • it allows people to get started quickly with our tooling, once we publish the release to docker hub

I also am not sure how to handle the initialization of servers for integration tests. Ideally the testsuite should take care of everything, meaning it should be easy to run and fast to get started (this is quite important for development cycle, and integration tests will very easily eat lots of time). It would be ideal to find a test runner for rust that can run fixture setup for that.

[plafer]

After discussion, we are going to go with simply running everything in the same process; i.e. the gRPC server/clients will be the same process.