Usability Issue: Multi-machine dataflows require a lot of manual setup work

[phil-opp]

In order to multi-machine dataflow, each machine needs to run an instance of the dora daemon. Each daemon instance currently requires the coordinator IP:Port and a unique machine ID as command line arguments. This leads to multiple usability issues:

• Setup is difficult to automate since the command differs on each machine (because of the unique ID).
• All machines need to be updated when the coordinator ID changes.
• Because of the prior points, it is difficult let the daemon autostart.
  • As a consequence, the whole setup work needs to be done again when the machines reboot.
  • This is problematic for machines that are rebooted often, e.g. embedded devices or robots.

[Xavier Edit]

• it is also hard to ensure that inter daemon connection work as daemon might be on different network. This might need some prerequisites ssh tunnels or routers, that might be hard to make people understand as we have abstracted away the network abstraction.
• let's also imagine the case of a robot manufacturer that just want to have a dora daemon running on his robot so that it can be super easy for people to get started without ssh.

[phil-opp]

Proposal: System ID as Machine ID + Zenoh Communication

This proposal tries to make both the machine ID and coordinator address arguments of the dora daemon command optional. Thus, users could just add the dora daemon command to the autostart on each machine once. Changes of the coordinator address would be possible without requiring reconfiguration of the machines.

System ID as Machine ID

Most operating systems provide a way to get a unique system identifier. By using this identifier as machine ID, we avoid the need to pass it as command-line argument.

For example, on many Linux distributions the system ID can be queried through cat /etc/machine-id. On Windows, the wmic os get serialnumber command could be used: https://stackoverflow.com/questions/47603786/where-do-windows-product-id-and-device-id-values-come-from-are-they-useful

Zenoh Communication

In our current architecture of dora, daemons need to register with the dora coordinator at startup. This requires that the daemons can find the coordinator and communicate with it. In our current implementation, we pass the coordinator address as command-line argument to the daemon, so that the daemon can open a normal TCP connection.

To avoid this command-line argument, we could use a communication middleware that supports auto-discovery in local networks. This way, the daemons would be able to find the coordinator without knowing its address. The Zenoh middleware implements a scouting protocol based on UDP multicast messages for local networks. So by using zenoh for the initial register message, we will be able to avoid the coordinator address argument for the dora daemon command.

Zenoh also even supports auto-discovery of zenoh routers within the local network, which then manage the connection to remote networks. So even for dataflows split across multiple networks, we would be able to omit the coordinator address argument for dora daemon as long as there is some zenoh router within each network.

[haixuanTao]

I mean it sounds like we're trying to solve complexity with even more complexity.

We're going to add zenoh so people will have to get familiar with zenoh.

They will need to understand how both zenoh broadcast and register node and also how dora broadcast and register node.

If there is any failure, they will need to be expert on zenoh-dora because none of the existing tooling such as nmap, ping, ... might be able to help them.

We will instead of having 2 background process have 3 background process that is already hard to keep in sync.

We might have increased latency...

I don't see why we need to have auto discovery. Most of the time, it's fairly easy to find an IP address.

[haixuanTao]

Proposal: connect daemon to other daemon from IP address

By being able to connect daemon to daemon with IP address without abstraction removes the need of a coordinator and is super simple to understand.

If IP address is not reachable from any of the daemon, it's pretty easy to debug.

Privacy concern can be dealt with environment variables so that they don't need to appear within git.

[haixuanTao]

I mean by removing IDs, and replacing them with IP Adress within the dataflow thus removing the need for registering them with a coordinator.

[phil-opp]

So, as I understand it, your idea is the following:

• When a daemon is started, it listens on some (hardcoded?) port number for incoming connections.
  • It does not try to connect to a coordinator or other daemons at this point.
• Inside the dataflow YAML file, the user specifies the IP address of every machine they want to use.
  • Each IP address must be reachable from all the machines.
    • If you want to mix cloud and local machines, a public IP is required for all machines.
  • If parts of the dataflow should be run on the local machine, its external IP address needs to specified too
    • Specifying 127.0.0.1 (or similar) is not possible since that address is not reachable from the other machines.
  • Instead of specifying the IP addresses directly, the dataflow.yml can also refer to environment variables.
• On dora start, we connect to all daemons and instruct them to prepare the dataflow.
  • We also send the full list of daemon IPs to each daemon at this point.
  • Upon receiving the IP list, each daemon initiates TCP connections to all the daemons it needs to interact with (because of inputs/outputs)
    • Daemons send regular heartbeat messages to each other to ensure that they are all still running
  • Once all daemons are ready, the CLI coordinates the synchronized dataflow start.
• There is no more coordinator, so everything is controlled by the CLI directly.
  • The CLI subscribes to all the log output of each daemon
  • The CLI is responsible for assigning an UUID and name to the dataflow
  • The CLI determines when a dataflow is finished (i.e. when it's finished on all daemons)
  • The dora stop command now requires the original dataflow.yml file as argument because it needs to know all the daemon IPs
    • If dora stop is executed from a different machine than the corresponding dora start, the latest dataflow.yml version needs to be copied to the machine running dora stop.
    • The dora stop command might leave parts of the dataflow running when the daemon IP list is not complete.
  • In order to use dora list, users need to pass the list of daemon IPs as command line argument
    • Or alternatively, the dataflow.yml file
    • The list is only complete when all daemon IP addresses are given to the command.

Is this correct?

[haixuanTao]

Inside the dataflow YAML file, the user specifies the IP address of every machine they want to use.

and

If parts of the dataflow should be run on the local machine, its external IP address needs to specified too

Technically, as our current design they only need to be reachable by the node that tries to send data to them. So If some of the data is in a closed local sub-graph this should be no problem

On dora start, we connect to all daemons and instruct them to prepare the dataflow.
We also send the full list of daemon IPs to each daemon at this point.
Upon receiving the IP list, each daemon initiates TCP connections to all the daemons it needs to interact with (because of inputs/?> outputs)
Once all daemons are ready, the CLI coordinates the synchronized dataflow start.

Yes this is fine.

   Daemons send regular heartbeat messages to each other to ensure that they are all still running

We might not need this at the very beginning. We can just use inputclosed message.

The CLI subscribes to all the log output of each daemon

We can just leave each daemon with its own output in its own stdout. We do not need to report everything to the cli.

The CLI is responsible for assigning an UUID and name to the dataflow

I wish we could run a dataflow as a single process and so without having to have UUID but if it makes refactoring easier. We can assign a UUID and a name within the CLI,

The CLI determines when a dataflow is finished (i.e. when it's finished on all daemons)

Yes

The dora stop command now requires the original dataflow.yml file as argument because it needs to know all the daemon IPs

I don't think we should have background process running. So basically, if we want to stop the dataflow, we should just stop the existing process.

The dora stop command now requires the original dataflow.yml file as argument because it needs to know all the daemon IPs
If dora stop is executed from a different machine than the corresponding dora start, the latest dataflow.yml version needs to be copied to the machine running dora stop.
The dora stop command might leave parts of the dataflow running when the daemon IP list is not complete.

We can just do something like: dora stop daemon_1_addr daemon_2_addr ... alternatively if we don't have the original dataflow.

   In order to use dora list, users need to pass the list of daemon IPs as command line argument
       Or alternatively, the dataflow.yml file
       The list is only complete when all daemon IP addresses are given to the command.

I mean the idea is that if they want to check on dora status they should refer to the running daemon process and its stdout. We can provide a quick health check but it should not be a required feature.

[phil-opp]

If parts of the dataflow should be run on the local machine, its external IP address needs to specified too

Technically, as our current design they only need to be reachable by the node that tries to send data to them. So If some of the data is in a closed local sub-graph this should be no problem

The CLI needs to be able to reach them to send the start command, the daemon IP list, etc.

The CLI is responsible for assigning an UUID and name to the dataflow

I wish we could run a dataflow as a single process and so without having to have UUID but if it makes refactoring easier. We can assign a UUID and a name within the CLI,

See #703 for a way to run dataflows as a single process. This only works for single-machine dataflows obviously.

The dora stop command now requires the original dataflow.yml file as argument because it needs to know all the daemon IPs

I don't think we should have background process running. So basically, if we want to stop the dataflow, we should just stop the existing process.

This is how #703 works, but for multi-machine dataflows we need to send stop commands to all daemons that are running parts of the dataflow.

The CLI subscribes to all the log output of each daemon

We can just leave each daemon with its own output in its own stdout. We do not need to report everything to the cli.

So if a node on a remote machine fails, you have to ssh into it in order to see the error?

[haixuanTao]

So if a node on a remote machine fails, you have to ssh into it in order to see the error?

Ok, it's true that it's inpractical.

I think that what we can do is pause this discussion as it seems that there is a lot of implementation details. I'm going to make a simpler proposal that keeps the coordinator to make it less of a breaking change.