Add support for distributed deployments with multiple daemons

[phil-opp]

• Adds support for a new deploy.machine key to the dataflow YAML format.
  • Can be specified for individual nodes or for the whole dataflow.
  • Dataflows can be split across multiple machines/daemons.
  • If not set, the machine ID defaults to the empty string.
• The dora daemon supports a new --machine-id argument, defaulting to the empty string. Multiple daemons can be connected to a coordinator, as long as they have different machine IDs.
• The dora coordinator supports a new --port argument to change the port that the coordinator listens on. Defaults to 53290.
• The dora daemon supports a new --coordinator-addr argument to set the IP and port number of the coordinator. Defaults to 127.0.0.1:53290.
• Nodes will synchronize their start across machines through new coordinator messages. Similarly, the coordinator will forward outputs and InputClosed events between machines.

[haixuanTao]

I think there is a lot of good ideas in this PR, so thanks Philipp!

I think that we maybe lacking a file-system management piece of software to manage file between machines but also changes. I was looking at the implementation of cargo and how it manages cargo.lock and git-rs and I wonder if there is thing we can salvage or even use cargo to manage our user-defined nodes.

In the way react and angular reuse npm to manage their user-implemented modules.

But this is a big features that needs its own PR, and I think we can assume in this PR that the user has made the appropriate changes within its filesystem.

I will test it maybe further later during the day.

[phil-opp]

Yeah, we still need some kind of deploy functionality to get the nodes and operators from the CLI machine to the target machines. We already support URL sources for nodes and operators,

I think that we maybe lacking a file-system management piece of software to manage file between machines but also changes.

Agreed. There are currently only two options to distribute the node/operator binaries across machines:

• Copy them manually, or through some custom script.
• Use URL sources for nodes and operators, e.g. a GitHub release.

I think the second option is already quite convenient for "finished" nodes and operators, but it's cumbersome during development. We should try to make the edit->compile->deploy cycle easier for distributed dataflows too.

Maybe we could make the CLI send the executables via TCP as part of the spawn command? The receiving daemon could store them to the file system and then run them. This would make things much easier for nodes/operators written in Rust, as you could just develop as usual on the CLI machine and the dora start command would take care of everything else. For Python, we probably need some way to copy additional files, and set up a correct environment too...

[haixuanTao]

So, I retested this branch with the implementation of the log branch and it is ok for me. An example of the YAML description with local and remote will be appreciated before merging.

[heyong4725]

Local and remote seems a little confusing to me regarding to distributed deployment, i.e., respect to which machine is local and/or remote? I think we should have an end-2-end dataflow graph describing distributed deployment at the control plane (CLI / coordinator layer), specifying pub/sub communication middleware provider (e.g., zenoh, DDS, or SOME/IP) with their corresponding configuration.

[phil-opp]

As discussed in today's meeting:

• coordinator should not be involved in data forwarding -> daemons should talk to themselves instead (via TCP or zenoh)
• prefix newly introduced YAML keys with e.g. _unstable -> consider this PR as an unstable feature -> postpone open design questions about YAML format deployment of node executables
• pass contents of dataflow YAML to daemons, instead of path (discussed with Xavier)

[phil-opp]

I implemented the points discussed in the latest meeting and merged the latest changes from main. I think this PR is ready to be merged now. Please let me know if you have any other concerns.