Goal: Make it easier to run local dataflow

[phil-opp]

Our current architecture always requires a coordinator and at least one daemon in order to run a dataflow. This makes starting a local dataflow quite complex:

• Terminal 1: dora coordinator
• Terminal 2: dora daemon
• Terminal 3: dora start dataflow.yml

We also have a dora up command that spawns the coordinator and daemon in the background, but this just avoids the need of multiple terminals and doesn't remove the complexity.

Local dataflows are quite common, especially when getting started. So it would be a good idea to make it easier to run a dataflow when coordinator, daemon, and the CLI are on the same machine.

[phil-opp]

Proposal: Add a dora subcommand to run a dataflow without spawning separate daemon and coordinator processes.

We already have an internal dora daemon --run-dataflow command that we use for testing. This command starts a temporary daemon instance that exits as soon as the dataflow is finished. Instead of communicating with an external coordinator, it uses an internal, virtualized coordinator.

We could expose this internal functionality as a dora run <dataflow> command. This command would behave in the following way:

• Error if any remote machines are specified in the dataflow.
  • Suggest to use the dora start command instead
• Write all log output to stdout, defaulting to INFO level.
• Write all node stdout/stderr to stdout
• Stop the dataflow on CTRL-C
• Force-stop (i.e. kill) the dataflow on second CTRL-C
• Kill all started nodes on abnormal exit (e.g. panic)
  • The goal is that you never have to do any cleanup, even if the dataflow execution fails in an uncommon way.

Once the command is implemented and working well, we could update our introductionary guides and documentation to use dora run <dataflow> instead of dora up; dora start <dataflow>; dora destroy.

[phil-opp]

(If we decide to implement this proposal, I think it might be a good idea to deprecate the dora up command afterwards because it is a common source of confusion. I opened #700 to track this idea.)

[haixuanTao]

Ok for this!