Skip to content

Data flow descriptor

Jump to bottom
Julien Loudet edited this page Jan 22, 2025 · 6 revisions

Table of Contents

The data flow descriptor tells Zenoh-Flow how to deploy the application. It provides the following information:

  • which nodes are involved and where to find their descriptors,
  • how the nodes are connected,
  • (optionally) where the nodes should run.

Below is a simple data flow descriptor that we will use to explain the different sections. Optional sections are indicated as such.

name: my-first-flow

# (optional)
vars:
  BASE_PATH: file:///home/zenoh/my-first-flow/nodes


# (optional)
configuration:
  default_timeout: 5


sources:
  - id: source
    library: "{{ BASE_PATH }}/libsource.so"
    outputs:
      - out

  # For convenience, the descriptor of a node can reside in a separate file.
  # - id: source-ext
  #   descriptor: "{{ BASE_PATH }}/source.yml"

  - id: zenoh-builtin-sub
    zenoh-subscribers:
      sub: "my-first-flow/in"
      # Multiple subscribers are supported.
      # sub2: "my-first-flow/in2"


operators:
  - id: operator
    library: "{{ BASE_PATH }}/liboperator.so"
    inputs:
      - in
      - in-sub
    outputs:
      - out
      - out-pub

  # For convenience, the descriptor of a node can reside in a separate file.
  # - id: operator-ext
  #   descriptor: "{{ BASE_PATH }}/operator.yml"


sinks:
  - id: sink
    library: "{{ BASE_PATH }}/libsink.so"
    inputs:
      - in

  # For convenience, the descriptor of a node can reside in a separate file.
  # - id: sink-ext
  #   descriptor: "{{ BASE_PATH }}/baz.yml"

  - id: zenoh-builtin-pub
    zenoh-publishers:
      pub: "my-first-flow/out"
      # Multiple publishers are supported.
      # pub2: "my-first-flow/out2"


links:
- from:
    id: source
    output: out
  to:
    id: operator
    input: in

- from:
    id: zenoh-builtin-sub
    output: sub
  to:
    id: operator
    input: in-sub

- from:
    id: operator
    output: out
  to:
    id: sink
    input: in

- from:
    id: operator
    output: out-pub
  to:
    id: sink
    input: pub


# (optional)
mapping:
  e891b621-8e4d-44ab-84ac-eeb6d92e5120:
    - source
    - sink
  2374cd39-a46f-4aaf-9a66-0b64e41f6068:
    - operator
  1231c70a-a309-47c3-a6d1-5c9370eed0f5:
    - source-builtin-sub
    - sink-builtin-pub

Let us now explain what each section does.

(optional) Vars

This section is used to tell Zenoh-Flow how to do string replacements in this descriptor (and only this one). More details can be found here.

(optional) Configuration

This section allows passing a dictionary of key-value pairs to all the nodes involved. This can be useful, for instance, to run several times the same node but with slightly different parameters or to modify the behaviour of a node without having to recompile it.

More details can be found here.

Sources

This section groups the declaration of all the sources used.

Zenoh-Flow supports 2 ways of declaring Sources: (i) inline within the data flow descriptor and (ii) in an external file. In the data flow descriptor above, the external declaration source-ext has been commented out.

An inline or external declaration contains:

  • a library indicating where to find the shared library containing the implementation of the Source,
  • (optional) a configuration of key-value pairs to pass to the implementation at run-time,
  • the names of the outputs.

An id must be associated to a Source declaration (inline or external) to make it unique within this data flow application.

Lastly, as illustrated with the source-builtin-sub Source, Zenoh-Flow facilitates the use of Zenoh subscribers acting as Sources through a specific declaration containing (in addition to an id) a single section zenoh-subscribers.

Operators

The same rules apply as for the Sources (and the Sinks), with the only difference that an Operator declaration must contain an inputs and an outputs section.

Sinks

The same rule apply as for the Sources (and the Operators), with the only difference that a Sink declaration must only contain an inputs section.

Similarly to the built-in subscribers for Sources, Zenoh-Flow facilitates the use of Zenoh publishers acting as Sinks through a specific declaration containing (in addition to an id) a single section zenoh-publishers.

Links

The links section in a data flow descriptor describes how the different nodes are connected. A link goes from the output of a node to the input of another one.

Hence, each link is composed of two subsections:

  • a from subsection that contains:
    • the id of the node which is sending data,
    • the output,
  • a to subsection that contains:
    • the id of the node which is receiving data,
    • the input.

The following description connects the output out of foo to the input in of bar:

- from:
    id: foo
    output: out
  to:
    id: bar
    input: in

💡 Zenoh-Flow will check the validity of links before instantiating a data flow. In particular, it will ensure that:

  1. all links go from an output to an input,
  2. an input only receives data from a single output,
  3. all ports are connected --- i.e. no node has an input or an output that is not connected.

There are no additional constraints on the links: loops are accepted, the same output can go to multiple inputs, several outputs can go to the same input, etc.

(optional) Mapping

Zenoh-Flow leverages this section to control on which daemons the different parts of a data flow run.

No mapping: single, randomly-selected, daemon

If the mapping section is absent, Zenoh-Flow will default to running all the nodes on one daemon. This daemon is randomly selected if several are available.

UUID-based mapping

Note

See this section of the Zenoh-Flow Daemon configuration to see how to control the UUID assigned to a Zenoh-Flow Daemon.

At this stage of the development of Zenoh-Flow, to deploy a data flow, each daemon must have access to the shared library or scripts. In other words, the library field in the descriptor file of the node must point to an accessible location on the file system where the daemon is running.

The same holds for the descriptor fields pointing to the descriptors.

Hence, if a data flow should be deployed on several daemons, the shared library or scripts must be uploaded on the device where the daemons are running and the different paths updated accordingly: (i) the path of the descriptors and, in each descriptor, (ii) the path of the implementation of the node. Note that, a daemon only needs access to the nodes it runs.

Assuming that the paths are correct and the implementations present on the device, to inform Zenoh-Flow of where each node should run one needs to write a mapping.

mapping:
  e891b621-8e4d-44ab-84ac-eeb6d92e5120:
    - source
    - sink
  2374cd39-a46f-4aaf-9a66-0b64e41f6068:
    - operator
  1231c70a-a309-47c3-a6d1-5c9370eed0f5:
    - source-builtin-sub
    - sink-builtin-pub

The above mapping indicates that:

  • the node whose id is equal to source and sink should run on the daemon whose UUID is equal to e891b621-8e4d-44ab-84ac-eeb6d92e5120,
  • the node whose id is equal to operator should run on the daemon whose UUID is equal to 2374cd39-a46f-4aaf-9a66-0b64e41f6068,
  • the node whose id is equal to source-builtin-sub and sink-builtin-pub should run on the daemon whose name is equal to 1231c70a-a309-47c3-a6d1-5c9370eed0f5.

The ids are given to the nodes in the data flow descriptor. The UUID can be given to the daemons in their respective configuration file and are, otherwise, randomly generated when the Daemon is started.

⚠️ All nodes that are absent from the mapping will run, by default, on the randomly selected daemon!

Clone this wiki locally