Dora record

[haixuanTao]

This is a first experimental dora-record node that uses arrow file format to record data.

Advantage of this format

• Being zero-copy and memory mapped. See: https://huggingface.co/docs/datasets/about_arrow
• Can use plenty of tools such as hadoop, pandas, torch, R, matlab and so on, very quickly without preprocessing
• Does not need to be serialized-deserialized which can be handy when partially reading data.
• Should benefit any column-based computation of data such as training and so on.

Advantage of dora-record

The benefit of using dora-record is that it is going to save in column format all input received and they will be split across several files.
We can merge those files using the trace_id that is going to be shared when using opentelemetry features.

• trace_id can also be queried from UI such as jaeger UI, influxDB and so on...
• trace_id keep tracks of the logical flow of data, compared to timestamp based merging that might not reflect the actual logical flow of data.

We can also replay messages according to the timestamp of the message.

Limitation of the current implementation

• data spread across multiple files. As the data is in columnar format, putting everything into one file with no prior information might make the data very sparse.
• As the data is spread accross muiltiple files, a merging step is required that can be costly.
• Bigger than alternative. Arrow format as it is not serialized, might be bigger than alternative. It might be interesting to also compress the data later on.

Currently not implemented in this version:

• Variable file name and path ( even remote )
• Async concurrently writen data, blocked by Async IPC Reader/Writer apache/arrow-rs#1207
• Handle dora system event for replay. Not sure how to handle replaying dora event.
• Additional metadata parameters logging.
• Add batch recording of multiple row to avoid single row overhead
• Possibly implement compression and also different files format

This should be reviewed and merged after #353

[phil-opp]

Thanks! I haven't reviewed the write_event function in detail yet, but the rest looks good.

[phil-opp]

The new dora record node never finishes as it keeps on listening for dora timer messages. Thus, a timeout error occurs on CI:

[haixuanTao]

The new dora record node never finishes as it keeps on listening for dora timer messages. Thus, a timeout error occurs on CI:

I'll remove logging the ticker for now in that case

[phil-opp]

Looks good, thanks!