Experiment/dataset update flow

[sergiimk]

No description provided.

[sergiimk]

Keeping this field up-to-date will require a lot of writes to event store. Imagine upon every push to kafka by a device that sends data every second we had to update this state in the flow. Since this type is a "start condition", not a state of batching I suggest we remove this field. And if we really want to show the state of a batch - we'll add a separate API to query it from Kafka or elsewhere when needed.

[zaychenko-sergei]

This isn't used yet. I removed, but we will have to have some source of truth in this regard.
Following the example with the IoT device, we still have to create an every-second secondary trigger event and must show each in the History view.

Currently this trigger is an empty struct:

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FlowTriggerPush {
    // TODO: source (HTTP, MQTT, CMD, ...)
}

similarly, we don't store yet the number of records added for the flow outcome, and can't use it yet for the derived dataset throttling:

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FlowTriggerInputDatasetFlow {
    pub input_dataset_id: DatasetID,
    pub input_flow_type: DatasetFlowType,
    pub input_flow_id: FlowID,
}

Perhaps both these structures should have information about the offset change. Sum of these numbers might be the criteria we are looking for.

[sergiimk]

Hm, I was imaging it differently:

• Kafka controller will know about batching configuration
• When Kafka queue becomes non-empty - it will trigger the flow (to show users that batch is pending)
• While more records are flowing - it will NOT touch the flow system, until the configured batch size is reached to finally trigger the flow

This way secondary trigger events will appear only for other types of trigger, e.g. "manual".

While the flow is already in the state of waiting for a batch - I don't think showing more data arriving as secondary triggers is useful, and can result in too many updates.

[zaychenko-sergei]

Assuming this suggestion, without the visibility of Kafka's queue state, let's say, showing something like accumulated 17,245/25,000 records, the user will not understand what is the flow waiting for.

[sergiimk]

Is this throttling config related to:

• "don't update free datasets more often than 30 min"
• or "after first push creates a flow - keep batching data for 10 more minutes before running ingest task"?

I feel like the former should be a system-wide setting, not something we store per flow.

If its latter - we should have a different name for it, e.g. "batching period".

[sergiimk]

Should we have something like Probe for the flow system too, so we could test it manually without spawning expensive tasks?

Perhaps a Probe flow could run Probe tasks on a certain dataset and spawn Probe flows for all downstream, like a cascading update but the actual work will be a no-op?

[zaychenko-sergei]

I would likely introduce something like this when testing.

[zaychenko-sergei]

There is no mechanism to pass any parameters to the flow yet. This would be necessary for Probe idea to make sense.

[sergiimk]

Some minor suggestions, but overall I'm really happy with the state of this PR 👍