Suggestion regarding kafka keys

[cameronbraid]

Kafka only guarantees ordering of messages within a kafka partition - this is determined by the message key.

Currently the exporter uses a RecordID as they Kafka key.

This RecodID includes the zeebe partition and offset values therefore the kafka key is different for every message produced to kafka.

Therefore the 'ordering within a partition' that zeebe emits is not guaranteed for kafka consumers.

So, to fix this, the kafka message key should be changed to be the zeebe partiton id only.

[tjwp]

I raised a similar question here: #29

A downside to using the Zeebe partition id as the message key with Kafka is that it limits the number of Kafka partitions that can be used. Also, I think there is a greater chance of the partitions being unbalanced since certain activity only goes to Zeebe node 1.

[cameronbraid]

Sure, though I would rather have unbalanced partitions, but with correct ordering

[cameronbraid]

However using workflowInstanceKey should provide the same ordering guarantees for all workflow instance related records.. And will spread the data out over more kafka partitions.. so that sounds like a good idea

The other record types would be much lower volumes of data and should probably have keys chosen based on their individual ordering requirements

[cameronbraid]

One other thing to consider for ordering is the ordering is only guaranteed within a topic. The default configuration for kafka-exporter uses a different topic per value type

[npepinpe]

That's the sample config - by default, if you don't configure anything at all, then all records are sent to the zeebe topic.