Make source variables filtering more powerful

[aragornfbm]

According to the documentation, the "job.variables" property should contain the name of a custom header that is defined in the source task and that contains a comma-separated list of variables that should be included in the produced Kafka message.

In the current implementation (0.32.0), the value put in "job.variables" is used as the actual list of variable names, not as a reference to the header containing the list of variables.

[berndruecker]

Ok - thx for reporting. I will update the docs.

[aragornfbm]

It would be better if this worked like it was initially planned, so that we can define the list of variables to send inside each workflow, not as part of the source connector configuration. This would give us more flexibility.

[berndruecker]

In the readme (https://github.com/camunda-community-hub/kafka-connect-zeebe) it says:

You can filter the variables being sent to Kafka by adding a custom header to the "sendMessage" task with the configuration option "job.variables". Set the value of this key to a comma-separated list of variables to pass to Kafka.

This might be a bit misleading - so we should be clear that this is defined by the source.json instead. I will fix this right away.

This property is used to limit querying the variables for every job.

If we would want to filter based on a task header, this would be a feature request that requires a bit of effort to make happen. And I guess we would need to query all variables - and then "just" limit what is sent to Kafka, as everything else would mean we somehow need to parse the BPMN model to read task header configuration statically (we might want to make that easy with the upcoming connector framework - but not yet there).

Could you imagine providing a PR for the "fetch all variables but limit what we send out by task header config" feature @aragornfbm?

[aragornfbm]

Let me see what I can do @berndruecker

[aragornfbm]

@berndruecker I created the pull request #99 to implement this feature.