3.1.1

What's Changed

• Re-enable QA tests in CI which had been mistakenly disabled when moving the CI to GitHub Actions
• Fix jar-with-dependencies assembly, broken in 3.1.0, which made it so the exporter itself could not be found.

Full Changelog: 3.1.0...3.1.1

3.1.0

What's Changed

This release bumps the default Zeebe dependency to 1.3.x, and switches assembling the exporter fat JAR to use the maven-assembly-plugin. This should ensure that all required classes, even optional ones included in the Kafka client library, are present at run time.

• Recommend loading as an external exporter via the JAR path for Zeebe 1.0.1 by @npepinpe in #188
• Use assembly to ensure all classes are there by @npepinpe in #238

Full Changelog: 3.0.0...3.1.0

3.0.0

What's Changed

This release brings, most importantly, much improved fault tolerance. The exporter can now safely tolerate faults such as the Kafka broker(s) failing, being unavailable randomly, and most application level errors returned by the broker(s) (e.g. incorrect offsets). Additionally, records are now also guaranteed to be correctly ordered.

NOTE: the exporter cannot automatically handle configuration errors - for example, if you enter the wrong Kafka broker URLs, this is not something we can detect. It will simply be detected as an unavailable broker. It's important when doing configuration changes that you test them beforehand, and monitor your logs immediately after deployment.

Fault tolerance via transactions

To simplify the error handling and retry mechanism, the exporter now makes use of transactions (see the Confluent blog for a quick intro). What this means is that consumers should, from now on, specify a read_committed isolation level (see here for more on this). Additionally, if you're running with less than 3 Kafka brokers, you will need to configure the transaction state correctly to allow for a lower replication factor and minimum ISR. For example, if you're running a single node Kafka broker, you would configure both parameters in your broker to 1. While I hope to offer an alternative to transactions in the future, it makes the exporter logic much, much simpler - by being able to handle errors in a single point instead of handling failures on every possible record and having to deal with additional out-of-order issues.

Guaranteed ordering

The ordering is now maintained by adopting a partitioning scheme which closes models Zeebe's. Since records in Zeebe are causal w.r.t their partition, all records from the same Zeebe partition must be exported to the same Kafka partition in order to preserve this order. This means you should configure your target topics to have at most the same number of partitions as Zeebe's. You can read more about this in the README.

Features

• Improved fault tolerance via transactions (#4) @npepinpe

Bug fixes

• Preserves ordering of exported records (#39) @npepinpe
• Reduce concurrent requests and remove explicit configuration parameter (#34) @npepinpe
• Fix job-batch topic configuration (#33) @tjwp

Tests

• Introduce more robust QA tests (#42) @npepinpe

📦 Dependency updates

• deps(master): bump version.jackson from 2.12.1 to 2.12.2 (#127) @dependabot
• deps(master): bump zeebe-protocol-immutables from 1.1.1 to 1.2.0 (#128) @dependabot
• deps(master): bump version.zeebe from 0.26.0 to 0.26.1 (#129) @dependabot
• deps(master): bump spotbugs-annotations from 4.2.0 to 4.2.2 (#123) @dependabot
• deps(master): bump junit from 4.13.1 to 4.13.2 (#118) @dependabot
• deps(master): bump elasticsearch-rest-client from 7.10.1 to 7.11.2 (#126) @dependabot
• deps(master): bump mockito-core from 3.7.0 to 3.8.0 (#121) @dependabot
• deps(master): bump version.testcontainers from 1.15.1 to 1.15.2 (#117) @dependabot
• deps(master): bump version.junit-jupiter from 5.7.0 to 5.7.1 (#111) @dependabot
• deps(master): bump maven-checkstyle-plugin from 3.1.1 to 3.1.2 (#109) @dependabot
• deps(master): bump assertj-core from 3.18.1 to 3.19.0 (#107) @dependabot
• deps(master): bump sonar-maven-plugin from 3.7.0.1746 to 3.8.0.2131 (#102) @dependabot
• deps(master): bump version.jackson from 2.12.0 to 2.12.1 (#99) @dependabot
• deps(master): bump kafka-clients from 2.6.0 to 2.7.0 (#93) @dependabot
• deps(master): bump version.jackson from 2.11.3 to 2.12.0 (#80) @dependabot
• deps(master): bump zeebe-test-container from 0.33.0 to 1.0.1 (#71) @dependabot
• deps(master): bump version.zeebe from 0.25.2 to 0.26.0 (#97) @dependabot
• deps(master): bump elasticsearch-rest-client from 7.10.0 to 7.10.1 (#89) @dependabot
• deps(master): bump version.testcontainers from 1.15.0 to 1.15.1 (#91) @dependabot
• deps(master): bump mockito-core from 3.6.28 to 3.7.0 (#95) @dependabot
• deps(master): bump version.zeebe from 0.25.1 to 0.25.2 (#85) @dependabot
• deps(master): bump httpcore from 4.4.13 to 4.4.14 (#83) @dependabot
• deps(master): bump spotbugs-annotations from 4.1.4 to 4.2.0 (#79) @dependabot
• deps(master): bump mockito-core from 3.6.0 to 3.6.28 (#76) @dependabot
• deps(java): bump maven-checkstyle-plugin from 3.0.0 to 3.1.1 (#59) @dependabot
• deps(master): bump elasticsearch-rest-client from 6.8.13 to 7.10.0 (#67) @dependabot
• deps(master): bump httpcore from 4.4.5 to 4.4.13 (#66) @dependabot
• deps(master): bump spotbugs-annotations from 3.1.12 to 4.1.4 (#64) @dependabot
• deps(master): bump version.zeebe from 0.24.3 to 0.25.1 (#63) @dependabot
• deps(java): bump maven-assembly-plugin from 3.1.1 to 3.3.0 (#60) @dependabot
• deps(java): bump maven-surefire-plugin from 2.22.1 to 2.22.2 (#56) @dependabot
• deps(java): bump assertj-core from 3.16.1 to 3.18.1 (#53) @dependabot
• deps(java): bump mockito-core from 3.0.0 to 3.6.0 (#46) @dependabot
• deps(java): bump version.testcontainers from 1.14.3 to 1.15.0 (#47) @dependabot
• deps(java): bump kafka-clients from 2.3.0 to 2.6.0 (#45) @dependabot
• deps(java): bump maven-failsafe-plugin from 2.22.1 to 2.22.2 (#44) @dependabot
• deps(java): bump version.jackson from 2.11.1 to 2.11.3 (#43) @dependabot
• chore(deps): bump junit from 4.12 to 4.13.1 (#41) @dependabot
• deps(master): bump zeebe-test-container from 1.0.1 to 3.0.0 (#162) @dependabot

Contributors

@npepinpe, @saig0, @tjwp

Zeebe Kafka Exporter 2.0.0

2.0.0

This new Zeebe Kafka Exporter releases is the first Zeebe 0.23+ compatible release. It was manually tested with both 0.23.x (up to 0.23.4) and 0.24.x (up to 0.24.1). It is not compatible with Zeebe versions lower than 0.23.

Changes

New serialization format

I've dropped the Protobuf format and replaced it with JSON. The previous format required users to use the specific Protobuf schema but wasn't well integrated with the Kafka ecosystem (e.g. via Schema Registry), was not human readable, was hard to use with Kafka Connect out of the box, and generally made for a worse user experience.

While JSON is schemaless, it addresses the points above by being easy to read, easy to use in the Kafka ecosystem, and doesn't require any additional dependencies to serialize/deserialize. The serde module included is entirely optional and can be ignored, and users will still be able to make use of the published records.

Configuration changes

On top of updating the configuration for Zeebe 0.23+, there were previously some issues with the free-form producer configuration. The format was changed now to avoid any issues by delegating the processing to Kafka. Essentially, to overwrite or add additional producer configuration (e.g. to support SASL, or OAuth), you specify the configuration as a properties file. So in your YAML file, you would add it as a multi-line string which would be the contents of your normal producer properties file. The README contains an example for this.

Minor improvements

A few other touchups were done as part of the 2.0 release. The code is now fully documented, and a new QA module was added, replacing the old examples module. The QA module now tests the exporter in an as-close-to-production deployment as possible, by leveraging Testcontainers to create a new Zeebe container, a new Kafka container, and injecting the exporter JAR into the Zeebe container along with its configuration file. The tests included are also a good way to show how you can configure the exporter through a mix of YAML and environment variables. It also acts as a proof of concept for exporter authors on how to test their own exporters.

Zeebe Kafka Exporter 1.1.0

Zeebe Kafka Exporter 1.1.0

Upgrade the exporter to be compatible with 0.20.0 (possibly 0.19.0 as well, but nothing beyond that).

Breaking changes

Some configuration options were removed, some added, and some renamed:

1. awaitInFlightRecordTimeout was removed as it was not particularly useful
2. inFlightRecordCheckIntervalMs was added to specify how often the exporter should drain the in flight records queue
3. requestTimeout and closeTimeout were migrated to requestTimeoutMs and closeTimeoutMs to remove a dependency on DurationUtil and use plain types

Zeebe Kafka Exporter 1.0.0

1.0.0