Skip to content

3.0.0
Compare
Choose a tag to compare
@npepinpe npepinpe released this 28 May 17:00
3.0.0
370bd04

What's Changed

This release brings, most importantly, much improved fault tolerance. The exporter can now safely tolerate faults such as the Kafka broker(s) failing, being unavailable randomly, and most application level errors returned by the broker(s) (e.g. incorrect offsets). Additionally, records are now also guaranteed to be correctly ordered.

NOTE: the exporter cannot automatically handle configuration errors - for example, if you enter the wrong Kafka broker URLs, this is not something we can detect. It will simply be detected as an unavailable broker. It's important when doing configuration changes that you test them beforehand, and monitor your logs immediately after deployment.

Fault tolerance via transactions

To simplify the error handling and retry mechanism, the exporter now makes use of transactions (see the Confluent blog for a quick intro). What this means is that consumers should, from now on, specify a read_committed isolation level (see here for more on this). Additionally, if you're running with less than 3 Kafka brokers, you will need to configure the transaction state correctly to allow for a lower replication factor and minimum ISR. For example, if you're running a single node Kafka broker, you would configure both parameters in your broker to 1. While I hope to offer an alternative to transactions in the future, it makes the exporter logic much, much simpler - by being able to handle errors in a single point instead of handling failures on every possible record and having to deal with additional out-of-order issues.

Guaranteed ordering

The ordering is now maintained by adopting a partitioning scheme which closes models Zeebe's. Since records in Zeebe are causal w.r.t their partition, all records from the same Zeebe partition must be exported to the same Kafka partition in order to preserve this order. This means you should configure your target topics to have at most the same number of partitions as Zeebe's. You can read more about this in the README.

Features

  • Improved fault tolerance via transactions (#4) @npepinpe

Bug fixes

  • Preserves ordering of exported records (#39) @npepinpe
  • Reduce concurrent requests and remove explicit configuration parameter (#34) @npepinpe
  • Fix job-batch topic configuration (#33) @tjwp

Tests

📦 Dependency updates

  • deps(master): bump version.jackson from 2.12.1 to 2.12.2 (#127) @dependabot
  • deps(master): bump zeebe-protocol-immutables from 1.1.1 to 1.2.0 (#128) @dependabot
  • deps(master): bump version.zeebe from 0.26.0 to 0.26.1 (#129) @dependabot
  • deps(master): bump spotbugs-annotations from 4.2.0 to 4.2.2 (#123) @dependabot
  • deps(master): bump junit from 4.13.1 to 4.13.2 (#118) @dependabot
  • deps(master): bump elasticsearch-rest-client from 7.10.1 to 7.11.2 (#126) @dependabot
  • deps(master): bump mockito-core from 3.7.0 to 3.8.0 (#121) @dependabot
  • deps(master): bump version.testcontainers from 1.15.1 to 1.15.2 (#117) @dependabot
  • deps(master): bump version.junit-jupiter from 5.7.0 to 5.7.1 (#111) @dependabot
  • deps(master): bump maven-checkstyle-plugin from 3.1.1 to 3.1.2 (#109) @dependabot
  • deps(master): bump assertj-core from 3.18.1 to 3.19.0 (#107) @dependabot
  • deps(master): bump sonar-maven-plugin from 3.7.0.1746 to 3.8.0.2131 (#102) @dependabot
  • deps(master): bump version.jackson from 2.12.0 to 2.12.1 (#99) @dependabot
  • deps(master): bump kafka-clients from 2.6.0 to 2.7.0 (#93) @dependabot
  • deps(master): bump version.jackson from 2.11.3 to 2.12.0 (#80) @dependabot
  • deps(master): bump zeebe-test-container from 0.33.0 to 1.0.1 (#71) @dependabot
  • deps(master): bump version.zeebe from 0.25.2 to 0.26.0 (#97) @dependabot
  • deps(master): bump elasticsearch-rest-client from 7.10.0 to 7.10.1 (#89) @dependabot
  • deps(master): bump version.testcontainers from 1.15.0 to 1.15.1 (#91) @dependabot
  • deps(master): bump mockito-core from 3.6.28 to 3.7.0 (#95) @dependabot
  • deps(master): bump version.zeebe from 0.25.1 to 0.25.2 (#85) @dependabot
  • deps(master): bump httpcore from 4.4.13 to 4.4.14 (#83) @dependabot
  • deps(master): bump spotbugs-annotations from 4.1.4 to 4.2.0 (#79) @dependabot
  • deps(master): bump mockito-core from 3.6.0 to 3.6.28 (#76) @dependabot
  • deps(java): bump maven-checkstyle-plugin from 3.0.0 to 3.1.1 (#59) @dependabot
  • deps(master): bump elasticsearch-rest-client from 6.8.13 to 7.10.0 (#67) @dependabot
  • deps(master): bump httpcore from 4.4.5 to 4.4.13 (#66) @dependabot
  • deps(master): bump spotbugs-annotations from 3.1.12 to 4.1.4 (#64) @dependabot
  • deps(master): bump version.zeebe from 0.24.3 to 0.25.1 (#63) @dependabot
  • deps(java): bump maven-assembly-plugin from 3.1.1 to 3.3.0 (#60) @dependabot
  • deps(java): bump maven-surefire-plugin from 2.22.1 to 2.22.2 (#56) @dependabot
  • deps(java): bump assertj-core from 3.16.1 to 3.18.1 (#53) @dependabot
  • deps(java): bump mockito-core from 3.0.0 to 3.6.0 (#46) @dependabot
  • deps(java): bump version.testcontainers from 1.14.3 to 1.15.0 (#47) @dependabot
  • deps(java): bump kafka-clients from 2.3.0 to 2.6.0 (#45) @dependabot
  • deps(java): bump maven-failsafe-plugin from 2.22.1 to 2.22.2 (#44) @dependabot
  • deps(java): bump version.jackson from 2.11.1 to 2.11.3 (#43) @dependabot
  • chore(deps): bump junit from 4.12 to 4.13.1 (#41) @dependabot
  • deps(master): bump zeebe-test-container from 1.0.1 to 3.0.0 (#162) @dependabot

Contributors

@npepinpe, @saig0, @tjwp

Assets 5
Loading