[Bug]: Kafka module cannot find java class written using java 17

[rahulgulati89]

Dear TestContainers team, it might be not a bug so if it's necessary please downgrade it.

We use Kafka TestContainers module and it helps us very much. Thank you for your work.

Could you please tell me did you have any specific issues/challenges making Kafka test containers (TC) workable for java17. It looks like that Kafka TestContainers module cannot find the jar compiled using java 17.

Let me explain the reason for this question.
Our project is Single Message Transfofmation(SMT) which is a part of Kafka Confluent Connect platform. SMT is a java class packed in jar file, when it's provided to Kafka Confluent Connect it's picked and used during a message processing. The project is written in java 8 and we use TestContainers to test it.
For our tests we created a wrapper around org.testcontainers.containers.GenericContainer and build our own Kafka Confluent Connect container using org.testcontainers.images.builder.ImageFromDockerfile. We do it this way:

ImageFromDockerfile()
.withDockerfileFromBuilder { builder ->
builder
.from("artifacts.cf.saxo/docker/confluentinc/cp-kafka-connect:$confluentVersion")
Our tests work perfect. SMT jar is copied into the container and works as expected. All this work was done using java 8.

Now we decided to migrate our SMT which is written in java 8 and tested using TestContainers to java17 because java8 won't be supported by Confluent after version 7.4.

We use maven to build the SMT project. We built a new jar file compiled using java 17, we checked the jar on the real environment with Confluent platform run on java 17 and it works correctly.
The problem is that our TestContainers tests which reproduce the same flow we have on the real environment are failing now, saying that a class from jar file cannot be found. I've checked the jar, the class is there. It looks like that Kafka TestContainers module cannot find the jar compiled using java 17.

The only difference in two versions of our SMT project is the java version which is used and it's handled by maven plugin, all other dependencies are the same: we use the same version of TestContainers, we import the same library.

 <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.11.0</version>
        <configuration>
            <source>1.8</source>
            <target>1.8</target>
        </configuration>
    </plugin>

we changed the config to be build using java17

  <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.11.0</version>
        <configuration>
            <source>17</source>
            <target>17</target>
        </configuration>
    </plugin>

The logic is the following:

we create Kafka Confluent Connect TC from docker image
copy the generated smt jar inside the container
communicate with Kafka Confluent Connect during the tests
I understand that the question is a bit generic but may be you can give any hint which can lead to the described above experience.

Any obstacles which you overcame making TestContainers workable for java 17 would be really great to know.

Some More details

Module
Kafka

Testcontainers version
1.18.3

Using the latest Testcontainers version?
Yes

Host OS
WSL2 on Windows, Linux

Host Arch
x64

Docker version
Client: Docker Engine - Community
Cloud integration: 1.0.17
Version: 20.10.8
API version: 1.41

[eddumelendez]

saying that a class from jar file cannot be found. I've checked the jar, the class is there. It looks like that Kafka TestContainers module cannot find the jar compiled using java 17.

This is not clear to me. The jar in your host machine or in the container? How are you moving the jar from host to the container?

Also, if you were using Java 8 previously to build the SMT, Confluent should also support it, right? So, there is also an upgrade of Confluent platform. What I mean is if there is a different location in this new version 7.4.x

In order to do a analysis, please provide an minimal example that reproduces the issue. It can be shared as a zip or in a github repository. It shouldn't be your exact project. Some basic example with java 8 showing that works and the one upgrading to java 17.

[alutski]

Hello @eddumelendez,

Thank you for joining the discussion.
I will clarify some moments now and later today I will provide the minimum setup which will reproduce the issue.

The jar is built in the host machine and then it's copied inside the Kafka Confluent Connect container to run tests.

Regarding the Confluent version it's what pushed us to upgrade to java 17 but it's not relevant for the tests. The tests are run for Confluent 7.2.1 and smt jar is copied into the same location inside test container.

What's more interesting we use the same Confluent version for java 8 and java 17 jars tests. The test setup is completely the same and the only difference is that taking the same project, the same TestContainers library and just changing the jdk which is used to build the jar we see the different behavior in tests. I understand that "just changing the jdk" makes a huge difference but eventually it looks like something missed in the jar processing inside TestContainers but I might be wrong here and your experience here is very much appreciated.

Let me remove all unnecessary things and make a clean test setup so it would be easy for you to investigate.

Thank you for readiness to help.

Kind regards,
Anatol Lutski

[alutski]

Hello @eddumelendez,

sorry for delay with the project provision. It took me sometime to do a proper cleanup to provide the minimal setup.
I cut off all functionality and left only the part which demonstrates the strange behaviour.

There are two parts to help to do the investigation:

1. struct-to-string-java8 and struct-to-string-java17 are completely the same projects written in java which contain the shrinked SMT code and StructToStringContainerTest class which uses TestContainers tests.
   The only difference between two projects is the configuration of maven-compiler-plugin (source and targer are 1.8 and 17 accordingly). Both projects use the same version of our TestContainers wrapper
   which itself uses the latest version of TestContainers library 1.18.3
2. kafka-streams-testcontainers is our wrapper written in Kotlin around TetstContainers library and it is used in the tests to create the necessary containers, for this minimal version i've left only ConnectContainer.

SMT background:
Single Message Transformation (SMT) works inside connectors which are part of Kafka Confluent Connect. A connector can connect a database with a Kafka topic or other way around.
We implement SMT in java and it's provided in jar file. To start to use SMT in Confluent Connect its jar file is needed to be copied into Confluent Connect plugin directory
and SMT class name should be provided in a connector config (struct-to-string-java:StructToStringContainerTest:72).
In this shrinked version the issue happens with the connector registration.

How we develop and test:

• kafka-streams-testcontainers is a separate project, wrapper aroung TestContainers library. It's imported inside smt project to be used during tests.
• struct-to-string-java project is the one where SMT logic is implemented. Project is build using maven. Inside its tests we use our test containers wrapper, please have a look at struct-to-string-java:StructToStringContainerTest.

To run test we do 'mvn clean test', during the compilation smt jar is created, then it's copied to Confluent Connect TestContainer and uses during the transformation.

Code highlights:
struct-to-string-java:StructToStringContainerTest:21 Connect Container is initialized using a TestContainers, one of arguments is smtJarResource - the name of SMT.
kafka-streams-testcontainers:KafkaTestContainerUtils:35 the function in our wrapper to create Connect Container and on the line kafka-streams-testcontainers:KafkaTestContainerUtils:49 we copied smt taken from the test resources of the tested project to Docker Image. When TestContainer is started for java 8 or java 17 project the necessary SMT jar is presented inside the running container, BUT somehow when we run test for a project with java 17 then TestContainers cannot find the SMT class in the SMT jar.
You can see that running mvn clean test for two projects give different result only based on different jdk usage. The error you can see below and it says it's not possbile to find a class for SMT.

The error happens during calling registerSinkConnector at struct-to-string-java:StructToStringContainerTest:39.

testMapInComplexOptionalComplexOptional(com.saxobank.datahub.smts.StructToStringContainerTest): 
Unexpected response: Response{protocol=http/1.1, code=400, message=Bad Request, url=http://localhost:60205/connectors/} ; 
Response Body: {"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\n
Invalid value com.saxobank.datahub.smts.StructToString$Value for configuration transforms.structToString.type: Class com.saxobank.datahub.smts.StructToString$Value could not be found.\n
Invalid value null for configuration transforms.structToString.type: Not a Transformation\n
You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}

To run a test please do: mvn clean test on struct-to-string-java8 and struct-to-string-java17. The only thing that initially it's necessary to build the wrapper project and copy the artifact to your local .m2 folder.

Why are we asking help from TestContainers community:
because we did the same setup on the real java8 and java17 environments and SMT works correctly. We've also checked the content of generated jars and the necessary class is presented at the right place both in java8 and java17 cases.

Please ask any question and I will clarify the situation or add additional details.

Kind regards,
Anatol

shrinked_java8_17_setup.zip

[eddumelendez]

is your kafka-streams-testcontainers module missing pom.xml?

[eddumelendez]

I see the connector is copied /usr/share/java/datahub-smts/ but the CONNECT_PLUGIN_PATH env var is /usr/share/java, /usr/share/confluent-hub-components/. I don't think this is a Testcontainers issue at all but about the connector setup in kafka connect itself. Testcontainers responsibility is to make sure the containers are ready to be used and as you mentioned the containers start and the files are properly copied.

If a reproducer that I can execute is properly submit I can still take a look.

[alutski]

@eddumelendez thank you for taking part in the discussion. Unfortunately CONNECT_PLUGIN_PATH isn't the reason of the strange behavior. I've just checked it by copying the smt to the default location the situation was the same: works for java 8, failed for java 17.

I really want to highlight that our test containers wrapper code stays the same while running SMT tests.
I will try to get rid of our wrapper, start to use pure TestContainers classes in SMT test suite and then i will come back to you with the results.

[alutski]

is your kafka-streams-testcontainers module missing pom.xml?

kafka-stream-testcontainers is using gradle to build, all dependencies are inside build.gradle.kts file

[eddumelendez]

The project you shared doesn't contain any build.gradle.kts file

[alutski]

@eddumelendez my apologies, i've mistakenly deleted build.gradle.kts during the preparation. I've attached the file here.

build.gradle.kts.zip

[alutski]

Hello @eddumelendez,

To continue investigation please use a new archive attached to this message.
The difference with the previous one: I've removed our wrapper around TestContainers library and moved all necessary code to initialize Kafka and Confluent Connect containers to this project. It should remove unnecessary dependencies which otherwise you need to setup. Please check package com.saxobank.dk.iitech.shared.testcontainers.
If you compare two projects in this new archive your will see that the only difference is the configuration of maven-compiler-plugin: 1.8 for java8 and 17 for java 17.

How do I do tests:

1. in my wsl I have
   java 8
   openjdk version "1.8.0_362"
   OpenJDK Runtime Environment (build 1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09)
   OpenJDK 64-Bit Server VM (build 25.362-b09, mixed mode)

and
java 17
openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment (build 17.0.6+10-Ubuntu-0ubuntu120.04.1)
OpenJDK 64-Bit Server VM (build 17.0.6+10-Ubuntu-0ubuntu120.04.1, mixed mode, sharing)

2. to choose the necessary java versions I run sudo update-alternatives --config java This command change the java version used by maven as well
3. then I run tests mvn clean test and it passes for java 8 but failed for java 17. There is no actual SMT logic test, I removed it to make the investigation easier. The test just tries to register a connector which the config referencing the smt and as a result of this reference Confluent Connect tries to initialize smt. It goes well while running test for java 8 but it fails for java17 because Connect cannot find the necessary class inside any jar despite the right jar is provided (KafkaTestContainerUtils:38).

In one of your replies you gave suggestion to check the plugin path for Confluent Connect, I've checked and also copied jar to the upper directory, the behavior was the same (it fails for java 17)

Please have a look, any your idea is highly appreciated.
Side note: you might need to adjust testcontainers.properties by removing the reference to our artifactory. Please change it to make the tests runnable on your PC.

Thank you for staying in the discussion.
@eddumelendez, I will be unavailable for some time so one of my colleague will provide additional details if they are needed for the investigation.

Kind regards,
Anatol Lutski

shrinked_java8_17_setup_only_java.zip

[eddumelendez]

when you access to the kafka connect container and perform java -version the result is openjdk version "11.0.15" 2022-04-19 LTS. For that reason, the project using Java 17 is not being picked. I tried compiling with Java 11 and it worked. CP images are using Java 11 at the moment, see this issue.

[alutski]

Dear @eddumelendez,

thank you so much for your investigation and pointing to Confluent thread. It’s clear now.

Kind regards,
Anatol Lutski