Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][connector] KCA: use reflection to get pulsar-client impl classes #17835

Merged
merged 1 commit into from
Sep 26, 2022

Conversation

nicoloboschi
Copy link
Contributor

@nicoloboschi nicoloboschi commented Sep 26, 2022

Motivation

With the latest changes to the KCA module (#16098) now you get this error if the message is batched:

java.lang.ClassCastException: class org.apache.pulsar.client.impl.BatchMessageIdImpl cannot be cast to class org.apache.pulsar.client.impl.MessageIdImpl (org.apache.pulsar.client.impl.BatchMessageIdImpl is in unnamed module of loader java.net.URLClassLoader @2ff4acd0; org.apache.pulsar.client.impl.MessageIdImpl is in unnamed module of loader org.apache.pulsar.common.nar.NarClassLoader @6ff415ad)
	at org.apache.pulsar.io.kafka.connect.KafkaConnectSink.getMessageOffset(KafkaConnectSink.java:327) ~[pulsar-io-kafka-connect-adaptor-?.jar]
	at org.apache.pulsar.io.kafka.connect.KafkaConnectSink.toSinkRecord(KafkaConnectSink.java:408) ~[pulsar-io-kafka-connect-adaptor-?.jar:]
	at org.apache.pulsar.io.kafka.connect.KafkaConnectSink.write(KafkaConnectSink.java:121) [pulsar-io-kafka-connect-adaptor-?.jar:]
	at org.apache.pulsar.functions.instance.JavaInstanceRunnable.sendOutputMessage(JavaInstanceRunnable.java:368) [com.datastax.oss-pulsar-functions-instance-?-SNAPSHOT.jar:?-SNAPSHOT]
	at org.apache.pulsar.functions.instance.JavaInstanceRunnable.handleResult(JavaInstanceRunnable.java:351) [com.datastax.oss-pulsar-functions-instance-?-SNAPSHOT.jar:?-SNAPSHOT]
	at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:300) [com.datastax.oss-pulsar-functions-instance-?-SNAPSHOT.jar:?-SNAPSHOT]

The root problem is that connectors should not bundle the Pulsar client impl because it's provided at runtime by the Java Functions framework.

Modifications

  • Use reflection to get the custom sequenceId
  • doc-not-needed
    (Please explain why)

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Sep 26, 2022
@nicoloboschi nicoloboschi requested review from dlg99, eolivelli and hangc0276 and removed request for dlg99 and eolivelli September 26, 2022 07:56
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@BewareMyPower
Copy link
Contributor

Hi Nico, I met a similar error (see https://github.com/BewareMyPower/pulsar-sink-demo) when I'm trying the connector.

because it's provided at runtime by the Java Functions framework.

Could you explain a bit more? You mean the Pulsar client impl JAR is in the classpath and the connector is loaded by the Java Functions framework so the connector cannot access classes in Pulsar broker's classpath?

BTW, if a connector cannot access methods from modules like pulsar-client-original directly, it would be a bad experience for Pulsar connector developers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants