Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Protobuf "Any" message type for unmixed topics #382

Merged
merged 2 commits into from
Jun 19, 2022

Conversation

gebinic
Copy link
Contributor

@gebinic gebinic commented May 31, 2022

  • add select box for "Any" message type support
  • all messages must be either "Any" or not "Any" in a topic

Copy link
Collaborator

@davideicardi davideicardi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution!

Can you please just explain a little bit the changes you have made and which problems this will solve?
I never used protobuf.

@gebinic
Copy link
Contributor Author

gebinic commented Jun 1, 2022

It is a common practice to put messages from one specific type (protos in this case) in a topic but sometimes it might be better to put messages from several types in a topic, depending on your requirements. From a consumer's perspective it is not possible to handle multiple message types in one topic. Protobuf's "Any" message type allows developers to wrap each proto and send its bytes to the topic. Consumers can treat these bytes as one specific type, i.e as "Any" message type and unwrap the real message from it.

The problem with the currently provided ProtobufMessageDeserializer implementation is that it simply parses each incoming byte buffer as DynamicMessage and prints it as a json string, but takes no care about the "Any" message type. The current default string representation for "Any" is only its containing message type url, but not the real message.

So, my solution in a nutshell:

  • select "Yes" in the "Is Any proto?" select box if the topic contains "Any" message types
  • ProtobufMessageDeserializer gets a new flag at runtime which says whether the incoming byte buffer will be an "Any" message type or not
  • if this flag is true, the byte buffer is treated as "Any" message type and its containing value, i.e the real kafka message is printed as json string instead of the "Any" message type itself
  • if the flag is false (so, not an "Any" message type), the current behavior takes place

Copy link
Collaborator

@davideicardi davideicardi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@ekoutanov Do you have any comment on this?

@davideicardi davideicardi merged commit 62f5fb0 into obsidiandynamics:master Jun 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants