Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(docker): update mae and mce consumer images to include glibc compat #1899

Conversation

grantatspothero
Copy link
Contributor

@grantatspothero grantatspothero commented Sep 28, 2020

Update mae and mce consumer images to include glibc compat layer. This allows the consumer jobs to deal with snappy compressed kafka topics when running on alpine linux. Otherwise you get errors like this:

java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-bb847a5e-21b5-4d9b-babd-f31afc7109a7-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/snappy-1.1.7-bb847a5e-21b5-4d9b-babd-f31afc7109a7-libsnappyjava.so)
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable). Not applicable.
  • Tests for the changes have been added/updated (if applicable). Not applicable.
  • Docs related to the changes have been added/updated (if applicable). Not applicable.

An alternative approach is upgrading the snappy-java client library to a higher version that supports a purely java implementation of snappy so there is no dependence on native libraries. See here for more details:
xerial/snappy-java#232 (comment)
https://github.com/xerial/snappy-java#using-pure-java-snappy-implementation

That is probably the better option, but I was concerned about some potential backwards incompatible changes in the snappy-java library since it is transitively included by the avro dependency. Wasn't sure of a great way to test it.

…at layer. allows the consumer jobs to deal with snappy compressed kafka topics when running on alpine linux
Copy link
Contributor

@mars-lan mars-lan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. Will merge this in once the CI passes.

@mars-lan mars-lan merged commit 9bcf273 into datahub-project:master Sep 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants