java.lang.UnsatisfiedLinkError: org.apache.tomcat.jni.Library.version(I)I

[shelined]

I'm attempting to use grpc-java with netty-tcnative-boringssl-static and getting the following exception.

DEBUG io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.
java.lang.UnsatisfiedLinkError: org.apache.tomcat.jni.Library.version(I)I
    at org.apache.tomcat.jni.Library.version(Native Method) ~[netty-tcnative-boringssl-static-1.1.33.Fork14-linux-x86_64.jar:1.1.33.Fork14]
    at org.apache.tomcat.jni.Library.initialize(Library.java:176) ~[netty-tcnative-boringssl-static-1.1.33.Fork14-linux-x86_64.jar:1.1.33.Fork14]
    at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:61) ~[netty-handler-4.1.0.CR3.jar:4.1.0.CR3]
    at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:135) [grpc-netty-0.13.2.jar:0.13.2]
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:117) [grpc-netty-0.13.2.jar:0.13.2]
    at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:88) [grpc-netty-0.13.2.jar:0.13.2]

I understand it might complicate things but I'm running inside Tomcat for the time being. I have set the -classpath to the path of netty-tcnative-boringssl-static-1.1.33.Fork14-linux-x86_64.jar as you can see below in the command to execute Tomcat.

/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djuli-logback.configurationFile=file:/usr/local/tomcat/conf/logback.xml? -Djava.endorsed.dirs=/usr/local/tomcat/endorsed -classpath /usr/local/tomcat/webapps/ROOT/WEB-INF/lib/netty-tcnative-boringssl-static-1.1.33.Fork14-linux-x86_64.jar:/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start

I've tried both statically and dynamically linked and keep running across the same exception.

grpc-java = 0.13.2
netty-tcnative-boringssl-static = 1.1.33.Fork14 on linux-x86_64 (also tried dynamic and Fork15)

$ java -version
openjdk version "1.8.0_72-internal"
OpenJDK Runtime Environment (build 1.8.0_72-internal-b15)
OpenJDK 64-Bit Server VM (build 25.72-b15, mixed mode)

$ uname -a     
Linux <docker container name> 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4 (2016-02-29) x86_64 GNU/Linux

CCing @nmittler and @normanmaurer per @nmittler

[nmittler]

Hmm ... so grpc-java 0.13.2 uses Netty 4.1.0.CR3. I think the issue is that 4.1.0.CR3 doesn't have the logic to look for the lib with a classifier (4.1.0.CR4 has the new logic). Try it with Netty 4.1.0.CR4 and see if it works for you - hopefully there weren't any breaking API changes.

FYI, grpc-java should be releasing soon. Once the new release is available, upgrading should do the trick.

[shelined]

I could be wrong but it seems like the new logic adds capabilities needed for the Uber jar where the native libraries have the classifier in the name. Regardless I did actually switch to CR4 and paid closer attention when stepping through the code in the debugger (albeit on osx-x86_64 for ease of debugging).

When NativeLibraryLoader.load(...) is invoked with first netty-tcnative-osx-x86_64 the following is logged, which makes sense since inside netty-tcnative-boringssl-static-1.1.33.Fork15-osx-x86_64.jar the library is named libnetty-tcnative.jnilib.

NativeLibraryLoader - Unable to load the library: netty-tcnative-osx-x86_64.
java.lang.UnsatisfiedLinkError: no netty-tcnative-osx-x86_64 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) ~[na:1.8.0_77]
    at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[na:1.8.0_77]
    at java.lang.System.loadLibrary(System.java:1122) ~[na:1.8.0_77]
    at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:189) ~[netty-common-4.1.0.CR4.jar:4.1.0.CR4]
    at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:161) ~[netty-common-4.1.0.CR4.jar:4.1.0.CR4]
    at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:78) [netty-handler-4.1.0.CR4.jar:4.1.0.CR4]
    at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:135) [grpc-all-0.13.2.jar:0.13.2]
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:117) [grpc-all-0.13.2.jar:0.13.2]

The second invocation with netty-tcnative works property and copies and loads the library exiting NativeLibraryLoader.load(...) after invoking System.load(...) without exception.

However the UnsatisfiedLinkError is still thrown as originally described on both osx-x86_64 and linux-x86_64.

OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.
java.lang.UnsatisfiedLinkError: org.apache.tomcat.jni.Library.version(I)I
    at org.apache.tomcat.jni.Library.version(Native Method) ~[netty-tcnative-boringssl-static-1.1.33.Fork15-osx-x86_64.jar:1.1.33.Fork15]
    at org.apache.tomcat.jni.Library.initialize(Library.java:176) ~[netty-tcnative-boringssl-static-1.1.33.Fork15-osx-x86_64.jar:1.1.33.Fork15]
    at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:81) ~[netty-handler-4.1.0.CR4.jar:4.1.0.CR4]
    at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:135) [grpc-all-0.13.2.jar:0.13.2]
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:117) [grpc-all-0.13.2.jar:0.13.2]
    at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:88) [grpc-all-0.13.2.jar:0.13.2]

[nmittler]

Ah you're right, my mistake. Can replicate the problem outside of tomcat?

[jjpp]

It seems that I have the same/similar problem -- netty-tcnative-boringssl-static in tomcat (tomcat8, openjdk8 under centos7) that fails with UnsatisfiedLinkError about Library.version(int). I need it to use HTTP/2 over TLS. I have removed tomcat-jni.jar from tomcat libraries to avoid conflict there.

So far I have found an extra hint: if I copy the temporary file that NativeLibraryLoader creates and loads with System.load to eg /usr/lib/libnetty-tcnative.so and try to start tomcat then (so System.loadLibrary succeeds), everything works. So it kind of seems that Runtime.loadLibrary and Runtime.load do something differently. JNI specification was updated for JDK8, but it should be compatible with old usage as far as i understand.

[jjpp]

Further experiments show that it seems to be related to hierarchy of classloaders and to the fact that some (er.. most)of the classes in tcnative are found in tomcat-jni.jar and are loaded during tomcat startup. Replacing tomcat-jni.jar with version that contains just org.apache.tomcat.jni.LibraryNotFoundError "helps".

I believe that using some netty-tcnative-specific class instead of org.apache.tomcat.jni.SSL in io.netty.handler.ssl.OpenSsl to find correct classloader might help.

[normanmaurer]

Can You maybe try to submit a pr?

Am 04.05.2016 um 17:26 schrieb Jaak Pruulmann-Vengerfeldt [email protected]:

Further experiments show that it seems to be related to hierarchy of classloaders and to the fact that some (er.. most)of the classes in tcnative are found in tomcat-jni.jar and are loaded during tomcat startup. Replacing tomcat-jni.jar with version that contains just org.apache.tomcat.jni.LibraryNotFoundError "helps".

I believe that using some netty-tcnative-specific class instead of org.apache.tomcat.jni.SSL in io.netty.handler.ssl.OpenSsl to find correct classloader might help.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[jjpp]

I tried and found out that it is not that simple. Tomcat runs each web app in its own classloader (http://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html) which mostly prefers classes from the app. With notable exception of "system classes" and "servlet apis". Which includes among other things org.apache.tomcat.* .. so as long there is org.apache.tomcat.jni.Library in classpath of "common" classloader of Tomcat, it will be used from there.

Also, JVM needs that the native library is loaded by the same classloader as is the class whose methods are implemented. And library is "loaded by" the same classloader that loaded the class which called System.load/System.loadLibrary (or Runtime-equivalents of those). That is, the classloader that loads NativeLibraryLoader ..

So, one needs to make sure that org.apache.tomcat.jni.Library (and other relevant classes) are loaded by the same classloader as NativeLibraryLoader.

The options of getting netty-tcnative work properly in tomcat seem to be:

• remove tomcat-jni.jar from $CATALINA_HOME/lib: cd $CATALINA_HOME/lib; unzip tomcat-jni.jar org/apache/tomcat/jni/LibraryNotFoundError.class; mv tomcat-jni.jar{,.off} . (org.apache.tomcat.jni.LibraryNotFoundError is required to start tomcat). Application has to (and can!) provide correct versions of everything, possibly with built in libraries, it should be possible to upgrade library version just by redeploy.
• replace tomcat-jni.jar in $CATALINA_HOME/lib with relevant parts of netty. One must make sure that they are not loaded from the app (as these classes are not special and version in war is preferred). This can be done by removing those jars from war or by setting delegate flag of tomcat's WebappClassLoader to true (making it always prefer common classloader).
• rename package org.apache.tomcat.jni in netty-tcnative to something else. Perhaps as a flavor or something?
• implement a classloader, similar to tomcat's, that prefers local resources and does not treat org.apache.tomcat.* specially. Run everything important through that loader. I have not tried and I do not know if there would be any problems with that. Though, creating new classloaders seems always like asking for trouble :)

Only package renaming might be in the scope of this project but this seems quite unlikely?

[TimYi]

Seems only package renaming will work.

[jchambers]

In case the data is helpful for assessing impact or assigning priority, it seems like Pushy users run into this problem pretty regularly.

As an aside, we actually ran into this problem ourselves; my first thought was to create an external RPC service so Pushy doesn't have to run inside a Tomcat container, but my first choice for an RPC framework also suffers from this problem. The irony! ;)

[xsir]

@shelined I encounter with the same problem is OSGi. Finally I resolved it and submitted a PR #5545 to netty project. I wish it cat do some help, best regards.

[shelined]

@xsir I just took a look at your PR but I am not as familiar with the loading of native libraries. Did you fix both the OSGi and non-OSGi case?

[normanmaurer]

This is already fixed in the master branch as we renamed the package etc.