94.3.0 release has a linkage problem in some devices

[tarikeshaq]

https://sentry.io/organizations/mozilla/issues/3527488899/?query=is%3Aunresolved+load+library&statsPeriod=14d

Unable to load library 'megazord': dlopen failed: cannot locate symbol "__extenddftf2" referenced by
Native library (android-x86-64/libmegazord.so) not found in resource path (.)

We are suspicious it was the NDK upgrade in #5142 but we are not certain since we can't reproduce locally yet

┆Issue is synchronized with this Jira Task
┆Epic: Prioritized from backlog
┆Sprint End Date: 2023-03-24

[cpeterso]

Here is the Bugzilla bug tracking these crash reports in Socorro: https://bugzilla.mozilla.org/show_bug.cgi?id=1792799

[rvandermeulen]

Thanks for that, Chris. Socorro definitely confirms that the revert to 94.2.1 resolved these crashes for now at least.

[rvandermeulen]

Maybe related:
android/ndk#1614

[rvandermeulen]

Looking at https://developer.android.com/ndk/guides/common-problems, I also wonder if Gecko's still being on r21 is possibly relevant.

[jonalmeida]

Looking at https://developer.android.com/ndk/guides/common-problems, I also wonder if Gecko's still being on r21 is possibly relevant.

It shouldn't. The crash is not happening when there is any interaction between Gecko and app services code.

[bendk]

I was able to reproduce this one on an emulator and confirmed that backing out the NDK upgrade fixed my issue there (details).

I'm going to make a 94.3.2 release today, what should the next steps be after that? I'm thinking we should maybe wait until the next release cycle then introduce it back into nightly?

[bendk]

I think we're running into something like mozilla/rust-android-gradle#105.

I see this error in logcat: AndroidRuntime: Native library (android-x86-64/libmegazord.so) not found in resource path (.) However, libmegazord.so is definitely in the APK. I see it in the lib/x86_64/ directory when I unzip it.

@ncalexan any ideas on how we can fix this

Even if we don't fix it, maybe it's not that big of an issue. The above discussion seems to indicate that it's only an issue for emulators and the sentry results also seem to indicate that. The main affected device was octupus, there was also a grunt, and a 2121D6U (which seems to be a lenovo mini desktop), there was a single event for XT1710-02 which I think is the name of a real device, and several events for HP Chromebook x360 11 G1 EE. Maybe some of those are real devices, but it's very possible that they're all emulators.

[ncalexan]

@ncalexan any ideas on how we can fix this

Not really; I never ran this all the way down in the r-a-g plugin issue. Rereading my comments, I don't think we ever established that we really were seeing symbol visibility issues/changes due to the NDK version, but that should be relatively simple to verify. (Start with my example commands from the linked ticket and see if visibility changes with and without your NDK version changes.)

Certainly I don't have any simple workarounds to hand, sorry :(

[bendk]

I think I should have linked to mozilla/rust-android-gradle#75, that one is closer to our issue.

I see dlopen failed: cannot locate symbol "__extenddftf2" referenced by "/data/app/~~q72vdWTOdkCVifYXVmpoCQ==/org.mozilla.fenix.debug-tmT3R9CQhG_vNvU1QFQ1iQ==/lib/x86_64/libmegazord.so in logcat and I think I finally understand what that's trying to tell me. It can't open the .so file because that file depends on the __extenddftf2 symbol, which can't be found.

When I run llvm-objdump I see lots of symbols, including __extenddftf2. __extenddftf2 is listed as undefined.

mozilla/rust-android-gradle#83 was supposed to fix #75, but I'm still seeing the issue. Do I need to do something to trigger the fix? Setting CARGO_NDK_MAJOR_VERSION=25 before building Fenix didn't do the trick. I tried setting cargo.apiLevel = 25 in our build.gradle file, but that caused other issues.

[bendk]

We're way past the 94.3.0 release at this point, let's use a new issue to track this work: #5436