When building an App with Hybrid aot, the app crashes on startup with

[gabriel-kozma]

Android application type

Classic Xamarin.Android (MonoAndroid12.0, etc.)

Affected platform version

Xamarin Android 12.2, 12.3 and 13.0

Description

After compiling my app with hybrid aot mode and llvm turned on, the app crashes on startup with the error

Nested exception detected.
Original Exception: at Android.Runtime.JNIEnvInit.Initialize (Android.Runtime.JNIEnvInit/JnienvInitializeArgs*) [0x000a3] in <56b2ba5e3196483aa45d2f5001dd2d37>:0

my release configuration is

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
		<DebugSymbols>false</DebugSymbols>
		<Optimize>true</Optimize>
		<OutputPath>bin\Release</OutputPath>
		<ErrorReport>prompt</ErrorReport>
		<WarningLevel>4</WarningLevel>
		<AndroidDexTool>d8</AndroidDexTool>
		<AndroidUseSharedRuntime>false</AndroidUseSharedRuntime>
		<AndroidManagedSymbols>true</AndroidManagedSymbols>
		<AndroidSupportedAbis>arm64-v8a</AndroidSupportedAbis>
		<JavaMaximumHeapSize>1G</JavaMaximumHeapSize>
		<AotAssemblies>true</AotAssemblies>
		<AndroidAotMode>Hybrid</AndroidAotMode>
		<AndroidAotAdditionalArguments>hybrid,no-write-symbols,nodebug</AndroidAotAdditionalArguments>
		<EnableLLVM>true</EnableLLVM>
		<BundleAssemblies>true</BundleAssemblies>
		<AndroidKeyStore>True</AndroidKeyStore>
		<AndroidEnableSGenConcurrent>true</AndroidEnableSGenConcurrent>
		<AndroidLinkTool>r8</AndroidLinkTool>
		<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
		<AndroidPackageFormat>apk</AndroidPackageFormat>
		<AndroidCreatePackagePerAbi>false</AndroidCreatePackagePerAbi>
		<MandroidI18n />
		<AndroidLinkMode>SdkOnly</AndroidLinkMode>
		<EmbedAssembliesIntoApk>True</EmbedAssembliesIntoApk>
		<AndroidR8ExtraArguments>--no-tree-shaking</AndroidR8ExtraArguments>
		<AndroidAapt2LinkExtraArgs>--no-version-vectors</AndroidAapt2LinkExtraArgs>
	</PropertyGroup>

Steps to Reproduce

Archive app to release
Install on device
Try to run

Did you find any workaround?

Reverting to Xamarin.Android 12.1

Relevant log output

No response

[grendello]

@gabriel-kozma I will need to see the full stack trace of the crash. Could you please to record it using the following steps:

$ adb shell setprop debug.mono.log default,assembly,timing=bare
$ adb logcat -G 16M
$ adb logcat -c
# Start the application here, after it crashes wait a couple of seconds and then
$ adb logcat -d > log.txt

Please attach the resulting log.txt file here, thanks.

[ghost]

Hi @gabriel-kozma. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[gabriel-kozma]

hi @grendello here's the log file, I've swapped the package name to com.packagename and the dll name to AppName for privacy reasons

log.txt

[grendello]

@gabriel-kozma thanks! I'm afraid I'll need to bother you for another log, alas :( Please change the first command in the instructions given above to:

$ adb shell setprop debug.mono.log default,assembly,mono_log_level=debug,mono_log_mask=all

This will show us more internal MonoVM log messages, as the log you provided doesn't unfortunately shed much light on the problem. Thanks!

[ghost]

Hi @gabriel-kozma. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[gabriel-kozma]

hi @grendello here's the new log file! hope it helps
log2.txt

[grendello]

@gabriel-kozma thanks!

@lambdageek so I see only this in the log:

03-01 11:06:26.843 22485 22485 D monodroid: Calling into managed runtime init
03-01 11:06:26.843 22485 22485 D Mono    : Requesting loading reference 2 (of 7) of Mono.Android.dll
03-01 11:06:26.843 22485 22485 D Mono    : Loading reference 2 of Mono.Android.dll asmctx DEFAULT, looking for System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
03-01 11:06:26.843 22485 22485 V Mono    : Request to load System in (domain 0x7992d24b50, alc 0x0)
03-01 11:06:26.843 22485 22485 V Mono    : Found assembly remapping for System and was for the same version 2.0.5.0
03-01 11:06:26.843 22485 22485 V Mono    : Found assembly remapping for System and was for the same version 2.0.5.0
03-01 11:06:26.843 22485 22485 D Mono    : Assembly Ref addref Mono.Android[0x7912d41170] -> System[0x7912d42880]: 2
03-01 11:06:26.844 22485 22485 D Mono    : Requesting loading reference 0 (of 7) of System.dll
03-01 11:06:26.844 22485 22485 D Mono    : Loading reference 0 of System.dll asmctx DEFAULT, looking for mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
03-01 11:06:26.844 22485 22485 V Mono    : Request to load mscorlib in (domain 0x7992d24b50, alc 0x0)
03-01 11:06:26.844 22485 22485 V Mono    : Found assembly remapping for mscorlib and was for the same version 2.0.5.0
03-01 11:06:26.844 22485 22485 V Mono    : Found assembly remapping for mscorlib and was for the same version 2.0.5.0
03-01 11:06:26.844 22485 22485 D Mono    : Assembly Ref addref System[0x7912d42880] -> mscorlib[0x7912d3f940]: 4
03-01 11:06:26.846 22485 22485 E mono    : Unhandled Exception:
03-01 11:06:26.846 22485 22485 E mono    : Nested exception detected.
03-01 11:06:26.846 22485 22485 E mono    : Original Exception: at Android.Runtime.JNIEnv.Initialize (Android.Runtime.JnienvInitializeArgs*) [0x0011c] in <c9e5d28c5371465c822261af707386f3>:0
03-01 11:06:26.846 22485 22485 E mono    : 
03-01 11:06:26.846 22485 22485 E mono    : Nested exception:managed backtrace not available
03-01 11:06:26.846 22485 22485 E mono    : 
03-01 11:06:26.846 22485 22485 E mono    : 
03-01 11:06:26.846 22485 22485 E mono-rt : [ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong

Android.Runtime.JNIEnv.Initialize is the very first call we make to the managed land, does anything above suggest anything to you? The method doesn't explicitly throw any exceptions, so it might be anything in there :(

[grendello]

@gabriel-kozma would you be able to come up with a small application that reproduces the issue? I'm afraid investigating this may require us to be able to reproduce it locally.

[gabriel-kozma]

@grendello at the moment, I can't, I spent the whole day tomorrow investigating this, but I can confirm rolling back to Xamarin.Android 12.1 fixed all my issues.

For me to create this app, I would need to go back to Xamarin.Android 13 and I cannot mess with my development flow right now

But I believe, by just enabling hybrid and llvm on 13.0 apps, would render this issue

[gabriel-kozma]

Hi there, any updates on this?

[grendello]

@gabriel-kozma I will need the repro app I mentioned before, to look into this.

[ghost]

Hi @gabriel-kozma. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[gabriel-kozma]

hey @grendello I've created the following project:
https://github.com/gabriel-kozma/HybridAotErrorSample

I just created a basic xamarin project from visual studio for mac and added the release properties, it crashes with the same error, should be good for you guys to test it

[gabriel-kozma]

here's the apk that produces the issue as well
com.companyname.hybridaotsample.apk.zip

[gabriel-kozma]

@grendello does this help?

[gabriel-kozma]

any updates on this?

[grendello]

@gabriel-kozma alas, not yet. I'll be looking into this issue this week, hopefully finding a clue or two

[jonpryor]

@gabriel-kozma: please keep in mind that Xamarin.Android end of support is on 2024-May-1, which is less than a year away. .NET Android on .NET 7 and later is getting most of our development resources.

Please migrate your app to .NET Android and let us know if HybridAOT works better for you there.

[gabriel-kozma]

@gabriel-kozma alas, not yet. I'll be looking into this issue this week, hopefully finding a clue or two

@grendello sounds good, thank you!

@gabriel-kozma: please keep in mind that Xamarin.Android end of support is on 2024-May-1, which is less than a year away. .NET Android on .NET 7 and later is getting most of our development resources.

Please migrate your app to .NET Android and let us know if HybridAOT works better for you there.

Hi @jonpryor, yeah, I intended to migrate as soon as possible, but HybridAOT is not working on .NET Android, there's a feature request open for 2.5 years and this has been preventing me and others to migrate

dotnet/runtime#44855

[jxbrenner]

@gabriel-kozma: please keep in mind that Xamarin.Android end of support is on 2024-May-1, which is less than a year away. .NET Android on .NET 7 and later is getting most of our development resources.

Please migrate your app to .NET Android and let us know if HybridAOT works better for you there.

@jonpryor You make a very good point and one that I hope Microsoft takes seriously. Much of my livelihood comes from an enterprise Xamarin app I spent years developing. With Android 13 and iOS 16 being the final platform versions supported by Xamarin, the clock is ticking on all of my apps being dropped from the Apple and Google app stores. AoT compilation and CIL stripping are broken in .NET Android and these features are requirements for security, performance, and protection of IP. The pitch from Microsoft is upgrade my Xamarin app .NET 6+/MAUI without a rewrite, but by breaking these features there is no upgrade path.

I've had dotnet/runtime#44855 open since the development of .NET 6. The request appears to have Microsoft's commitment but was pushed from .NET 6 to .NET 7, and then from .NET 7 to .NET 8. I hope that its clear that if the regressions in hybrid AoT and CIL stripping aren't resolved for .NET 8, every developer who relies on these features will see their apps dropped from the app stores due to support ending on Xamarin.

[jxbrenner]

@gabriel-kozma Does setting the build property <AndroidAotAdditionalArguments>hybrid</AndroidAotAdditionalArguments> help? This was the fix for me in #7088. Seems <AndroidAotMode>Hybrid</AndroidAotMode> no longer sets the Mono compiler to hybrid AoT after Xamarin.Android 12.2.

[gabriel-kozma]

hi @jxbrenner it doesn't it strips and everything, but the app "crashes" at runtime, it doesn't actually crash, but doesn't go past the splash screen, take a look at the sample apk I provided and the example project

[jonpryor]

@jxbrenner: I understand your concerns, and unfortunately I can't fix everything. :-(

I also believe that Hybrid AOT was only ever called "Experimental", and also that Visual Studio no longer allows you to even select Hybrid as an AOT mode, even in Classic Xamarin.Android. I don't believe we ever considered it to be stable enough to remove the "experimental" moniker.

(And in a "silly in retrospect" bit of oversight, the Hybrid AOT tests we do have don't run on-device. 😅)

Regarding "protection of IP", I hope that with the introduction of Assembly Stores (the default in .NET 7?) -- which compresses assemblies and stores all assemblies in a fairly inscrutable .blob file instead of separate .dll files -- extracting the assemblies will be (nominally) more difficult than in previous releases. They're not encrypted, but they are harder to obtain.

[jxbrenner]

@jonpryor Of course, thank you for all the things you have and continue to fix.

I am aware of assembly stores, but regarding protection of IP, unfortunately assemblies.blob can be unpacked in the latest version of .NET Reactor.

I've always wondered about this point about hybrid AoT being an "experimental build property" in VS. Disclaimer, please forgive my ignorance and please correct me where I'm wrong. As I understand it, full AoT and mono-cil-strip were a part of Mono since 2008, and hybrid AoT was added in 2016. And doesn't hybrid AoT just JIT the Android bindings (Mono.Android) and statically compile everything else? So didn't the ability to statically compile Android apps and strip the IL come as working features come with Mono when it was acquired by Microsoft? So if the Mono compiler and its toolchain have all that functionality, what does hybrid AoT being an experimental build property even mean? That Microsoft didn't want to take responsibility for ensuring that the Xamarin.Android SDK, less the bindings, could be statically compiled without crashing? I'm not trying to find fault or attribute blame, just trying to understand where it all breaks down.

All this may be irrelevant though, as it sounds like the feature request to fully support hybrid AoT and IL stripping for .NET Android are on track to be delivered with .NET 8 (dotnet/runtime#44855).

[jonpryor]

A bit of history :-)

"In the beginning" (.NET Framework 1.0) there was NGEN.EXE, which would Ahead-of-Time compile an assembly (with various tradeoffs). NGEN output still required the JIT, and had a variety of tradeoffs (NGEN output was larger, and "cold start" was slower than the JIT, etc.)

MonoTouch in 2009 required "Full" AOT -- AOT everything -- in order to work, as iOS did not support a JIT at all. This required a fair bit of work to get Mono to support Full AOT. However, the way it was implemented was as an "either/or": you had Full AOT and no JIT, or JIT and no Full AOT. (Mono also had "normal" AOT as well, which was a different file format than Full AOT.)

Android supported a JIT, so Mono for Android didn't use Full AOT, and instead relied on a JIT, and in fact still requires a JIT for various constructs such as constructor invocation. (When Java instantiates a C# type, we use System.Reflection.Emit to create an "intermediate" constructor which sets Java.Lang.Object.handle before the C# constructor executes. This required a JIT, before the introduction of the IL interpreter, which {mostly} isn't used on Android.)

"Hybrid" AOT was a way to use the "Full" AOT file format + runtime changes and also use a JIT. This was the only way to use Full AOT on Android, short of significant architectural changes to Xamarin.Android to remove the need for a JIT (which still haven't happened, and aren't planned to happen).

Full and Hybrid AOT work across assembly boundaries, requiring that the entire app be known (so you can't download assemblies at runtime and have them work w/o a JIT).

To answer your questions:

doesn't hybrid AoT just JIT the Android bindings (Mono.Android) and statically compile everything else?

Not quite. Hybrid AOT AOTs all assemblies in the app, while allowing the JIT to be used for constructs which require the JIT, which are spread across all assemblies.

didn't the ability to statically compile Android apps and strip the IL come as working features come with Mono when it was acquired by Microsoft?

.NET Android née Xamarin.Android née Mono for Android use Mono(VM), and have always used Mono, so this question doesn't quite make sense. The Microsoft acquisition doesn't have a lot to do with Hybrid AOT or the way it works; the same wonderful Mono runtime people were involved both before and after acquisition. (There is also very large overlap before and after acquisition on the Android team.) "Static compilation", while a feature of Mono, had tradeoffs that made it not quite right for use on Android. This is why the Mono team needed to create the "Hybrid" variant for Android, with a different set of tradeoffs.

While Hybrid AOT shared many things with Full AOT, it also involved new code, and wasn't widely tested. When we did begin testing, we found various problems, which in turn required that we add various checks such as the XA1025 error (which is likely moot now, but wasn't moot then). This is why it shipped as "Experimental": it works (-ish) for us (in some contexts), but our testing can only go so far. We wanted wider testing to know that it wouldn't break everybody in unexpected ways.

Hybrid AOT also had other "limitations"/requirements, such as that an Android NDK was required to use it (large download) and you had to be a Visual Studio Enterprise customer to even try it. This further reduced the amount of possible external testing that could occur.

Which is where things largely stagnated: our priorities weren't on Hybrid AOT (fewer customers could use it), and we focused on Profiled AOT and reducing startup times and other things which would help more of our customers, while navigating the ".NET Transition" (engineering time "spent" on migrating away from Mono to .NET MonoVM, "fixing" our build system so that it could work under dotnet build, etc., etc.) and later MAUI.

The benefit of dotnet/runtime#44855 is that it should be available to everyone, thus avoiding the "no widespread testing" problem, and if it lands early enough, it might be something worth enabling by default. (Maybe. No promises.)

[jxbrenner]

@jonpryor Thanks for the history and the insight! I didn't realize hybrid AOT was restricted to VS Enterprise. I'm glad to hear dotnet/runtime#44855 will provide AOT and IL stripping for Android to everyone.

[gabriel-kozma]

Hi there @jonpryor and @grendello do we have any workaround for this?

Google will start not allowing upload apps not targeting android 13 (api 33) and because of this issue, I'm stuck with api 31