Unable to rebuild project with hybrid AOT enabled #1240

pjcollins · 2018-01-25T22:14:49Z

Steps to Reproduce

Attempt to compile the Codegen test suite with AOT, LLVM, and Hybrid AOT (/p:AndroidAotMode=Hybrid) enabled.

Expected Behavior

I'm able to compile and run any XA project with Hybrid AOT enabled.

Actual Behavior

Notice multiple errors coming from the aot task informing that methods can not be compiled successfully:

[aot-compiler stderr] Unable to compile method 'bool System.Collections.Specialized.ListDictionary/NodeKeyValueCollection/NodeKeyValueEnumerator:MoveNext ()' due to: 'Invalid IL code in System.Collections.Specialized.ListDictionary/NodeKeyValueCollection/NodeKeyValueEnumerator:MoveNext (): IL_0000: ret (TaskId:1362)

The build ultimately fails with error XA3001: System.IO.IOException: Win32 IO returned ERROR_ALREADY_EXISTS. during Xamarin.Android.Tasks.CilStrip.Execute.

Version Information

https://gist.github.com/pjcollins/7d20f81013c7e00982c91340af2cf560

Log File

http://xqa.blob.core.windows.net/gist/log-63b871d9098d441d80a2838bb9fa2860.txt

VS bug #947891

The text was updated successfully, but these errors were encountered:

jonpryor · 2018-01-26T04:05:00Z

It looks like this error is happening before we IL-strip the assemblies, so it doesn't appear to be a build task ordering issue.

vargaz · 2018-01-26T05:12:14Z

Could you upload the tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/obj/Release directory ?

pjcollins · 2018-01-26T13:55:44Z

@vargaz here's the intermediate output - https://microsoft-my.sharepoint.com/:u:/p/pecolli/EZcB-AYr2z5NlFocWnr_u1MBsJcXYNImuhgF8CSzqIsaVQ?e=ypELnj

I'll try to remember to include this in the report for such issues in the future.

vargaz · 2018-01-26T15:54:57Z

Are you sure that is the correct obj dir ? I get a mscorlib version mismatch error:

Corlib not in sync with this runtime: expected corlib version 1050800000, found 1050400003.
Loaded from: /Users/vargaz/Projects/xamarin-android-1240/jcwgen-obj/Release/android/assets/shrunk/mscorlib.dll

1050800000 is for mono 2.10, while 10504 is for an older version.

vargaz · 2018-01-26T15:58:47Z

The invalid IL messages all reference 'ret', which means the AOT compiler is ran on the IL stripped code. Also, the aot compiler command line references the 'shrunk' directory which seems to contain the IL stripped versions:

MONO_PATH="/Users/xamarinqa/myagent/_work/r3/a/monodroid/external/xamarin-android/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/obj/Release/android/assets/shrunk" MONO_ENV_OPTIONS="" /Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/Darwin/cross-arm --llvm --aot=hybrid,outfile=/Users/xamarinqa/myagent/_work/r3/a/monodroid/external/xamarin-android/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/obj/Release/aot/armeabi/libaot-Java.Interop.dll.so,asmwriter,mtriple=armv5-linux-gnueabi,tool-prefix=/Users/xamarinqa/Library/Developer/Xamarin/android-sdk-macosx/ndk-bundle/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-,ld-flags=/Users/xamarinqa/Library/Developer/Xamarin/android-sdk-macosx/ndk-bundle/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/libgcc.a;/Users/xamarinqa/Library/Developer/Xamarin/android-sdk-macosx/ndk-bundle/platforms/android-9/arch-arm/usr/lib/libc.so;/Users/xamarinqa/Library/Developer/Xamarin/android-sdk-macosx/ndk-bundle/platforms/android-9/arch-arm/usr/lib/libm.so,llvm-path=/Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/Darwin,temp-path=/Users/xamarinqa/myagent/_work/r3/a/monodroid/external/xamarin-android/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/obj/Release/aot/armeabi/Java.Interop.dll /Users/xamarinqa/myagent/_work/r3/a/monodroid/external/xamarin-android/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/obj/Release/android/assets/shrunk/Java.Interop.dll (TaskId:1362)

pjcollins · 2018-01-26T16:20:53Z

@vargaz The local repro uploaded in the previous comment is from d15-5, the issue is reproducible both there and in d15-6. Should I upload another obj dir built against d15-6 or master?

vargaz · 2018-01-26T16:37:21Z

Not needed any more, I think
#1240 (comment)
is the problem.

jonpryor · 2018-01-26T20:56:40Z

@vargaz: I don't understand how it's the problem. The <CilStrip/> task is executed on line 317911.

AOT compilation ends on line 317765 -- 146 lines before the IL strip.

How are we attempting to AOT stripped assemblies when we don't execute the task to strip assemblies until after AOT runs?

A plausible answer would be "the project is built multiple times, with the second build processing the stripped outputs of the first build." However, I see no evidence of that scenario in the log file.

vargaz · 2018-01-26T21:00:49Z

Does the 'shrunk' directory contain linker output files ? It could be a problem with the linker, i.e. it creates invalid il ?

vargaz · 2018-01-26T21:03:12Z

The input file contains stuff like this:

.method public hidebysig static string
GetText(string msg) cil managed noinlining
{
// Code size 1 (0x1)
.maxstack 8
IL_0000: ret
} // end of method Locale::GetText

which is not valid IL.

jonpryor · 2018-01-26T21:54:01Z

Does the 'shrunk' directory contain linker output files?

The _RemoveRegisterAttribute target copies the obj/Release/android/assets files into obj/Release/android/assets/shrunk, then updates the assemblies within obj/Release/android/assets/shrunk.

It could be a problem with the linker, i.e. it creates invalid il ?

Possible, but what's really weird is that the linker doesn't appear to be running: the LinkAssemblies task isn't executed, and the _LinkAssembliesShrink target finishes in 1ms.

@radekdoulik: Do you know why the assemblies used as input to AOT would appear to be invalid?

pjcollins · 2018-05-21T18:33:09Z

This is still reproducible using XA d15.8 - 8.3.99.12 (monodroid/de662f58d).

http://xqa.blob.core.windows.net/gist/log-8294c709d0bc4d099ae5273bb88bf162.txt

jonpryor · 2018-06-05T23:12:07Z

@radekdoulik: Please take another look at this.

radekdoulik · 2018-06-18T10:26:19Z

I wasn't able to reproduce with XA master and d15.8 branch. Will try the monodroid/de662f58d

radekdoulik · 2018-06-18T13:31:10Z

OK, I am now able to reproduce with master.

The steps are following:

build in Release configuration without AOT
build in Release configuration with hybrid AOT

It looks different to the original problem where aot-compiler reported errors.

In my case I don't see aot-compiler errors, only CilStrip problem at the end of the log mentioned in #1240 (comment)

From my build log:

/Users/rodo/git/xa5/bin/Debug/lib/xamarin.android/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(2660,4): error XA3001: System.IO.IOException: Win32 IO returned ERROR_ALREADY_EXISTS. Path:  [/Users/rodo/git/xa5/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj]
/Users/rodo/git/xa5/bin/Debug/lib/xamarin.android/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(2660,4): error XA3001:   at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00102] in <7e59c4ad7b424f9eab0db1b78ec1804f>:0  [/Users/rodo/git/xa5/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj]
/Users/rodo/git/xa5/bin/Debug/lib/xamarin.android/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(2660,4): error XA3001:   at Xamarin.Android.Tasks.CilStrip.DoExecute () [0x000ca] in <71e70fc60389471c98c06e92be3df0b7>:0  [/Users/rodo/git/xa5/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj]
/Users/rodo/git/xa5/bin/Debug/lib/xamarin.android/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(2660,4): error XA3001:   at Xamarin.Android.Tasks.CilStrip.Execute () [0x00002] in <71e70fc60389471c98c06e92be3df0b7>:0  [/Users/rodo/git/xa5/tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj]
Done executing task "CilStrip" -- FAILED. (TaskId:603)

radekdoulik · 2018-06-18T13:45:56Z

Better repro, with aot-compiler errors as well:

build in Release configuration with hybrid AOT
build in Release configuration with hybrid AOT again

Looks like build process error to me, where we try to AOT compile the already stripped assemblies in the second build. Linker doesn't even run as the output files are newer than input files.

@dellis1972 could you take a look at this please?

This is my command line to build the test with hybrid AOT
./tools/scripts/xabuild /v:diag /p:AotAssemblies=True /p:AndroidAotMode=Hybrid /p:Configuration=Release /t:SignAndroidPackage tests/CodeGen-Binding/Xamarin.Android.Jcwn.Android.JcwGen-Tests.csproj

dellis1972 · 2019-07-16T09:50:10Z

ok I think there are a couple of problems here. The main on is that the Aot task is running when it really shouldn't be. No changes have been made. The logs reviled the following on many of the binding projects.

 Input file "/var/folders/n_/vt_nzq1n04schx5yy2jgcgcw0000gn/T/MonoAndroid,Version=v9.0.AssemblyAttributes.cs" is newer than output file "obj/Release/Xamarin.Android.McwGen-Tests.pdb".

This then results is CoreCompile running on the main app, because the binding project assemblies have been updated. That is only part of the problem. I still need to look at CilStrip and see what is going on there.

…tribute.cs Context dotnet#1240 As part of the build process MSBuild generates an `AssemblyAttribute.cs` class, such as MonoAndroid,Version=v9.0.AssemblyAttributes.cs This is the file which adds the `TargetFrameworkAttribute` to the assembly. While investigating dotnet#1240 I noticed that `_GenerateJavaStubs` was ALWAYS running, even on incremental builds with no changes. Digging into the build logs, `CoreCompile` was running because it thought one of the binding project assemblies was newer..? Digging in a bit more and I found this in the Binding Library build output on the `CoreCompile` target. Input file "/var/folders/n_/vt_nzq1n04schx5yy2jgcgcw0000gn/T/MonoAndroid,Version=v9.0.AssemblyAttributes.cs" is newer than output file "obj/Release/Xamarin.Android.McwGen-Tests.pdb". It turns out our binding targets were missing a few lines which exist in the common targets. These line set the `TargetFrameworkMonikerAssemblyAttributesPath` msbuild property to be located in the `$(IntermediateOutputPath)`. If this value is NOT set it turns out MSBuild defaults to dropping this file into a temp directory. That means it can be cleaned up at any point which will trigger a full build. :face_palm. So lets use the same few lines from the common targets to make sure we generate this file in our `$(IntermediateOutputPath)`. This stopped `_GenerateJavaStubs` from running when there were no changes (as expected). I updated one of our binding tests to check to make sure certain targets DO NOT run on an incremental build. Similar to what we do with our application targets.

…tribute.cs (#3354) Context #1240 As part of the build process MSBuild generates an `AssemblyAttribute.cs` class, such as MonoAndroid,Version=v9.0.AssemblyAttributes.cs This is the file which adds the `TargetFrameworkAttribute` to the assembly. While investigating #1240 I noticed that `_GenerateJavaStubs` was ALWAYS running, even on incremental builds with no changes. Digging into the build logs, `CoreCompile` was running because it thought one of the binding project assemblies was newer..? Digging in a bit more and I found this in the Binding Library build output on the `CoreCompile` target. Input file "/var/folders/n_/vt_nzq1n04schx5yy2jgcgcw0000gn/T/MonoAndroid,Version=v9.0.AssemblyAttributes.cs" is newer than output file "obj/Release/Xamarin.Android.McwGen-Tests.pdb". It turns out our binding targets were missing a few lines which exist in the common targets. These line set the `TargetFrameworkMonikerAssemblyAttributesPath` msbuild property to be located in the `$(IntermediateOutputPath)`. If this value is NOT set it turns out MSBuild defaults to dropping this file into a temp directory. That means it can be cleaned up at any point which will trigger a full build. :face_palm. So lets use the same few lines from the common targets to make sure we generate this file in our `$(IntermediateOutputPath)`. This stopped `_GenerateJavaStubs` from running when there were no changes (as expected). I updated one of our binding tests to check to make sure certain targets DO NOT run on an incremental build. Similar to what we do with our application targets.

Fixes dotnet#1240 The `CilStrp` task was not able to handle being called incrementally. When called a second time the resulting error would be raised. error XA3001: System.IO.IOException: The file '/Users/dean/Projects/Foo5F/Foo/obj/Release/android/assets/non-stripped/Foo.dll' already exists. at System.IO.FileSystem.LinkOrCopyFile (System.String sourceFullPath, System.String destFullPath) [0x000b9] in <06589f46f7f64859943a6a461f88c56e>:0 at System.IO.FileSystem.MoveFile (System.String sourceFullPath, System.String destFullPath) [0x0003a] in <06589f46f7f64859943a6a461f88c56e>:0 at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00083] in <06589f46f7f64859943a6a461f88c56e>:0 at Xamarin.Android.Tasks.CilStrip.DoExecute () [0x000a8] in <fd1a3c6c66de470d989dd70ca1f91267>:0 at Xamarin.Android.Tasks.CilStrip.Execute () [0x00000] in <fd1a3c6c66de470d989dd70ca1f91267>:0 This commit make sure we do not try to strip an assembly if it has already been stripped. It also adds a new Unit Test to make sure the build system behaves as we expect.

) Fixes: #1240 The `<CilStrip/>` task was not able to handle being called incrementally. When called a second time the resulting error would be raised. error XA3001: System.IO.IOException: The file '/Users/dean/Projects/Foo5F/Foo/obj/Release/android/assets/non-stripped/Foo.dll' already exists. at System.IO.FileSystem.LinkOrCopyFile (System.String sourceFullPath, System.String destFullPath) [0x000b9] in <06589f46f7f64859943a6a461f88c56e>:0 at System.IO.FileSystem.MoveFile (System.String sourceFullPath, System.String destFullPath) [0x0003a] in <06589f46f7f64859943a6a461f88c56e>:0 at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00083] in <06589f46f7f64859943a6a461f88c56e>:0 at Xamarin.Android.Tasks.CilStrip.DoExecute () [0x000a8] in <fd1a3c6c66de470d989dd70ca1f91267>:0 at Xamarin.Android.Tasks.CilStrip.Execute () [0x00000] in <fd1a3c6c66de470d989dd70ca1f91267>:0 This commit make sure we do not try to strip an assembly if it has already been stripped. It also adds a new Unit Test to make sure the build system behaves as we expect.

brendanzagaeski · 2019-10-19T00:09:30Z

Release status update

A new Preview version has now been published that includes the fix for this item. The fix is not yet included in a Release version. I will update this item again when a Release version is available that includes the fix.

Fix included in Xamarin.Android 10.1.0.1.

Fix included on Windows in Visual Studio 2019 version 16.4 Preview 2. To try the Preview version that includes the fix, check for the latest updates in Visual Studio Preview.

Fix included on macOS in Visual Studio 2019 for Mac version 8.4 Preview 1. To try the Preview version that includes the fix, check for the latest updates on the Preview updater channel.

brendanzagaeski · 2019-12-04T01:59:12Z

Release status update

A new Release version has now been published on Windows that includes the fix for this item. The fix is not yet published in a Release version on macOS. I will update this item again when a Release version is available on macOS that includes the fix.

Fix included in Xamarin.Android 10.1.0.30.

Fix included on Windows in Visual Studio 2019 version 16.4. To get the new version that includes the fix, check for the latest updates or install the latest version from https://visualstudio.microsoft.com/downloads/.

(Fix also included on macOS in Visual Studio 2019 for Mac version 8.4 Preview 2.1 and higher. To try the Preview version that includes the fix, check for the latest updates on the Preview updater channel.)

brendanzagaeski · 2020-01-08T21:08:06Z

Release status update

A new Release version has now been published on macOS that includes the fix for this item.

Fix included in Xamarin.Android 10.1.1.0.

Fix included on macOS in Visual Studio 2019 for Mac version 8.4. To get the new version that includes the fix, check for the latest updates on the Stable updater channel.

(Fix also included on Windows in Visual Studio 2019 version 16.4 and higher. To get the new version that includes the fix, check for the latest updates or install the latest version from https://visualstudio.microsoft.com/downloads/.)

Context: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1471933 Changes: xamarin/monodroid@297767f...2c3e786 * xamarin/monodroid@2c3e786bb: Bump to xamarin/android-sdk-installer@da944c3a (dotnet#1240) * xamarin/monodroid@4c43d82ee: Bump to xamarin/androidtools@c5729b16 (dotnet#1239)

Context: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1471933 Changes: xamarin/monodroid@297767f...2c3e786 * xamarin/monodroid@2c3e786bb: Bump to xamarin/android-sdk-installer@da944c3a (#1240) * xamarin/monodroid@4c43d82ee: Bump to xamarin/androidtools@c5729b16 (#1239)

pjcollins added the Area: App+Library Build Issues when building Library projects or Application projects. label Jan 25, 2018

jonpryor assigned vargaz Jan 26, 2018

pjcollins changed the title ~~Unable to compile some unit test suites with hybrid AOT enabled~~ Unable to Codegen test suite with hybrid AOT enabled May 21, 2018

jonpryor assigned radekdoulik Jun 3, 2018

radekdoulik assigned dellis1972 Jun 18, 2018

pjcollins changed the title ~~Unable to Codegen test suite with hybrid AOT enabled~~ Unable to rebuild project with hybrid AOT enabled Jun 27, 2018

dellis1972 mentioned this issue Jul 16, 2019

[Xamarin.Android.Build.Tasks] Dont use a temp location for AssemblyAttribute.cs #3354

Merged

JonDouglas added bug Component does not function as intended. vs-sync For internal use only; creates a VSTS "mirror" issue. labels Jul 16, 2019

JonDouglas added this to the d16-3 milestone Jul 16, 2019

xamarin-release-manager removed this from the d16-3 milestone Jul 16, 2019

xamarin-release-manager unassigned vargaz Jul 16, 2019

xamarin-release-manager unassigned radekdoulik Jul 16, 2019

dellis1972 mentioned this issue Sep 9, 2019

[Xamarin.Android.Build.Tasks] Fix issues with re running CilStrip. #3357

Merged

jonathanpeppers closed this as completed in #3357 Sep 10, 2019

brendanzagaeski added this to the In preview: 10.1 (d16-4) milestone Oct 19, 2019

jonpryor mentioned this issue Feb 19, 2022

Bump to xamarin/monodroid/main@2c3e786b #6781

Merged

ghost locked as resolved and limited conversation to collaborators Jun 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to rebuild project with hybrid AOT enabled #1240

Unable to rebuild project with hybrid AOT enabled #1240

pjcollins commented Jan 25, 2018 •

edited by xamarin-release-manager

Loading

jonpryor commented Jan 26, 2018

vargaz commented Jan 26, 2018

pjcollins commented Jan 26, 2018

vargaz commented Jan 26, 2018

vargaz commented Jan 26, 2018

pjcollins commented Jan 26, 2018 •

edited

Loading

vargaz commented Jan 26, 2018

jonpryor commented Jan 26, 2018

vargaz commented Jan 26, 2018

vargaz commented Jan 26, 2018

jonpryor commented Jan 26, 2018

pjcollins commented May 21, 2018

jonpryor commented Jun 5, 2018

radekdoulik commented Jun 18, 2018

radekdoulik commented Jun 18, 2018

radekdoulik commented Jun 18, 2018

dellis1972 commented Jul 16, 2019

brendanzagaeski commented Oct 19, 2019

brendanzagaeski commented Dec 4, 2019

brendanzagaeski commented Jan 8, 2020

Unable to rebuild project with hybrid AOT enabled #1240

Unable to rebuild project with hybrid AOT enabled #1240

Comments

pjcollins commented Jan 25, 2018 • edited by xamarin-release-manager Loading

Steps to Reproduce

Expected Behavior

Actual Behavior

Version Information

Log File

jonpryor commented Jan 26, 2018

vargaz commented Jan 26, 2018

pjcollins commented Jan 26, 2018

vargaz commented Jan 26, 2018

vargaz commented Jan 26, 2018

pjcollins commented Jan 26, 2018 • edited Loading

vargaz commented Jan 26, 2018

jonpryor commented Jan 26, 2018

vargaz commented Jan 26, 2018

vargaz commented Jan 26, 2018

jonpryor commented Jan 26, 2018

pjcollins commented May 21, 2018

jonpryor commented Jun 5, 2018

radekdoulik commented Jun 18, 2018

radekdoulik commented Jun 18, 2018

radekdoulik commented Jun 18, 2018

dellis1972 commented Jul 16, 2019

brendanzagaeski commented Oct 19, 2019

brendanzagaeski commented Dec 4, 2019

brendanzagaeski commented Jan 8, 2020

pjcollins commented Jan 25, 2018 •

edited by xamarin-release-manager

Loading

pjcollins commented Jan 26, 2018 •

edited

Loading