Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reflection error with Unity when using il2cpp #5422

Closed
leohahn opened this issue Dec 4, 2018 · 18 comments
Closed

Reflection error with Unity when using il2cpp #5422

leohahn opened this issue Dec 4, 2018 · 18 comments
Assignees
Labels
bug c# inactive Denotes the issue/PR has not seen activity in the last 90 days.

Comments

@leohahn
Copy link

leohahn commented Dec 4, 2018

What version of protobuf and what language are you using?
Version: v3.6.1
Language: C#

What operating system (Linux, Windows, ...) and version?
MacOS Mojave 10.14

What runtime / compiler are you using (e.g., python version or gcc version)
Apple LLVM version 10.0.0 (clang-1000.11.45.2)
Target: x86_64-apple-darwin18.0.0
Thread model: posix

What did you do?

  • Open the following project on Unity 2018.3.0b10 (I think that this happens with any Unity 2018, maybe other years as well)
  • Run on the editor to see that the file Test.cs executes correctly.
  • Compile with il2cpp to iOS and run. The code fails with the following stacktrace:
DescriptorValidationException: google.protobuf.Value.kind: Method ClearKind not found in Google.Protobuf.WellKnownTypes.Value
  at Google.Protobuf.Reflection.OneofDescriptor.CreateAccessor (System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.OneofDescriptor..ctor (Google.Protobuf.Reflection.OneofDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor file, Google.Protobuf.Reflection.MessageDescriptor parent, System.Int32 index, System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.MessageDescriptor+<>c__DisplayClass4_0.<.ctor>b__0 (Google.Protobuf.Reflection.OneofDescriptorProto oneof, System.Int32 index) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.DescriptorUtil.ConvertAndMakeReadOnly[TInput,TOutput] (System.Collections.Generic.IList`1[T] input, Google.Protobuf.Reflection.DescriptorUtil+IndexedConverter`2[TInput,TOutput] converter) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.MessageDescriptor..ctor (Google.Protobuf.Reflection.DescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor file, Google.Protobuf.Reflection.MessageDescriptor parent, System.Int32 typeIndex, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor+<>c__DisplayClass1_0.<.ctor>b__0 (Google.Protobuf.Reflection.DescriptorProto message, System.Int32 index) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.DescriptorUtil.ConvertAndMakeReadOnly[TInput,TOutput] (System.Collections.Generic.IList`1[T] input, Google.Protobuf.Reflection.DescriptorUtil+IndexedConverter`2[TInput,TOutput] converter) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor..ctor (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.DescriptorPool pool, System.Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor.BuildFrom (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, System.Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.StructReflection..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.Value.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.JsonParser..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as ArgumentException: Invalid embedded descriptor for "google/protobuf/struct.proto".
  at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.StructReflection..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.Value.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.JsonParser..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as TypeInitializationException: The type initializer for 'Google.Protobuf.WellKnownTypes.StructReflection' threw an exception.
  at Google.Protobuf.WellKnownTypes.Value.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.JsonParser..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as TypeInitializationException: The type initializer for 'Google.Protobuf.JsonParser' threw an exception.
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
 
(Filename: currently not available on il2cpp Line: -1)
@leohahn leohahn changed the title Reflection error with unity when using il2cpp and Unity Reflection error with Unity when using il2cpp Dec 4, 2018
@jskeet jskeet self-assigned this Dec 4, 2018
@jskeet
Copy link
Contributor

jskeet commented Dec 4, 2018

Hmm. I've managed to get the same code to pass in a standalone app with il2cpp:

using Google.Protobuf;
using System;

namespace TestApp
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                var res = (IMessage) Activator.CreateInstance(typeof(TestApp.Error));
                var msg = JsonParser.Default.Parse("{\"code\":\"PIT-000\",\"msg\":\"GetError is returning a custom error\"}", res.Descriptor);
                Console.WriteLine(msg);
            }
            catch (TypeInitializationException ex)
            {
                Console.WriteLine("Type initialization exception");
                Console.WriteLine(ex.InnerException);
            }
        }
    }
}

Output:

{ "code": "PIT-000", "msg": "GetError is returning a custom error" }

Here's the il2cpp invocation I'm using:

"$IL2CPP" --convert-to-cpp \
  $GENERIC_SHARING \
  -architecture=x64 \
  -configuration=Release \
  -platform=WindowsDesktop \
  --dotnetprofile=Net45 \
  --libil2cpp-static \
  --emit-null-checks \
  --enable-array-bounds-check \
  --enable-stats \
  --compile-cpp \
  --directory=$IL2CPP_INPUT \
  --generatedcppdir=$IL2CPP_OUTPUT \
  --outputpath=$IL2CPP_OUTPUT/Program.exe \
  --cachedirectory=$IL2CPP_CACHE \
  --libil2cpp-cache-directory=$IL2CPP_CACHE

That's using Google.Protobuf 3.6.1 as well. I don't know which il2cpp flags are used by Unity. Hmm.

@leohahn
Copy link
Author

leohahn commented Dec 4, 2018

That's interesting. I'll maybe try with another version of Unity.

@jskeet
Copy link
Contributor

jskeet commented Dec 4, 2018

The version of Unity I have installed is 2018.2.0b2. Unfortunately I can't see a way of finding a version of il2cpp...

Here's my complete build script, if you want to try my test app for yourself - change the directories in the obvious way:

#!/bin/bash

set -e

declare -r UNITY_HOME=/c/Program\ Files/Unity/Hub/Editor/2018.2.0b2
declare -r IL2CPP=$UNITY_HOME/Editor/Data/il2cpp/build/il2cpp.exe
declare -r AOT_ASSEMBLIES=$UNITY_HOME/Editor/Data/MonoBleedingEdge/lib/mono/unityaot
declare -r IL2CPP_CACHE=il2cppcache
declare -r IL2CPP_OUTPUT=il2cppoutput
declare -r IL2CPP_INPUT=il2cppinput
declare -r PROTOBUF_ROOT=../../packages/Google.Protobuf.Tools.3.5.1/tools
declare -r PROTOC=$PROTOBUF_ROOT/windows_x64/protoc.exe
#declare -r GENERIC_SHARING=--enable-primitive-value-type-generic-sharing


rm -rf $IL2CPP_OUTPUT
mkdir $IL2CPP_OUTPUT
rm -rf $IL2CPP_INPUT
mkdir $IL2CPP_INPUT

$PROTOC -ITestApp -I$(echo $PROTOBUF_ROOT | sed 's/\//\\/g') --csharp_out=TestApp TestApp/*.proto

dotnet build

cp TestApp/bin/Debug/net471/* $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/mscorlib.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/Mono.Security.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/System.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/System.Core.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/System.Xml.dll" $IL2CPP_INPUT


"$IL2CPP" --convert-to-cpp \
  $GENERIC_SHARING \
  -architecture=x64 \
  -configuration=Release \
  -platform=WindowsDesktop \
  --dotnetprofile=Net45 \
  --libil2cpp-static \
  --emit-null-checks \
  --enable-array-bounds-check \
  --enable-stats \
  --compile-cpp \
  --directory=$IL2CPP_INPUT \
  --generatedcppdir=$IL2CPP_OUTPUT \
  --outputpath=$IL2CPP_OUTPUT/Program.exe \
  --cachedirectory=$IL2CPP_CACHE \
  --libil2cpp-cache-directory=$IL2CPP_CACHE

echo Running...

$IL2CPP_OUTPUT/Program.exe

(I note that I'm using protoc from 3.5.1, but I'd be astonished if that's changed enough to be an issue.)

@aaron-bray
Copy link
Contributor

Hey,

My team also recently had a few problems with C# reflections using Unity with il2cpp and enums using the 3.6.1 build. Note I am not the unity dev who dug into this, but this is a list of things we worked through. This sounds like the same issue as this, but maybe it's not?

  1. That code accessed only through reflection was stripped because IL2CPP did not know it was needed, and that was solved by link.xml with the contents (DataModelBindings is a .NET lib of all my protoc generated code)

<linker> <assembly fullname="DataModelBindings" preserve="all"/> </linker>

This is probably something that should be mentioned in Protobuf documentation?

Relevant part of Unity documentation: https://docs.unity3d.com/Manual/IL2CPP-BytecodeStripping.html

  1. Next this error came up:

Attempting to call method 'Google.Protobuf.Reflection.ReflectionUtil+ReflectionHelper`2[[Cdm.SerializeStateData, DataModelBindings, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null],[Cdm.eSerialization+Types+Type, DataModelBindings, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]]::.ctor' for which no ahead of time (AOT) code was generated.

(eSerialization is an enum)

The issue appears to be with this file in particular: https://github.com/protocolbuffers/protobuf/blob/master/csharp/src/Google.Protobuf/Reflection/ReflectionUtil.cs
GetReflectionHelper() with its call to Activator.CreateInstance() is where things seem to go wrong.

The use of generics through reflection + not letting the IL2CPP compiler know what classes will be needed is what caused the error - code was requested that was not generated, and there was no way to generate it on demand (as it would normally happen with JIT runtimes).
Somewhat relevant part of Unity documentation: https://docs.unity3d.com/Manual/ScriptingRestrictions.html ("Ahead-of-time compile" section)

To get past this we ended up replacing the smarts and quickness of the generics via reflection in ReflectionUtils.cs with pretty simple code that calls invoke instead.

But I don't think this is going to be something we want in protubuf as invoke can be kind of slow, but there is probably some way to apply Google.Protobuf.Reflection.FileDescriptor.ForceReflectionInitialization to the custom types to make sure AOT code is generated for them but I'm not sure how that's done exactly, and how easy it would be.

Again, this issue description is a summary of notes of what we did over a week or so and maybe difficult to follow. Please let me know if parts need further explanation. But with all these changes we were able to create and pass protobuf JSON from C# to C++ in our packaged Unity UWP game.

@joshpeterson
Copy link

Unfortunately I can't see a way of finding a version of il2cpp

Note that we don't version il2cpp separately from Unity. The IL2CPP version is the same as the Unity editor version.

@na-kubota
Copy link

is there any update on this issue?

@jskeet
Copy link
Contributor

jskeet commented Mar 20, 2019

@na-ka-na: Not from me, I'm afraid. This is only an area I can dip into occasionally, and this problem requires significantly more time. (This sort of brittleness is precisely why I've always been very nervous of supporting Unity - things that work on one version then failing on a different one.)

@na-kubota
Copy link

na-kubota commented Mar 20, 2019

i try Unity 18.3.8f1 same Error

DescriptorValidationException: google.protobuf.Value.kind: Method ClearKind not found in Google.Protobuf.WellKnownTypes.Value
  at Google.Protobuf.Reflection.OneofDescriptor.CreateAccessor (System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.OneofDescriptor..ctor (Google.Protobuf.Reflection.OneofDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor file, Google.Protobuf.Reflection.MessageDescriptor parent, System.Int32 index, System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.MessageDescriptor+<>c__DisplayClass4_0.<.ctor>b__0 (Google.Protobuf.Reflection.OneofDescriptorProto oneof, System.Int32 index) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.DescriptorUtil.ConvertAndMakeReadOnly[TInput,TOutput] (System.Collections.Generic.IList`1[T] input, Google.Protobuf.Reflection.DescriptorUtil+IndexedConverter`2[TInput,TOutput] converter) [0x00000] in <0000000000

in this Simple Code

public class NewBehaviourScript : MonoBehaviour
{
    // Start is called before the first frame update
    void Start()
    {
        var data = ProtoData.Parser.ParseJson("{\"id\":1,\"name\":\"\u58f0\u512a1\"}");
    }
}

Is the cause Unity Version?
please...

@jskeet
Copy link
Contributor

jskeet commented Mar 20, 2019

Is the cause Unity Version?

Well sort of - in that I've had this working with some versions of Unity. But I'm not suggesting that you should change which version of Unity you're using. I'm suggesting that fixing this properly requires a significant amount of time and effort, ideally with automated testing against multiple versions of il2cpp.

@na-kubota
Copy link

na-kubota commented Mar 20, 2019

ok, very thank you.
What should I do?

@jskeet
Copy link
Contributor

jskeet commented Mar 20, 2019

Unfortunately at the moment I would suggest avoiding any requirement on reflection - that's calling ToString() or parsing JSON - within Unity. Depending on the JSON and your protos, you may be able to use another JSON parser to populate your proto model, and then use it as normal with binary serialization, which doesn't require reflection.

@na-kubota
Copy link

ok i try other Parse
thank you!

@leohahn
Copy link
Author

leohahn commented Mar 20, 2019

I have managed to fix the issue with the latest version of protobuf by adding a link.xml file in the Assets folder of the project with the following contents:

<linker>
    <assembly fullname="Google.Protobuf" preserve="all"/>
</linker>

Since we also have to support .NET3.5, we compiled protobuf for that platform as well.

@jartsa
Copy link

jartsa commented May 10, 2019

I have managed to fix the issue with the latest version of protobuf by adding a link.xml file in the Assets folder of the project with the following contents:

<linker>
    <assembly fullname="Google.Protobuf" preserve="all"/>
</linker>

Since we also have to support .NET3.5, we compiled protobuf for that platform as well.

Thank You!

@ghost
Copy link

ghost commented Nov 5, 2019

  1. That code accessed only through reflection was stripped because IL2CPP did not know it was needed, and that was solved by link.xml with the contents (DataModelBindings is a .NET lib of all my protoc generated code)

<linker> <assembly fullname="DataModelBindings" preserve="all"/> </linker>

Fixed the issue for me - thanks!

@vitorbaraujo
Copy link

I'm having a similar issue: when trying to get a message descriptor I got a ExecutionEngineException error. This is the stack:

ExecutionEngineException: Attempting to call method 'Google.Protobuf.Reflection.ReflectionUtil+ReflectionHelper`2[[Google.Protobuf.Reflection.FieldDescriptorProto, Google.Protobuf, Version=3.6.1.0, Culture=neutral, PublicKeyToken=a7d26565bac4d604],[Google.Protobuf.Reflection.FieldDescriptorProto+Types+Label, Google.Protobuf, Version=3.6.1.0, Culture=neutral, PublicKeyToken=a7d26565bac4d604]]::.ctor' for which no ahead of time (AOT) code was generated.
  at System.Reflection.MonoCMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.ReflectionUtil.GetReflectionHelper (System.Type t1, System.Type t2) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.ReflectionUtil.CreateFuncIMessageObject (System.Reflection.MethodInfo method) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FieldAccessorBase..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.SingleFieldAccessor..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FieldDescriptor.CreateAccessor () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FieldDescriptor.CrossLink () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.MessageDescriptor.CrossLink () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.CrossLink () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.BuildFrom (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.DescriptorReflection..cctor () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.get_DescriptorProtoFileDescriptor () [0x00000] in <filename unknown>:0 

I can't seem to use FileDescriptor.ForceReflectionInitialization<T> here since Google.Protobuf.Reflection.FieldDescriptorProto.Types.Label enum is internal.

I'm using version 3.6.1 compiled to .NET 3.5 and using Unity 2018.4.23f1.

Copy link

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

@github-actions github-actions bot added the inactive Denotes the issue/PR has not seen activity in the last 90 days. label May 12, 2024
Copy link

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug c# inactive Denotes the issue/PR has not seen activity in the last 90 days.
Projects
None yet
Development

No branches or pull requests

7 participants