RFC: Migrate bgen to use roslyn instead of the reflection API. #21308
Comments
|
I like this idea, it can make the bindings a lot simpler, both for us, and particularly for customers. The current binding procedure/API is rather unintuitive. One potential place where things would get uglier is availability attributes, and how we use them to exclude APIs on certain platforms. Roslyn isn't able to remove stuff from the compilation, so any of the [BindingType]
public class CBManager : NSObject {
#if __TVOS__
[SupportedOSPlatform ("tvos13.0")]
[UnsupportedOSPlatform ("ios")]
[UnsupportedOSPlatform ("maccatalyst")]
[UnsupportedOSPlatform ("macos")]
[Export ("authorization", ArgumentSemantic.Assign)]
public partial static CBManagerAuthorization Authorization { get; }
#endif
}Another is the [BindFrom (typeof (NSNumber))]
[Export ("roll", ArgumentSemantic.Strong)]
nfloat? Roll { get; }I think a good plan forward would be:
Misc notes:
This is fixable with bgen; the problem is that design-time builds are disabled. Enabling design-time builds is easy, but we don't want to run design-time builds remotely from Windows, so then the problem becomes how to not run design-time builds remotely, which turned out to be a rather bigger problem, which I've been working on on and off for quite a while now (related to the next point).
bgen works fine from Windows already (we run the bgen tests on Windows in fact), the problem is that resources in binding projects don't (which I'm working on fixing). Switching to Roslyn won't make this easier, it's completely unrelated. |
rolfbjarne
added
enhancement
I consider these problems too in the same exact way, flip BindAs to be from and move to conditional compilation and have a csprof per platform. Should make things a lot easier. We could consider adding an extra step to the linker and mark methods to be fully removed via an attribute. |
|
This looks good to me, sorry I took longer to read fully and get back to you, once we start implementing should we create sub issues for the attribute design? For example |
Yes, that is already considered in the POC. |
|
Adding here some information about the possible need to add partial constructors and partial events in C# for .net 10. ConstructorsConstructors in the current bindings are defined with the following API (this are real examples from the SDK): [MacCatalyst (13, 1)]
[BaseType (typeof (AVAudioBuffer))]
[DisableDefaultCtor] // just like base class (AVAudioBuffer) can't, avoid crash when ToString call `description`
interface AVAudioCompressedBuffer {
[Export ("initWithFormat:packetCapacity:maximumPacketSize:")]
NativeHandle Constructor (AVAudioFormat format, uint packetCapacity, nint maximumPacketSize);
[Export ("initWithFormat:packetCapacity:")]
NativeHandle Constructor (AVAudioFormat format, uint packetCapacity);
}The above code is used to generate the following code in the bindings: [Export ("initWithFormat:packetCapacity:maximumPacketSize:")]
[BindingImpl (BindingImplOptions.GeneratedCode | BindingImplOptions.Optimizable)]
public AVAudioCompressedBuffer (AVAudioFormat format, uint packetCapacity, nint maximumPacketSize)
: base (NSObjectFlag.Empty)
{
var format__handle__ = format!.GetNonNullHandle (nameof (format));
if (IsDirectBinding) {
InitializeHandle (global::ObjCRuntime.Messaging.NativeHandle_objc_msgSend_NativeHandle_UInt32_IntPtr (this.Handle, Selector.GetHandle ("initWithFormat:packetCapacity:maximumPacketSize:"), format__handle__, packetCapacity, maximumPacketSize), "initWithFormat:packetCapacity:maximumPacketSize:");
} else {
InitializeHandle (global::ObjCRuntime.Messaging.NativeHandle_objc_msgSendSuper_NativeHandle_UInt32_IntPtr (this.SuperHandle, Selector.GetHandle ("initWithFormat:packetCapacity:maximumPacketSize:"), format__handle__, packetCapacity, maximumPacketSize), "initWithFormat:packetCapacity:maximumPacketSize:");
}
}
[Export ("initWithFormat:packetCapacity:")]
[BindingImpl (BindingImplOptions.GeneratedCode | BindingImplOptions.Optimizable)]
public AVAudioCompressedBuffer (AVAudioFormat format, uint packetCapacity)
: base (NSObjectFlag.Empty)
{
var format__handle__ = format!.GetNonNullHandle (nameof (format));
if (IsDirectBinding) {
InitializeHandle (global::ObjCRuntime.Messaging.NativeHandle_objc_msgSend_NativeHandle_UInt32 (this.Handle, Selector.GetHandle ("initWithFormat:packetCapacity:"), format__handle__, packetCapacity), "initWithFormat:packetCapacity:");
} else {
InitializeHandle (global::ObjCRuntime.Messaging.NativeHandle_objc_msgSendSuper_NativeHandle_UInt32 (this.SuperHandle, Selector.GetHandle ("initWithFormat:packetCapacity:"), format__handle__, packetCapacity), "initWithFormat:packetCapacity:");
}
}We could workaround the fact that constructors are not partial by creating private methods and writing the constructors from scripts, but this is less than ideal and would probably mean we could leak APIs we don't want public. EventsOur APIS generate different type of events, which is hard to do with the current C# version in which we cannot have partial events. Examples: UIKit events exampleBinding: [Export ("actionSheet:clickedButtonAtIndex:"), EventArgs ("UIButton")]
void Clicked (UIActionSheet actionSheet, nint buttonIndex);
[Export ("actionSheetCancel:"), EventArgs ("UIActionSheet")]
void Canceled (UIActionSheet actionSheet);
[Export ("willPresentActionSheet:"), EventArgs ("UIActionSheet")]
void WillPresent (UIActionSheet actionSheet);
[Export ("didPresentActionSheet:"), EventArgs ("UIActionSheet")]
void Presented (UIActionSheet actionSheet);
[Export ("actionSheet:willDismissWithButtonIndex:"), EventArgs ("UIButton")]
void WillDismiss (UIActionSheet actionSheet, nint buttonIndex);
[Export ("actionSheet:didDismissWithButtonIndex:"), EventArgs ("UIButton")]
void Dismissed (UIActionSheet actionSheet, nint buttonIndex);Generated code: public unsafe partial class UIAlertView : UIView, INSCoding {
public event EventHandler Canceled {
add { EnsureUIAlertViewDelegate (this)!.canceled += value; }
remove { EnsureUIAlertViewDelegate (this)!.canceled -= value; }
}
public event EventHandler<UIButtonEventArgs> Clicked {
add { EnsureUIAlertViewDelegate (this)!.clicked += value; }
remove { EnsureUIAlertViewDelegate (this)!.clicked -= value; }
}
public event EventHandler<UIButtonEventArgs> Dismissed {
add { EnsureUIAlertViewDelegate (this)!.dismissed += value; }
remove { EnsureUIAlertViewDelegate (this)!.dismissed -= value; }
}
public event EventHandler Presented {
add { EnsureUIAlertViewDelegate (this)!.presented += value; }
remove { EnsureUIAlertViewDelegate (this)!.presented -= value; }
}
public UIAlertViewPredicate? ShouldEnableFirstOtherButton {
get { return EnsureUIAlertViewDelegate (this)!.shouldEnableFirstOtherButton; }
set { EnsureUIAlertViewDelegate (this)!.shouldEnableFirstOtherButton = value; }
}
public event EventHandler<UIButtonEventArgs> WillDismiss {
add { EnsureUIAlertViewDelegate (this)!.willDismiss += value; }
remove { EnsureUIAlertViewDelegate (this)!.willDismiss -= value; }
}
public event EventHandler WillPresent {
add { EnsureUIAlertViewDelegate (this)!.willPresent += value; }
remove { EnsureUIAlertViewDelegate (this)!.willPresent -= value; }
}
}WeakEvent exampleBinding [Wrap ("WeakDelegate")]
IUIAlertViewDelegate Delegate { get; set; }
[Export ("delegate", ArgumentSemantic.Assign)]
[NullAllowed]
NSObject WeakDelegate { get; set; }Generated code [BindingImpl (BindingImplOptions.GeneratedCode | BindingImplOptions.Optimizable)]
object? __mt_WeakDelegate_var;
[BindingImpl (BindingImplOptions.GeneratedCode | BindingImplOptions.Optimizable)]
public virtual NSObject? WeakDelegate {
[Export ("delegate", ArgumentSemantic.Assign)]
get {
global::UIKit.UIApplication.EnsureUIThread ();
NSObject? ret;
if (IsDirectBinding) {
ret = Runtime.GetNSObject (global::ObjCRuntime.Messaging.NativeHandle_objc_msgSend (this.Handle, Selector.GetHandle ("delegate")))!;
} else {
ret = Runtime.GetNSObject (global::ObjCRuntime.Messaging.NativeHandle_objc_msgSendSuper (this.SuperHandle, Selector.GetHandle ("delegate")))!;
}
MarkDirty ();
__mt_WeakDelegate_var = ret;
return ret!;
}
[Export ("setDelegate:", ArgumentSemantic.Assign)]
set {
UIApplication.EnsureDelegateAssignIsNotOverwritingInternalDelegate (__mt_WeakDelegate_var, value, GetInternalEventDelegateType);
global::UIKit.UIApplication.EnsureUIThread ();
var value__handle__ = value.GetHandle ();
if (IsDirectBinding) {
global::ObjCRuntime.Messaging.void_objc_msgSend_NativeHandle (this.Handle, Selector.GetHandle ("setDelegate:"), value__handle__);
} else {
global::ObjCRuntime.Messaging.void_objc_msgSendSuper_NativeHandle (this.SuperHandle, Selector.GetHandle ("setDelegate:"), value__handle__);
}
MarkDirty ();
__mt_WeakDelegate_var = value;
}
}
[BindingImpl (BindingImplOptions.GeneratedCode | BindingImplOptions.Optimizable)]
public IUIAlertViewDelegate Delegate {
get {
return (WeakDelegate as IUIAlertViewDelegate)!;
}
set {
var rvalue = value as NSObject;
if (!(value is null) && rvalue is null)
throw new ArgumentException ("The object passed of type " + value.GetType () + " does not derive from NSObject");
WeakDelegate = rvalue;
}
}As far as I can tell there is no reasonable workaround with a roslyn code generator and the current version of the language. |
RFC: Migrate bgen to use roslyn instead of the reflection API.
The following is a proposal to migrate the current bgen implementation, based on the reflection API, to a roslyn generator. A proof of concept can be found at https://github.com/mandel-macaque/macios-generator
Introduction
The Microsoft.iOS/MacOS bindings relies in a custom tool called bgen. bgen generates code by reading a dll that contains all the binding definitions described as a interface. Each interface represents a native class, bgen loads the assembly, reads the metadata that was used to tag the interfaces and generates the final classes and any needed trampoline.
Although the approach was the best possible one when roslyn did not exists, the solution is limited by the reflection API and the fact that an intermediate generation process is needed outside the normal compilation. The following problems come to mind:
For developers:
For the SDK result:
Timing
Up until the release of net9 and C# 13 the language did not support partial properties which limited the possibility of using Roslyn for the binding generations since if would heavily impact the quality of the final API. The new language addition allows to have a Roslyn code generator that is able to cover all the binding cases.
Benefits
A Roslyn Code Generator has access to both the SyntaxTree and the Semantic model of the current compilation, both pieces allow to minimize the need of type annotations, adds better support for nullability and all possible new language features as well as it allows to share knowledge between a code generator and a Roslyn analyzer improving the feedback loop for developers and the quality of the SDK. Some of the direct benefits are:
to use intellisense from the binding project.
Known issues with bgen
Due to the nature of the reflection API, there are several issues that the current generator cannot fix easily:
Build complexity
The build of the Microsoft.iOS SDK is very complex because it occurs in several phases. This makes most new developers struggle to understand in which of the available sections under frameworks.sources each of the files has to go. This is specially painful when we are adding structure that need to have the same size and layout in all steps of the build. Moving to Roslyn allows to all files be in the same compilation unit as well as it allows to have a csproj per platform to organize the build.
Having all code in the same build also allows to remove several classes that we have that are used to indentify which frameworks are present per platform. With all files in the same context, Roslyns semantic model can be used to find a symbol and decide if a class is generated or not.
API changes
Moving away from bgen to Roslyn means that we have to change the API to support intellisense on customer projects. The API change is 1:1 meaning that the migration from one version to another can be automated. Moving to Roslyn does not force our customers to move to Roslyn. bgen can be distributed customers as long as it is needed.
The following code snippets show the differences between API
Class definitions
bgen requires all classes to be interfaces, and use the BaseType attribute to define the class hierarchy:
With a move to Roslyn the bindings can be changed to use classes and normal class inheritance allow users to access the class in their current compilation unit.
Protocol implementations in bgen are represented by class inheritance. That is complicated and in some occasions it forced developers to create empty interfaces as with the following example:
With Roslyn this becomes much simpler:
Class visibility is also simplified, while bgen requires classes to be marked with [Abstract] to state that they are, a roslyn code generator does not have that
limitation, same with static classes:
Which becomes
Because we will be working with classes, we can simply list the interfaces that are implemented and the code generator will write the needed interface methods.
Notes
Class must ALWAYS be
partiala Roslyn Analyzer can help avoiding common mistakes and Sharpie will usually be the tool used to write bindings.Methods and Properties
The new code generation relies heavily on the fact that properties and methods can be partial. The changes for method and properties are minimun, while for constructors we have to do some extra changes.
The bgen binding of methods and properties look like the following:
With roslyn we can define the method and properties as follows, pay attention at the fact that we added partial and virtual as needed. The Static and Abstract attributes are not longer needed. Nullability is directly supported, that removes the need of the NullAllowed attribute.
The WeakDelegate removal deserves as special mention. Because we do not longer need the WarpAttribute, the weak delegate property can be written in plain C#, which will be generated by objective sharpie.
Notes
All methods in the SDK are virtual, this will have to be respected in the Microsoft.iOS binding by adding the
virtualkeyword in the method definition. A roslyn analyzer can help spot this possible common mistakeConstructor
Constructor are a special case of methods, the biggest problem is that a constructor in C# 13 cannot be partial, nevertheless that does not suppose a major problem, since we can declare the init methods as constructors and the Roslyn code generator can create a constructor that will take the same parameters. This init methods should be marked as private and can me marked to be inlined by the compiler.
In bgen constructors are defined as follows:
which becomes:
The above code can easily generate the following constructors for the class:
Notes
Adding more than on constructor with the same parameters will result in a compilation error, this is an issue present already in bgen with a simple workaround that can be implemented as a code generation in a Roslyn analyzer.
Categories
Categories are probably the easiest bindings to port (see migration). They represent a extension class for a specific type. Currently in bgen, as with all other bindings, they are represented as an interface decorated with the Category attribute:
Easily converted to a class via:
The generator can easily generate the partial method that uses the self pointer.
Protocols
Protocols represent interfaces that are implemented in the native world. bgen does not add
Iin front of protocols which is very confusing to C# developers, but that is done because everything is an interface and later theIwill be added in the generated code. This is not longer the need with Roslyn which can define the interface and then in the implementing class add the missing methods.Old bgen binding:
new code:
More importantly we can support optional methods in protocols with the use of default interface methods.
bgen code
Roslyn approach:
Enums and Smart Enums
Enums perse should be a problem for bgen but they add some compliations to the build:
Moving to a Roslyn based solution solve this problem, there are no two steps in the build process and enums are allways present.
Smart enums are, with categories, the simples biding type to port since they are extension classes. The only work to do is to annotate them with the BindingType attribute.
General
All classed, interfaces and enums that have to be generated will have to be annotated with the BindingType attribute. This annotations is to allow the Roslyn code generator filter the nodes that need to be generated. bgen does not need that since it assumes that is interested in all types, the code generator on the other hand needs a way to filter.
Multi language support
Because Roslyn allow use to separate the understanding of the syntaxt tree and the semantic model from the code generation, moving to this approach will allow to have different backends that can write different types of trampolines opening the door for swift bindings once there is support from the runtime team.
Work items
Moving from bgen to any new implementation is a scary task and we need to be able to do it in a way that if we face a set back, priorities might psuh back, people come and go etc. The good thing of moving to Roslyn is that we do not have to port all binding. Since Roslyn is part of the last step, we can add the code generator to the last compilation, after bgen has generated code and just focus on smaller tasks. A tentative road map would be:
Migration
The migration of the SDK does not need to be done by hand. We can have 3 different tools that will share a lot of knowledge that will help with the migration of the SDK and that can later be a product for customers:
The text was updated successfully, but these errors were encountered: