Skip to content

Commit

Permalink
[mono] new Sdk that selects mono runtime components (#54432)
Browse files Browse the repository at this point in the history
* Work in progress: new Sdk that selects mono runtime components

* Add props and targets description to the components doc

* condition the _MonoRuntimeAvailableComponents by RuntimeIdentifier

* [cmake] Write a component-manifest.props file during build

If we're not building a mono aot cross compiler, generate a
component-manifest.props file in artifacts/obj/mono/<host>/ that indicates if
the host will use static or dynamic components, and a list of the available
components, and the properties for constructing their names.

* Build Microsoft.NETCore.App.Runtime.Mono.<RID>.Sdk shared framework nuget

It seems to also generate a symbols nuget.  And in the nuget there's a
tools/mono-sdk-what-is-this.deps.json file from the
SharedFrameworkHostFileNameOverride property.  It would be nice to exclude that
stuff.

* put the compoonent-manifest.targets into the Sdk

* delete WIP in mono/nuget/

* fixup static component names in component-manifest.targets

* delete fixed fixme

* add missing $

* fix whitespace

* [cmake] switch to configure_file instead of file(CONFIGURE)

* add missing trailing slashes in .props.in file

* Add new Sdk packs to the workload manifest

* rework component-manifest.targets to use ItemGroups; move to new SDK

* Rename shared framework to Microsoft.NETCore.App.Runtime.Mono.<RID>.Props.Sdk

And only include component-manifest.props, not the targets

* Update manifest to include the new Props.Sdk and MonoTargets.Sdk

* Move RuntimeConfigParserTask into Microsoft.NET.Runtime.MonoTargets.Sdk

Consolidate all platform-independent tasks and targets into a single Sdk

* Add iossimulator-x86 props

* update components design doc

* Fix typo

* improve docs

* Add _MonoRuntimeComponentDontLink target output

* Drop component-manifest.props into runtime pack build/ directory

Remove from the Microsoft.NETCore.App.Mono.Props.Sdk workload nuget

* remove Microsoft.NETCore.App.Mono.Props.Sdk

* Import component-manifest.props from the runtime pack

Move the MonoTargets.Sdk import to each target platform that we support

* Fix typos

Co-authored-by: Rolf Bjarne Kvinge <[email protected]>

* Apply suggestions from code review

Co-authored-by: Ankit Jain <[email protected]>

* Add JsonToItemsTaskFactory

* fix whitespace

* Do some validation earlier in _MonoComputeAvailableComponentDefinitions

* Read component-manifest.json using the JsonToItemsTaskFactory

and bundle it in Microsoft.NET.Runtime.MonoTargets.Sdk

* remove ResolvedRuntimePack import from WorkloadManifest.targets

it's too early, and we have the JsonToItemsTaskFactory now to read the manifest

* Generate component-manifest.json in CMakeLists.txt

* Fix some copy-paste nits

* Use RuntimeFlavor==mono for runtime pack build directory

instead of TargetsMobile.  We want the build files (mono-components.json)
in every mono runtime pack, not just on mobile targets

* Apply suggestions from code review

Co-authored-by: Ankit Jain <[email protected]>

* rename component-manifest to RuntimeComponentManifest

* fixup nullability annotations

* fix whitespace

* fix formatting

* Misc fixes to JsonToItemsTaskFactory

* Rename MonoRuntimeComponentManifestReadTask

from MonoRuntimeComponentsReadManifestTask

* undo nullability annotation

Build doesn't like it for some reason (probably net472)

* fix incorrect task parameter name

* Remove Identity metadata from dictionary at json parsing time

Also improve nullability a bit by making the properties immutable

* Throw correct json deserializer exceptions

* Catch JsonException in an async function

We get nice error messages now like

```
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5): error : Failed to deserialize json from file 'artifacts/bin/mono/iOSSimulator.x64.Release/build/RuntimeComponentManifest.json', JSON Path: $.items._MonoRuntimeAvailableComponents[2], Line: 14, Position: 1 [component-manifest.sample.proj]
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5): error : JsonException: The JSON value could not be converted to System.Collections.Generic.Dictionary`2[System.String,System.String]. Path: $.identity | LineNumber: 0 | BytePositionInLine: 16. Path: $.items._MonoRuntimeAvailableComponents[2] | LineNumber: 14 | BytePositionInLine: 1. [component-manifest.sample.proj]
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5): error : InvalidOperationException: Cannot get the value of a token type 'Number' as a string. [component-manifest.sample.proj]
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5):
error :
[component-manifest.sample.proj]
```

* fixup comments

Co-authored-by: Rolf Bjarne Kvinge <[email protected]>
Co-authored-by: Ankit Jain <[email protected]>
  • Loading branch information
3 people authored Jul 8, 2021
1 parent 30d770d commit 893739f
Show file tree
Hide file tree
Showing 21 changed files with 858 additions and 79 deletions.
88 changes: 69 additions & 19 deletions docs/design/mono/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,10 +164,10 @@ To implement `feature_X` as a component. Carry out the following steps:
{ feature_X_cleanup },
feature_X_hello,
};
MonoComponentFeatureX *
mono_component_feature_X_init (void) { return &fn_table; }
void feature_X_cleanup (MonoComponent *self)
{
static int cleaned = 0;
Expand Down Expand Up @@ -207,10 +207,10 @@ To implement `feature_X` as a component. Carry out the following steps:
{ feature_X_cleanup },
feature_X_hello,
};
MonoComponentFeatureX *
mono_component_feature_X_init (void) { return &fn_table; }
void feature_X_cleanup (MonoComponent *self)
{
static int cleaned = 0;
Expand All @@ -229,7 +229,7 @@ To implement `feature_X` as a component. Carry out the following steps:
```c
MonoComponentFeatureX*
mono_component_feature_X (void);

...
MonoComponentFeatureX*
mono_component_feature_X_stub_init (void);
Expand All @@ -238,7 +238,7 @@ To implement `feature_X` as a component. Carry out the following steps:
* Add an entry to the `components` list to load the component to `mono/metadata/components.c`, and also implement the getter for the component:
```c
static MonoComponentFeatureX *feature_X = NULL;
MonoComponentEntry components[] = {
...
{"feature_X", "feature_X", COMPONENT_INIT_FUNC (feature_X), (MonoComponent**)&feature_X, NULL },
Expand All @@ -265,16 +265,66 @@ To implement `feature_X` as a component. Carry out the following steps:
## Detailed design - Packaging and runtime packs

The components are building blocks to put together a functional runtime. The
runtime pack includes the base runtime and the components and additional
properties and targets that enable the workload to construct a runtime for
various scenarios.

In each runtime pack we include:

- The compiled compnents for the apropriate host architectures in a well-known subdirectory
- An MSBuild props file that defines an item group that list each component name and has metadata that indicates:
- the path to the component in the runtime pack
- the path to the stub component in the runtime pack (if components are static)
- An MSBuild targets file that defines targets to copy a specified set of components to the app publish folder (if components are dynamic); or to link the runtime together with stubs and a set of enabled components (if components are static)

** TODO ** Write this up in more detail
runtime pack includes the base runtime and the components. The mono workload
includes the runtime pack and additional tasks, properties and targets that
enable the workload to construct a runtime for various scenarios.

For the target RID, we expose:

- `@(_MonoRuntimeComponentLinking)` set to either `'static'` or `'dynamic'` depending on whether the
current runtime pack for the current target includes runtime components as static archives or as
shared libraries, respectively.
- `@(_MonoRuntimeComponentSharedLibExt)` and `@(_MonoRuntimeComponentStaticLibExt)` set to the file
extension of the runtime components for the current target (ie, `'.a', '.so', '.dylib'` etc).
- `@(_MonoRuntimeAvailableComponents)` a list of component names without the `lib` prefix (if any)
or file extensions. For example: `'hot_reload; diagnostics_tracing'`.

Each of the above item lists has `RuntimeIdentifier` metadata. For technical reasons the mono
workload will provide a single `@(_MonoRuntimeAvailableComponent)` item list for all platforms. We
use the `RuntimeIdentifier` metadata to filter out the details applicable for the current platform.

- The target `_MonoSelectRuntimeComponents` that has the following inputs and outputs:
- input `@(_MonoComponent)` (to be set by the workload) : a list of components that a workload wants to use for the current
app. It is an error if this specifies any unknown component name.
- output `@(_MonoRuntimeSelectedComponents)` and `@(_MonoRuntimeSelectedStubComponents)` The names
of the components that were (resp, were not) selected. For example `'hot_reload;
diagnostics_tracing'`. Each item has two metadata properties `ComponentLib` and
`ComponentStubLib` (which may be empty) that specify the name of the static or dynamic library
of the component. This is not the main output of the target, it's primarily for debugging.
- output `@(_MonoRuntimeComponentLink)` a list of library names (relative to the `native/`
subdirectory of the runtime pack) that (for dynamic components) must be placed next to the
runtime in the application bundle, or (for static components) that must be linked with the
runtime to enable the components' functionality. Each item in the list has metadata
`ComponentName` (e.g. `'hot_reload'`), `IsStub` (`true` or `false`), `Linking` (`'static'` or
`'dynamic'`). This output should be used by the workloads when linking the app and runtime if
the workload uses an allow list of native libraries to link or bundle.
- output `@(_MonoRuntimeComponentDontLink)` a list of library names (relative to the `native/`
subdirectory of the runtime pack) that should be excluded from the application bundle (for
dynamic linking) or that should not be passed to the native linker (for static linking). This
output should be used by workloads that just link or bundle every native library from `native/`
in order to filter the contents of the subdirectory to exclude the disabled components (and to
exclude the static library stubs for the enabled components when static linking).

Generally workloads should only use one of `@(_MonoRuntimeComponentLink)` or
`@(_MonoRuntimeComponentDontLink)`, depending on whether they use an allow or block list for the
contents of the `native/` subdirectory.

Example fragment (assuming the mono workload has been imported):

```xml
<Project>
<ItemGroup Condition="'$(Configuration)' == 'Debug'">
<_MonoComponent Include="hot_reload;diagnostics_tracing" />
</ItemGroup>

<Target Name="PrintComponents" DependsOnTargets="_MonoSelectRuntimeComponents">
<Message Importance="High" Text="Runtime identifier: $(RuntimeIdentifier)" />
<Message Importance="High" Text="Selected : @(_MonoRuntimeSelectedComponents) %(ComponentLib)" />
<Message Importance="High" Text="Stubbed out : @(_MonoRuntimeSelectedStubComponents) %(ComponentStubLib)" />
<Message Importance="High" Text="Linking with lib @(_MonoRuntimeComponentLink) Stub: %(IsStub) Linking: %(Linking) Component: %(ComponentName)"/>

<Message Importance="High" Text="UnSelected : @(_MonoRuntimeUnSelectedComponents) %(ComponentLib)" />
<Message Importance="High" Text="Exclude these from linking: @(_MonoRuntimeComponentDontLink) Stub: %(IsStub) Linking: %(Linking) Component: %(ComponentName)" />
</Target>
</Project>
```
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@
<TargetPath>runtimes/$(RuntimeIdentifier)/native/include/%(RecursiveDir)</TargetPath>
</RuntimeFiles>

<RuntimeFiles Condition="'$(RuntimeFlavor)' == 'mono'"
Include="$(MonoArtifactsPath)\build\**\*.*"
ExcludeFromDataFiles="true">
<TargetPath>runtimes/$(RuntimeIdentifier)/build/%(RecursiveDir)</TargetPath>
</RuntimeFiles>

<CoreCLRCrossTargetFiles PackOnly="true" />
<CoreCLRCrossTargetFiles Condition="'%(FileName)' == 'clrjit' or '%(FileName)' == 'libclrjit'">
<TargetPath>runtimes/$(CoreCLRCrossTargetComponentDirName)_$(TargetArchitecture)/native</TargetPath>
Expand Down
9 changes: 9 additions & 0 deletions src/mono/mono.proj
Original file line number Diff line number Diff line change
Expand Up @@ -431,6 +431,9 @@
<ItemGroup Condition="'$(MonoComponentsStatic)' == 'true'">
<_MonoCMakeArgs Include="-DSTATIC_COMPONENTS=1" />
</ItemGroup>
<ItemGroup>
<_MonoCMakeArgs Include="-DMONO_COMPONENTS_RID=$(TargetOS)-$(TargetArchitecture)" />
</ItemGroup>

<PropertyGroup>
<_MonoCFLAGSOption>-DCMAKE_C_FLAGS="@(_MonoCPPFLAGS, ' ') @(_MonoCFLAGS, ' ')"</_MonoCFLAGSOption>
Expand Down Expand Up @@ -770,6 +773,7 @@
<Destination>$(RuntimeBinDir)cross\$(PackageRID)\opt$(ExeExt)</Destination>
</_MonoRuntimeArtifacts>
<_MonoIncludeArtifacts Include="$(MonoObjDir)out\include\**" />
<_MonoRuntimeBuildArtifacts Include="$(MonoObjDir)\build\**" />
<_MonoRuntimeArtifacts Condition="'$(_MonoIncludeInterpStaticFiles)' == 'true'" Include="$(MonoObjDir)out\lib\libmono-ee-interp.a">
<Destination>$(RuntimeBinDir)libmono-ee-interp.a</Destination>
</_MonoRuntimeArtifacts>
Expand Down Expand Up @@ -816,6 +820,11 @@
SkipUnchangedFiles="true"
Condition="'$(MonoGenerateOffsetsOSGroups)' == '' and ('$(TargetsMacCatalyst)' == 'true' or '$(TargetsiOS)' == 'true' or '$(TargetstvOS)' == 'true' or '$(TargetsAndroid)' == 'true' or '$(TargetsBrowser)' == 'true')"/>

<Copy SourceFiles="@(_MonoRuntimeBuildArtifacts)"
DestinationFiles="@(_MonoRuntimeBuildArtifacts->'$(RuntimeBinDir)build\%(RecursiveDir)%(Filename)%(Extension)')"
SkipUnchangedFiles="true"
Condition="'$(BuildMonoAOTCrossCompilerOnly)' != 'true'" />

<Exec Condition="'$(BuildMonoAOTCrossCompilerOnly)' != 'true' and '$(MonoGenerateOffsetsOSGroups)' == '' and ('$(TargetsOSX)' == 'true' or '$(TargetsMacCatalyst)' == 'true' or '$(TargetsiOS)' == 'true' or '$(TargetstvOS)' == 'true')" Command="install_name_tool -id @rpath/$(MonoFileName) $(RuntimeBinDir)$(MonoFileName)" />
</Target>

Expand Down
22 changes: 22 additions & 0 deletions src/mono/mono/component/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ set(${MONO_DIAGNOSTICS_TRACING_COMPONENT_NAME}-dependencies

#define a library for each component and component stub
function(define_component_libs)
# NOTE: keep library naming pattern in sync with RuntimeComponentManifest.targets
if (NOT DISABLE_LIBS)
foreach(component IN LISTS components)
add_library("mono-component-${component}-static" STATIC $<TARGET_OBJECTS:${component}-objects>)
Expand Down Expand Up @@ -121,6 +122,7 @@ endforeach()
if(NOT DISABLE_COMPONENTS AND NOT STATIC_COMPONENTS)
# define a shared library for each component
foreach(component IN LISTS components)
# NOTE: keep library naming pattern in sync with RuntimeComponentManifest.targets
if(HOST_WIN32)
add_library("mono-component-${component}" SHARED "${${component}-sources}")
target_compile_definitions("mono-component-${component}" PRIVATE -DCOMPILING_COMPONENT_DYNAMIC;-DMONO_DLL_IMPORT)
Expand Down Expand Up @@ -172,6 +174,26 @@ foreach(component IN LISTS components)
list(APPEND mono-components-stub-objects $<TARGET_OBJECTS:${component}-stub-objects>)
endforeach()

if(NOT MONO_CROSS_COMPILE)
set(TemplateMonoRuntimeComponentSharedLibExt "${CMAKE_SHARED_LIBRARY_SUFFIX}")
set(TemplateMonoRuntimeComponentStaticLibExt "${CMAKE_STATIC_LIBRARY_SUFFIX}")
set(TemplateRuntimeIdentifier "${MONO_COMPONENTS_RID}")
if(DISABLE_COMPONENTS)
set(TemplateMonoRuntimeComponentLinking "static")
set(TemplateMonoRuntimeAvailableComponents "")
else()
list(TRANSFORM components REPLACE "^(.+)$" "{ \"identity\": \"\\1\", \"RuntimeIdentifier\": \"${TemplateRuntimeIdentifier}\" }," OUTPUT_VARIABLE TemplateMonoRuntimeAvailableComponentsList)
list(JOIN TemplateMonoRuntimeAvailableComponentsList "\n" TemplateMonoRuntimeAvailableComponents)
if(STATIC_COMPONENTS)
set(TemplateMonoRuntimeComponentLinking "static")
else()
set(TemplateMonoRuntimeComponentLinking "dynamic")
endif()
endif()
# Write a RuntimeComponentManifest.json file in the artifacts/obj/mono/<host>/build/ directory
# without the ../.. the file would go in artifacts/obj/mono/<host>/mono/mini
configure_file( "${MONO_COMPONENT_PATH}/RuntimeComponentManifest.json.in" "../../build/RuntimeComponentManifest.json")
endif()

# component tests
set(MONO_EVENTPIPE_TEST_SOURCE_PATH "${MONO_EVENTPIPE_SHIM_SOURCE_PATH}/test")
Expand Down
16 changes: 16 additions & 0 deletions src/mono/mono/component/RuntimeComponentManifest.json.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"items": {
"_MonoRuntimeComponentLinking": [
{ "identity": "${TemplateMonoRuntimeComponentLinking}", "RuntimeIdentifier": "${TemplateRuntimeIdentifier}" },
],
"_MonoRuntimeComponentSharedLibExt": [
{ "identity": "${TemplateMonoRuntimeComponentSharedLibExt}", "RuntimeIdentifier": "${TemplateRuntimeIdentifier}" },
],
"_MonoRuntimeComponentStaticLibExt": [
{ "identity": "${TemplateMonoRuntimeComponentStaticLibExt}", "RuntimeIdentifier": "${TemplateRuntimeIdentifier}" },
],
"_MonoRuntimeAvailableComponents": [
${TemplateMonoRuntimeAvailableComponents}
],
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />

<PropertyGroup>
<PackageDescription>Provides the tasks+targets, for consumption by mono-based workloads</PackageDescription>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="$(RepoTasksDir)RuntimeConfigParser\RuntimeConfigParser.csproj" />
<ProjectReference Include="$(RepoTasksDir)JsonToItemsTaskFactory\JsonToItemsTaskFactory.csproj" />
</ItemGroup>

<ItemGroup>
<PackageFile Include="Sdk\Sdk.props" TargetPath="Sdk" />
<PackageFile Include="Sdk\Sdk.targets" TargetPath="Sdk" />
<PackageFile Include="build\$(MSBuildProjectName).props" TargetPath="build" />
<PackageFile Include="Sdk\RuntimeConfigParserTask.props" TargetPath="Sdk" />
<PackageFile Include="Sdk\RuntimeComponentManifest.props" TargetPath="Sdk" />
<PackageFile Include="Sdk\RuntimeComponentManifest.targets" TargetPath="Sdk" />
</ItemGroup>

<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
</Project>
38 changes: 38 additions & 0 deletions src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Mono Runtime Host support targets

This Sdk provides additional tasks and targets for workloads hosting the MonoVM .NET runtime.

## component-manifest.targets

See https://github.com/dotnet/runtime/blob/main/docs/design/mono/components.md

## RuntimeConfigParserTask
The `RuntimeConfigParserTask` task converts a json `runtimeconfig.json` to a binary blob for MonoVM's `monovm_runtimeconfig_initialize` API.
To use the task in a project, reference the NuGet package, with the appropriate nuget source.

### NuGet.config
```xml
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="dotnet6" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json" />
</packageSources>
</configuration>
```

### In the project file
```xml
<!-- Import the NuGet package into the project -->
<ItemGroup>
<PackageReference Include="Microsoft.NET.Runtime.MonoTargets.Sdk" Version="<desired-dotnet-6-sdk-version>" />
</ItemGroup>

<!-- Use the RuntimeConfigParser task in a target -->
<Target>
<RuntimeConfigParserTask
RuntimeConfigFile="$(Path_to_runtimeconfig.json_file)"
OutputFile="$(Path_to_generated_binary_file)"
RuntimeConfigReservedProperties="@(runtime_properties_reserved_by_host)">
</RuntimeConfigParserTask>
</Target>
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<Project>
<PropertyGroup>
<JsonToItemsTaskFactoryTasksAssemblyPath Condition="'$(MSBuildRuntimeType)' == 'Core'">$(MSBuildThisFileDirectory)..\tasks\net6.0\JsonToItemsTaskFactory.dll</JsonToItemsTaskFactoryTasksAssemblyPath>
<JsonToItemsTaskFactoryTasksAssemblyPath Condition="'$(MSBuildRuntimeType)' != 'Core'">$(MSBuildThisFileDirectory)..\tasks\net472\JsonToItemsTaskFactory.dll</JsonToItemsTaskFactoryTasksAssemblyPath>
</PropertyGroup>
<UsingTask TaskName="MonoRuntimeComponentManifestReadTask" TaskFactory="JsonToItemsTaskFactory.JsonToItemsTaskFactory" AssemblyFile="$(JsonToItemsTaskFactoryTasksAssemblyPath)">
<ParameterGroup>
<_MonoRuntimeComponentSharedLibExt ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
<_MonoRuntimeComponentStaticLibExt ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
<_MonoRuntimeComponentLinking ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
<_MonoRuntimeAvailableComponents ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
</ParameterGroup>
</UsingTask>
</Project>
Loading

0 comments on commit 893739f

Please sign in to comment.