Evaluate Release Infrastructure for building from the VMR

[mmitche]

When building from the VMR, all repos are built from the same repository. This will have some effect on the gather, staging, and release pipelines. We need to identity and file issues for areas that will need changes.

The VMR build artifacts will match what we get from the union of all the .NET builds used to ship the release today. For example, instead of getting a series of repos that produce sets of shipping and non-shipping packages, we'll get two huge sets. It is expected that the artifact identification process will be identical (or very close) to what is in use today (https://github.com/dotnet/arcade/blob/96e79593b458ac7aeff2571604c41fe915da5bff/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L66-L181).

One way to approach this exercise might be to imagine that the input to gather-drop is a single build BAR id from the dotnet-dotnet AzDO repo. Then work downstream through darc's gather-drop and the staging pipelines to see what could be affected. File issues accordingly,

Examples include:

• gather-drop knows about the release NuGet package layout. It identifies a discreet set of repos that produce shipping Nuget packages that .NET's release processes should handle. Anything else (e.g. Roslyn packages) would get dropped and not published by .NET. This is by design since those repo owners should publish their own packages on their own schedule. If all packages start showing up from dotnet/dotnet, this package identification will no longer work.
• The staging pipeline splits outputs by repo and checks signing status based on the signing exclusions in that repo. It won't be as easy to split these artifacts by repo, so how do we check signing? We can check signing of all the binaries, but we do need some level of information about exclusions.
• Similar as above, but for localization?
• How might our publishing be affected? When we publish outputs to staging locations, will something break?

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mmitche]

@MilenaHristova

[MilenaHristova]

So far I used the manifest.json from the 8.0 release and moved all of its assets to one build of dotnet-dotnet. I ran a test build of the staging pipeline (link) using this generated manifest to see where it will break and opened a few issues for stages that depend on the manifest format containing multiple repo builds.
What I am looking at now is

• what stages depend on the MergedManifest.xml. I have now created a manifest that merges all "merged manifests" from the 8.0 release and will try to use in it the pipeline
• creating a full picture of the interface between staging and release pipelines. (Most of it is documented in a wiki page)
• which part of this interface will be affected by the VMR

[MilenaHristova]

Next steps

• investigate how the release layout in dotnetstage will be affected - it's created by the Staging ring in the staging pipeline and is then used by the release pipeline
• investigate which stages use the MergedManifest.xml and how will they be affected

[MilenaHristova]

I believe the publishing that uses Arcades PublishArtifactsInManifest task will work fine. I didn't manage to properly test it though because the task is getting build information from Maestro for a BAR build ID and we don't have that for a dotnet-dotnet build.
But I looked through the code and I think it won't be affected much. We have a script (Microsoft internal link for publish-v3.ps1) that calls Arcades PublishArtifactsInManifest for each pair (build, channel) from manifest.json. This will work for one build of dotnet-dotnet as long as we have the channel information. I believe no change is needed inside PublishArtifactsInManifest since it uses the MergedManifest.xml and we will have that.

One concern that I have is how the blob asset ids are going to look inside MergedManifest.xml - now they show the relative path in the storage account for that asset - for example Runtime/8.0.0-rtm.23531.3/dotnet-nethost-symbols-linux-bionic-x64-8.0.0.tar.gz which is important for the layout inside dotnetcli.

[MilenaHristova]

Summary

Assumptions

• the VMR build will be published to BAR
• the VMR build will produce one MergedManifest.xml which contains some information about the assets layout. For example a asset id can look like Runtime/8.0.0-rtm.23531.3/dotnet-nethost-symbols-linux-bionic-x64-8.0.0.tar.gz meaning it's a runtime asset (I am not completely sure about that)
• the staging pipeline will call gather-drop with a single BAR ID for a VMR build
• the result of gather-drop will have similar structure to

├── manifest.json
├── shipping
        ├── packages
        └── assets
                 ├──  Runtime
                 ├──  Sdk
                 ├──  WindowsDesktop
                 ├──  aspnetcore
                 ├──  emsdk
                 └── symbols
├── nonshipping/
        ├── assets/  
        └── packages/  

with manifest.json containing information for a single build of dotnet-dotnet

Method

I took the manifest.json file from the .NET 8 release and copied all the assets from it to a new manifest file containing a single build of dotnet-dotnet. In the prep stage of a test staging build I downloaded the shipping and nonshipping folders from the GA staging and my fake manifest (from my dev storage account) and published them as artifacts so all the next stages in the pipeline will use those artifacts. Issues for the stages that failed are listed in the description.

Main tasks/issues

• Remove the release packages layout and find another way to describe packages ownership - Move release package layout creation to the release infrastructure issue
• Rethink aka.ms links creation - we need information on which repo the assets came from - internal Microsoft Create aka.ms config issue
• Find a way to configure custom settings on the repo level for SDL checks and signing validation (exceptions, Trust Services Automation configuration - internal Microsoft documentation)
• Optimize validation stages to be able to run on all assets without timing out or running out of memory

Other observations

• Many stages rely on the existence of manifest.json but most of them use the complete list of assets from it and not the separate builds. I opened issues for the ones that rely on having multiple repo builds.
• The config.json file is used in most of the stages both in staging and release. It's created from the manifest and the stage that generates it requires a small change to work with the VMR - explained in internal Microsoft Generate config.json issue
• I think publishing will work fine, see comment

Questions

• Are we going to create release tags in the product repos like .NET 7.0.13 tag in sdk? If yes, the tag creation will need redesigning - Create release tags issue

[tkapin]

@mmitche - The initial investigation is done, is there anything else you'd see being done here or it can be closed?

[MilenaHristova]

I'm closing the initial investigation as completed

[andriipatsula]

Added:

• https://github.com/dotnet/release/issues/815