Isolate package flow between repos

[mthalman]

Contributes to dotnet/source-build#3608

As a prerequisite to enabling parallel builds of the repos in the VMR, it's necessary to isolate the package flow between repos. This ensures that repo dependencies are defined correctly. Prior to these changes, the packages outputted by repos went to the same directory (Shipping or NonShipping) and those two directories were used as the package feed inputs to dependent repos. This would be dangerous in the context of running builds in parallel across repos because any ill-defined dependencies could lead to race conditions. For example, consider the msbuild repo's dependency on System.Text.Json from the runtime repo. If runtime was not defined as a dependency of msbuild, then there's no guarantee that System.Text.Json will exist when it restores it. Based on timing, it may exist in one build and then not exist in another build.

To solve this problem, the packages output from a repos build are placed in a repo-specific package location as a sub-directory of the package location (Shipping or NonShipping). Previously, all repos would output their packages to artifacts/packages/Release/[NonShipping|Shipping]. With these changes, they output to artifacts/packages/Release/[NonShipping|Shipping]/<repo-name>. This isolates all packages on a per-repo basis. The next step is to provide access to these packages based on dependencies. Going back to msbuild's dependency on runtime, this is accomplished by modifying msbuild's nuget.config file to include feeds specific to runtime:

<add key="source-built-runtime" value="/vmr/artifacts/packages/Release/Shipping/runtime/" />
<add key="source-built-transport-runtime" value="/vmr/artifacts/packages/Release/NonShipping/runtime/" />

[mthalman]

This kind of package isolation also has use in other scenarios outside the context of parallel build support. @mmitche - Can you comment more on that?

[NikolaMilosavljevic]

Presumably blob-feed/assets can continue to be shared and does not require repo-level isolation?

[mthalman]

Presumably blob-feed/assets can continue to be shared and does not require repo-level isolation?

I believe so since I don't know of any repos that depend on the assets directory (except for installer but that's at the end of the build anyway).

[ViktorHofer]

How did you determine that list? Does this represent the source-build or the vertical build graph? As an example, is the removal of templating here due to the repository not being a dependency for source build but for vertical build? cc @mmitche

[mthalman]

The removal of templating here is because there's literally no dependency on the templating repo output.

But you bring up a good point about the difference between vertical and source build. In some cases, there are dependencies that exist purely to satisfy source build. In order words, the dependencies for vertical build would be a subset of the dependencies for source build. I'll investigate that more so that we can define these depends with the source-only flag as appropriate.

[mmitche]

Yeah we will need to go and annotate these dependencies with SB properties in some cases. The "Microsoft" graph should emulate as closely the active flow that happens via Maestro on a per-release basis. The SB graph will start with this flow, and then add new edges where we repos are redistributing implementation packages of older versions.

[MichaelSimons]

Are there a set of rules/guidelines for determining when to/when not to declare a dependency as a repository dependency? It would be good to capture this in a guideline.

[ViktorHofer]

... and how to keep those in sync.

[mthalman]

Ugh, looks like I need to resolve conflicts still.

[mthalman]

Yeah, the changes from #18591 dealt a serious blow here. This will take a bit of time.

[ViktorHofer]

@mthalman This is really great! Would you mind updating the PR description to the current state? I think this will help devs coming back to this PR. When merging please also make sure that the commit description is up-to-date in regards to the added feeds and the output paths (artifacts/packages vs artifacts/obj/x64/Release/blob-feed/...).