Put artifacts in well-known locations #18591

ViktorHofer · 2024-02-09T09:42:13Z

Fixes dotnet/source-build#4104
Fixes dotnet/source-build#3696
Contributes to dotnet/source-build#4101
Contributes to dotnet/source-build#4158

Split shipping and non-shipping packages and put them into the Arcade default locations:
-> artifacts/packages//Shipping
-> artifacts/packages//NonShipping

Put artifacts under -> artifacts/assets/<config>

Add the local feed for the addition package directory to NuGet.config.

Remove the non-shipping packages list generation now that artifacts are grouped.
Generate the intermediate PackageVersions* files in -> artifacts/obj/PackageVersions
Extract the repo symbol archives intermediates to -> artifacts/obj/Symbols/
Improve artifacts logging and ensuring that artifacts are created
Sequence targets better and apply correct DependsOnTargets
Small clean-up
Allow the poison infrastructure to analyze SBRP packages based on item metadata instead of an expected directory structure in the archive. Make the poison infrastructure analyze nuget packages directly.

src/SourceBuild/content/repo-projects/installer.proj

ViktorHofer · 2024-02-09T10:32:02Z

src/SourceBuild/content/repo-projects/Directory.Build.targets

+  <Target Name="SetSourceBuiltSdkOverrides"
+          Condition="'@(SourceBuiltSdkOverride)' != ''">
+    <ItemGroup>
+      <!-- Set the environment variables for MSBuild to look for our additional SDK Resolvers and or our resolver to find our source-built SDKs. -->
+      <EnvironmentVariables Include="MSBUILDADDITIONALSDKRESOLVERSFOLDER=$(VSMSBuildSdkResolversDir)" />
+      <EnvironmentVariables Include="SOURCE_BUILT_SDK_ID_%(SourceBuiltSdkOverride.Group)=%(SourceBuiltSdkOverride.Identity)" />
+      <EnvironmentVariables Include="SOURCE_BUILT_SDK_VERSION_%(SourceBuiltSdkOverride.Group)=%(SourceBuiltSdkOverride.Version)" />
+      <EnvironmentVariables Condition="'%(SourceBuiltSdkOverride.Location)' != ''" Include="SOURCE_BUILT_SDK_DIR_%(SourceBuiltSdkOverride.Group)=%(SourceBuiltSdkOverride.Location)/" />
+      <EnvironmentVariables Condition="'%(SourceBuiltSdkOverride.Location)' == ''" Include="SOURCE_BUILT_SDK_DIR_%(SourceBuiltSdkOverride.Group)=$(SourceBuiltSdksDir)%(SourceBuiltSdkOverride.Identity)/" />
+    </ItemGroup>
+  </Target>


This target just got moved up to the build stage.

ViktorHofer · 2024-02-09T10:32:19Z

src/SourceBuild/content/repo-projects/Directory.Build.targets

-  <Target Name="Build" DependsOnTargets="BuildRepoReferences;RepoBuild" />
-
-  <Target Name="Package"
-          AfterTargets="Build"
-          DependsOnTargets="
-    CopyRepoArtifacts;
-    CopyInnerBuildRestoredPackages;
-    EnsurePackagesCreated;
-    ExtractToolPackage;
-    CleanupRepo" />
-


These targets got combined and moved down.

1. Shipping and non-shipping artifacts to Arcade default locations: -> artifacts/packages/<config>/Shipping -> artifacts/packages/<config>/NonShipping Add the local feed for the addition package directory to NuGet.config Remove the non-shipping packages list generation now that artifacts are grouped. 2. Genereated the intermediate PackageVersions* files in -> artifacts/obj/PackageVersions 3. Extract the repo symbol archives intermediates to -> artifacts/obj/Symbols/ 4. Sequence targets better and apply correct DependsOnTargets 5. Small clean-up Fixes dotnet/source-build#4104 Fixes dotnet/source-build#3696

src/SourceBuild/content/repo-projects/source-build-reference-packages.proj

src/SourceBuild/content/repo-projects/installer.proj

.devcontainer/vmr-source-build/README.md

MichaelSimons · 2024-02-09T15:28:02Z

src/SourceBuild/content/Directory.Build.props

-    <SourceBuiltBlobFeedDir>$([MSBuild]::NormalizeDirectory('$(SharedIntermediateOutputPath)', 'blob-feed'))</SourceBuiltBlobFeedDir>
-    <SourceBuiltPackagesPath>$([MSBuild]::NormalizeDirectory('$(SourceBuiltBlobFeedDir)', 'packages'))</SourceBuiltPackagesPath>
-    <SourceBuiltAssetsDir>$([MSBuild]::NormalizeDirectory('$(SourceBuiltBlobFeedDir)', 'assets'))</SourceBuiltAssetsDir>
+    <IntermediateSymbolsRootDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsObjDir)', 'Symbols'))</IntermediateSymbolsRootDir>


How does the Microsoft build gather/publish symbols? Does it want the individual symbol packages? I ask wondering if the symbols should be places next to the packages they are for?

How does the Microsoft build gather/publish symbols? Does it want the individual symbol packages?

Correct. Symbol packages are placed next to the non symbol packages in MSFT builds. We want symbol packages here as well but they are currently filtered out: dotnet/source-build#4110

src/SourceBuild/content/Directory.Build.props

MichaelSimons · 2024-02-09T15:37:40Z

Because of the scope of these changes please run a full CI run.

src/SourceBuild/content/eng/build.sourcebuild.targets

eng/pipelines/templates/jobs/vmr-build.yml

…nstaller into ArtifactsLocationRevamp

ViktorHofer · 2024-02-10T11:53:37Z

eng/pipelines/vmr-build.yml

+        scope: full
      ${{ else }}:
-        scope: lite
+        scope: full


Temp change to queue a full build

ViktorHofer · 2024-02-12T10:14:37Z

The failures are due to dotnet/source-build#4111. I validated that the Private.* source-build artifacts are produced correctly and with the expected content. Also verified the other pipeline artifacts (packages & logs). Also did offline builds with these changes.

src/SourceBuild/content/Directory.Build.props

MichaelSimons · 2024-02-12T15:30:10Z

The failures are due to dotnet/source-build#4111. Validate that the Private.* source-build artifacts are produced correctly and with the expected content. Also verified the other pipeline artifacts (packages & logs). Also did offline builds with these changes.

Unfortunately the Alpine w/previous SDK leg is the one that has the poison test enabled. Would you mind temporarily enabling poison on the alpine leg with the MSFT Sdk to validate there are no regressions in the poison infra?

eng/pipelines/templates/stages/vmr-build.yml

MichaelSimons · 2024-02-12T17:17:14Z

Looks like there are numerous poison leaks. @NikolaMilosavljevic can you please review the changes to help RCA the Poison infra regressions.

…tnet/installer into ArtifactsLocationRevamp

ViktorHofer · 2024-02-13T10:11:07Z

@MichaelSimons @NikolaMilosavljevic I fixed the leak detection issue and updated the branch with latest main. The remaining failure exists in main as well. This PR is now ready. PTAL.

In comparison, here's a full build from main: https://dnceng.visualstudio.com/internal/_build/results?buildId=2377522&view=results

MichaelSimons · 2024-02-13T15:13:07Z

src/SourceBuild/content/eng/build.sourcebuild.targets

@@ -111,21 +83,22 @@
          Inputs="$(MSBuildProjectFullPath)"
          Outputs="$(BaseIntermediateOutputPath)ReportPoisonUsage.complete" >
    <ItemGroup>
-      <FinalCliTarball Include="$(SharedOutputPath)**/*$(ArchiveExtension)" />
+      <!-- Exclude the Private.SourceBuilt.Artifacts archive from poison usage scan. -->


Would it make sense to move this comment to be directly above the line that is doing the removal?

MichaelSimons · 2024-02-13T15:19:29Z

Merging ahead of PR validation since the last change was to revert the changes to run additional legs for validation purposes.

ViktorHofer requested a review from a team as a code owner February 9, 2024 09:42

ViktorHofer marked this pull request as draft February 9, 2024 09:42

ViktorHofer force-pushed the ArtifactsLocationRevamp branch from 7ad9ae6 to 63d9377 Compare February 9, 2024 10:09

ViktorHofer commented Feb 9, 2024

View reviewed changes

src/SourceBuild/content/repo-projects/installer.proj Outdated Show resolved Hide resolved

ViktorHofer force-pushed the ArtifactsLocationRevamp branch 3 times, most recently from 69692a9 to fbea002 Compare February 9, 2024 10:30

ViktorHofer commented Feb 9, 2024

View reviewed changes

ViktorHofer force-pushed the ArtifactsLocationRevamp branch from fbea002 to 9eb4b9b Compare February 9, 2024 10:40

ViktorHofer force-pushed the ArtifactsLocationRevamp branch from 9eb4b9b to c8400db Compare February 9, 2024 10:48

akoeplinger reviewed Feb 9, 2024

View reviewed changes

src/SourceBuild/content/repo-projects/source-build-reference-packages.proj Outdated Show resolved Hide resolved

src/SourceBuild/content/repo-projects/installer.proj Show resolved Hide resolved

ViktorHofer added 2 commits February 9, 2024 11:40

comment feedback

cf16a69

Update READMEs

4dc3360

dotnet deleted a comment from azure-pipelines bot Feb 9, 2024

Combine CopyRepoArtifacts and EnsurePackageCreated target

ffa342c

MichaelSimons reviewed Feb 9, 2024

View reviewed changes

.devcontainer/vmr-source-build/README.md Outdated Show resolved Hide resolved

MichaelSimons reviewed Feb 9, 2024

View reviewed changes

src/SourceBuild/content/Directory.Build.props Show resolved Hide resolved

PR feedback and fixes

5b6cdd4

NikolaMilosavljevic reviewed Feb 9, 2024

View reviewed changes

src/SourceBuild/content/eng/build.sourcebuild.targets Show resolved Hide resolved

Update vmr-build.yml

161c22d

MichaelSimons reviewed Feb 9, 2024

View reviewed changes

eng/pipelines/templates/jobs/vmr-build.yml Outdated Show resolved Hide resolved

ViktorHofer added 5 commits February 9, 2024 20:58

PR feedback & fix

a83daea

Merge branch 'ArtifactsLocationRevamp' of https://github.com/dotnet/i…

7241ab2

…nstaller into ArtifactsLocationRevamp

Fixes

44e2d4e

Merge branch 'main' into ArtifactsLocationRevamp

7256f5f

Fix wrong package source mapping

79219d9

Merge branch 'ArtifactsLocationRevamp' of https://github.com/dotnet/i…

009d119

…nstaller into ArtifactsLocationRevamp

ViktorHofer marked this pull request as ready for review February 10, 2024 11:51

Update vmr-build.yml

e7c6d8b

ViktorHofer commented Feb 10, 2024

View reviewed changes

MichaelSimons reviewed Feb 12, 2024

View reviewed changes

src/SourceBuild/content/Directory.Build.props Show resolved Hide resolved

Update vmr-build.yml

8ae223c

ViktorHofer commented Feb 12, 2024

View reviewed changes

eng/pipelines/templates/stages/vmr-build.yml Outdated Show resolved Hide resolved

ViktorHofer added 9 commits February 12, 2024 18:03

Exclude Private.SourceBuilt.Artifacts and include nuget packages

c78bdd8

it pusMerge branch 'ArtifactsLocationRevamp' of https://github.com/do…

a1e979b

…tnet/installer into ArtifactsLocationRevamp

Add SBRPs back to poison check

fdf6d59

Build fix

865376b

Fix typo

f371313

Fix exclude

ec19464

Arrgh. Remove differently

99ee8a5

Merge branch 'main' into ArtifactsLocationRevamp

e5ceb3b

Update vmr-build.yml

1f525ad

Merge branch 'main' into ArtifactsLocationRevamp

ade3ba3

MichaelSimons reviewed Feb 13, 2024

View reviewed changes

MichaelSimons approved these changes Feb 13, 2024

View reviewed changes

Update vmr-build.yml

25402f3

MichaelSimons merged commit 94def2e into main Feb 13, 2024
3 of 21 checks passed

MichaelSimons deleted the ArtifactsLocationRevamp branch February 13, 2024 15:19

build-analysis bot mentioned this pull request Feb 13, 2024

Checkout failure: "Git fetch failed with exit code 128" dotnet/arcade#9009

Open

2 tasks

mthalman mentioned this pull request Feb 13, 2024

Isolate package flow between repos #18557

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Put artifacts in well-known locations #18591

Put artifacts in well-known locations #18591

ViktorHofer commented Feb 9, 2024 •

edited

Loading

ViktorHofer Feb 9, 2024

ViktorHofer Feb 9, 2024

MichaelSimons Feb 9, 2024

ViktorHofer Feb 9, 2024

MichaelSimons commented Feb 9, 2024

ViktorHofer Feb 10, 2024

ViktorHofer commented Feb 12, 2024 •

edited

Loading

MichaelSimons commented Feb 12, 2024 •

edited

Loading

MichaelSimons commented Feb 12, 2024

ViktorHofer commented Feb 13, 2024 •

edited

Loading

MichaelSimons Feb 13, 2024

MichaelSimons commented Feb 13, 2024

Put artifacts in well-known locations #18591

Put artifacts in well-known locations #18591

Conversation

ViktorHofer commented Feb 9, 2024 • edited Loading

ViktorHofer Feb 9, 2024

Choose a reason for hiding this comment

ViktorHofer Feb 9, 2024

Choose a reason for hiding this comment

MichaelSimons Feb 9, 2024

Choose a reason for hiding this comment

ViktorHofer Feb 9, 2024

Choose a reason for hiding this comment

MichaelSimons commented Feb 9, 2024

ViktorHofer Feb 10, 2024

Choose a reason for hiding this comment

ViktorHofer commented Feb 12, 2024 • edited Loading

MichaelSimons commented Feb 12, 2024 • edited Loading

MichaelSimons commented Feb 12, 2024

ViktorHofer commented Feb 13, 2024 • edited Loading

MichaelSimons Feb 13, 2024

Choose a reason for hiding this comment

MichaelSimons commented Feb 13, 2024

ViktorHofer commented Feb 9, 2024 •

edited

Loading

ViktorHofer commented Feb 12, 2024 •

edited

Loading

MichaelSimons commented Feb 12, 2024 •

edited

Loading

ViktorHofer commented Feb 13, 2024 •

edited

Loading