Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Isolate package flow between repos #18557

Merged
merged 44 commits into from
Feb 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
0ea4e6c
Isolate package flow between repos
mthalman Feb 6, 2024
3878626
Compute transitive dependencies
mthalman Feb 12, 2024
823a8aa
Swap dep graph visualizer with a simple YAML-based representation
mthalman Feb 12, 2024
edebea7
Fix dependency on scenario-tests
mthalman Feb 12, 2024
5b8a5a2
Remove duplicate installer dependency
mthalman Feb 12, 2024
566f613
Delete obsolete installer-deps
mthalman Feb 12, 2024
8ce569e
Add conditional deps specific to source build
mthalman Feb 13, 2024
4611a5e
Merge branch 'main' into sb3608-parallel
mthalman Feb 13, 2024
fe61fb5
Merge branch 'main' into sb3608-parallel
mthalman Feb 16, 2024
d18498a
Updates to integrate with artifact path refactoring
mthalman Feb 16, 2024
6f33ebb
Merge branch 'main' into sb3608-parallel
mthalman Feb 20, 2024
608e3ad
Add back primary dependencies
mthalman Feb 21, 2024
a7a6f79
Move psb references
mthalman Feb 21, 2024
bdaa440
Use Exclude instead of Condition
mthalman Feb 21, 2024
241ad6f
whitespace
mthalman Feb 21, 2024
a50512d
Remove MakeDir for package dirs
mthalman Feb 21, 2024
c97b8d2
Add comment for embedding
mthalman Feb 21, 2024
8b3f3b2
indenting
mthalman Feb 21, 2024
8834743
Remove TransitiveRepositoryReference instead of RepositoryReference, …
mthalman Feb 21, 2024
0540025
Remove SkipEnsurePackagesCreated
mthalman Feb 21, 2024
3c0edf6
Remove SharedOutputPath MakeDir
mthalman Feb 21, 2024
fe82c36
Use Copy instead of ln
mthalman Feb 21, 2024
816e8dc
Separate SB refs into separate ItemGroup
mthalman Feb 21, 2024
4d7ac80
dependency adjustments
mthalman Feb 21, 2024
2be9e44
Renames to remove "SourceBuild"
mthalman Feb 21, 2024
a0117ac
Reclassify fsharp dep on runtime
mthalman Feb 22, 2024
afb34ea
Reclassify msbuild dep on runtime
mthalman Feb 22, 2024
bb5af84
Add runtime dep to templating
mthalman Feb 22, 2024
fe469d7
Remove whitespace
mthalman Feb 22, 2024
308159f
Remove psb dep on scenario-tests
mthalman Feb 22, 2024
19bc3f3
Remove whitespace
mthalman Feb 22, 2024
5f14aa3
Reclassify format dep on symreader
mthalman Feb 22, 2024
80ba493
Use directory normalization
mthalman Feb 22, 2024
7d574e2
Simplify package file collection
mthalman Feb 22, 2024
a0599e1
Add whitespace
mthalman Feb 22, 2024
6e800d7
Simplify _BuildSources
mthalman Feb 22, 2024
59013c6
Get repo refs in parallel
mthalman Feb 22, 2024
28f645a
Use Distinct
mthalman Feb 22, 2024
0dfac72
whitespace
mthalman Feb 22, 2024
d1b1de0
Merge branch 'main' into sb3608-parallel
mthalman Feb 22, 2024
2c3836b
Move windowsdesktop dep from sdk to installer
mthalman Feb 22, 2024
9a25ae6
Revert "Use Distinct"
mthalman Feb 22, 2024
5420993
Remove unnecessary normalize calls
mthalman Feb 22, 2024
2303610
Update src/SourceBuild/content/repo-projects/package-source-build.proj
mthalman Feb 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

5 changes: 5 additions & 0 deletions src/SourceBuild/content/repo-projects/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@
<NuGetConfigFile Condition="'$(OriginalNuGetConfigFile)' != ''">$(BaseIntermediateOutputPath)$([System.IO.Path]::GetFileName('$(OriginalNuGetConfigFile)'))</NuGetConfigFile>

<SourceBuiltSdksDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsDir)', 'source-built-sdks'))</SourceBuiltSdksDir>
<RepoArtifactsShippingPackagesDir Condition="'$(RepositoryName)' != 'source-build-reference-packages'">$([MSBuild]::NormalizeDirectory('$(ArtifactsShippingPackagesDir)', '$(RepositoryName)'))</RepoArtifactsShippingPackagesDir>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love that we have a pretty consistent pattern of comparing '$(RepositoryName)' != 'source-build-reference-packages' in various places throughout Directory.Build.targets and Directory.Build.props.

@jkoritzinsky added a new concept of a reference only repo here that would be good to use after his PR goes in. https://github.com/dotnet/installer/pull/18672/files#diff-0802dca0f4d533e4f44a76e021486aefa13bc1b193f018433b0d056ae84c9f3cR407-R408

<RepoArtifactsShippingPackagesDir Condition="'$(RepositoryName)' == 'source-build-reference-packages'">$(ReferencePackagesDir)</RepoArtifactsShippingPackagesDir>
<RepoArtifactsNonShippingPackagesDir Condition="'$(RepositoryName)' != 'source-build-reference-packages'">$([MSBuild]::NormalizeDirectory('$(ArtifactsNonShippingPackagesDir)', '$(RepositoryName)'))</RepoArtifactsNonShippingPackagesDir>
<RepoArtifactsNonShippingPackagesDir Condition="'$(RepositoryName)' == 'source-build-reference-packages'">$(ReferencePackagesDir)</RepoArtifactsNonShippingPackagesDir>


<!-- Set the bootstrap version to the VMR's version if empty. (no bootstrap set). -->
<ArcadeBootstrapVersion>$([MSBuild]::ValueOrDefault('$(ARCADE_BOOTSTRAP_VERSION)', '$(ArcadeSdkVersion)'))</ArcadeBootstrapVersion>
Expand Down
168 changes: 130 additions & 38 deletions src/SourceBuild/content/repo-projects/Directory.Build.targets

Large diffs are not rendered by default.

13 changes: 9 additions & 4 deletions src/SourceBuild/content/repo-projects/aspnetcore.proj
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,16 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
<RepositoryReference Include="runtime" />
<RepositoryReference Include="msbuild" />
<RepositoryReference Include="roslyn" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="roslyn-analyzers" />
<RepositoryReference Include="runtime" />
<RepositoryReference Include="xdt" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="nuget-client" />
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="source-build-reference-packages" />
<RepositoryReference Include="symreader" />
</ItemGroup>

<ItemGroup>
Expand Down
4 changes: 4 additions & 0 deletions src/SourceBuild/content/repo-projects/cecil.proj
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,8 @@
<RepositoryReference Include="arcade" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
4 changes: 4 additions & 0 deletions src/SourceBuild/content/repo-projects/command-line-api.proj
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,8 @@
<RepositoryReference Include="arcade" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
6 changes: 5 additions & 1 deletion src/SourceBuild/content/repo-projects/deployment-tools.proj
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
<RepositoryReference Include="runtime" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

<ItemGroup>
Expand Down
4 changes: 4 additions & 0 deletions src/SourceBuild/content/repo-projects/diagnostics.proj
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,8 @@
<RepositoryReference Include="arcade" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
38 changes: 4 additions & 34 deletions src/SourceBuild/content/repo-projects/dotnet.proj
Original file line number Diff line number Diff line change
Expand Up @@ -11,42 +11,12 @@
- If we have a repo that is not in sdk's dependency tree, we can still build it by including it here. -->

<ItemGroup>
<!-- Toolsets -->
<RepositoryReference Include="source-build-reference-packages" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
<RepositoryReference Include="arcade" />

<!-- Product Repos -->
<RepositoryReference Include="command-line-api" />
<RepositoryReference Include="sourcelink" />
<RepositoryReference Include="diagnostics" />
<RepositoryReference Include="emsdk" />
<RepositoryReference Include="cecil" />
<RepositoryReference Include="symreader" />
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
<RepositoryReference Include="runtime" />
<RepositoryReference Include="roslyn" />
<RepositoryReference Include="windowsdesktop" Condition="'$(TargetOS)' == 'windows'" />
<RepositoryReference Include="xdt" />
<RepositoryReference Include="msbuild" />
<RepositoryReference Include="roslyn-analyzers" />
<RepositoryReference Include="aspnetcore" />
<RepositoryReference Include="razor" />
<RepositoryReference Include="deployment-tools" />
<RepositoryReference Include="format" />
<RepositoryReference Include="nuget-client" />
<RepositoryReference Include="templating" />
<RepositoryReference Include="test-templates" />
<RepositoryReference Include="fsharp" />
<RepositoryReference Include="vstest" />
<RepositoryReference Include="sdk" />
<RepositoryReference Include="aspire" />
<RepositoryReference Include="installer" />

<!-- Package source-build artifacts -->
<RepositoryReference Include="package-source-build" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />

<!-- Testing. -->
<RepositoryReference Include="scenario-tests" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="package-source-build" />
</ItemGroup>

</Project>
4 changes: 4 additions & 0 deletions src/SourceBuild/content/repo-projects/emsdk.proj
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,8 @@
<RepositoryReference Include="arcade" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
10 changes: 9 additions & 1 deletion src/SourceBuild/content/repo-projects/format.proj
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,17 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="command-line-api" />
<RepositoryReference Include="roslyn" />
<RepositoryReference Include="runtime" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="msbuild" />
<RepositoryReference Include="roslyn-analyzers" />
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="source-build-reference-packages" />
<RepositoryReference Include="symreader" />
</ItemGroup>

</Project>
8 changes: 6 additions & 2 deletions src/SourceBuild/content/repo-projects/fsharp.proj
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,12 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="msbuild" />
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="msbuild" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="runtime" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
11 changes: 10 additions & 1 deletion src/SourceBuild/content/repo-projects/installer.proj
Original file line number Diff line number Diff line change
Expand Up @@ -37,17 +37,26 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
<RepositoryReference Include="aspire" />
<RepositoryReference Include="aspnetcore" />
<RepositoryReference Include="command-line-api" />
<RepositoryReference Include="deployment-tools" />
<RepositoryReference Include="emsdk" />
<RepositoryReference Include="fsharp" />
<RepositoryReference Include="msbuild" />
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
<RepositoryReference Include="nuget-client" />
<RepositoryReference Include="roslyn" />
<RepositoryReference Include="runtime" />
<RepositoryReference Include="sdk" />
<RepositoryReference Include="symreader" />
<RepositoryReference Include="test-templates" />
<RepositoryReference Include="vstest" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="windowsdesktop" Condition="'$(TargetOS)' == 'windows'" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

<!--
Expand Down
6 changes: 5 additions & 1 deletion src/SourceBuild/content/repo-projects/msbuild.proj
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,12 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
<RepositoryReference Include="runtime" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="roslyn" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="runtime" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
4 changes: 2 additions & 2 deletions src/SourceBuild/content/repo-projects/nuget-client.proj
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
<BuildScript>$([MSBuild]::NormalizePath('$(ProjectDirectory)', 'eng', 'source-build', 'build$(ShellExtension)'))</BuildScript>
</PropertyGroup>

<ItemGroup>
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="msbuild" />
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="xdt" />
</ItemGroup>

Expand Down
65 changes: 30 additions & 35 deletions src/SourceBuild/content/repo-projects/package-source-build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -14,56 +14,51 @@
<Target Name="CustomRepoBuild"
AfterTargets="RepoBuild"
DependsOnTargets="CreateBuildInputProps;DetermineSourceBuiltSdkVersion">
<PropertyGroup>
<SourceBuildTarballStagingDir>$([MSBuild]::NormalizeDirectory('$(BaseIntermediateOutputPath)', '$(SourceBuiltArtifactsTarballName)'))</SourceBuildTarballStagingDir>
<SourceBuildReferencePackagesDestination>$([MSBuild]::NormalizeDirectory('$(SourceBuildTarballStagingDir)', 'SourceBuildReferencePackages'))</SourceBuildReferencePackagesDestination>
</PropertyGroup>

<!-- Copy PVP to staging dir in order to package them together. -->
<Copy SourceFiles="$(CurrentSourceBuiltPackageVersionPropsPath)"
DestinationFiles="$(SourceBuildTarballStagingDir)PackageVersions.props" />

<ItemGroup>
<SourceBuildReferencePackagesNupkgFiles Include="$(ReferencePackagesDir)**/*.nupkg" />
</ItemGroup>

<!-- Copy reference packages from ReferencePackagesDir to staging dir. -->
<Copy
SourceFiles="@(SourceBuildReferencePackagesNupkgFiles)"
DestinationFiles="@(SourceBuildReferencePackagesNupkgFiles -> '$(SourceBuildReferencePackagesDestination)%(Filename)%(Extension)')"
Condition="'@(SourceBuildReferencePackagesNupkgFiles)' != ''" />

<ItemGroup>
<ArtifactsPackage Include="$(ArtifactsPackagesDir)**\*.nupkg" />
</ItemGroup>

<!-- Copy packages to staging dir. -->
<Copy SourceFiles="@(ArtifactsPackage)"
DestinationFolder="$(SourceBuildTarballStagingDir)"
Condition="'@(ArtifactsPackage)' != ''" />

<PropertyGroup>
<SourceBuiltTarballName>$(ArtifactsAssetsDir)$(SourceBuiltArtifactsTarballName).$(SourceBuiltSdkVersion).$(TargetRid)$(ArchiveExtension)</SourceBuiltTarballName>
<SourceBuiltVersionFileName>.version</SourceBuiltVersionFileName>
</PropertyGroup>

<!-- Content of the .version file to include in the tarball. Write to staging dir. -->
<!-- Content of the .version file to include in the tarball -->
<ItemGroup>
<VersionFileContent Include="$(RepositoryCommit);$(SourceBuiltSdkVersion)" />
</ItemGroup>

<WriteLinesToFile
File="$(SourceBuildTarballStagingDir)$(SourceBuiltVersionFileName)"
File="$(BaseIntermediateOutputPath)$(SourceBuiltVersionFileName)"
Lines="@(VersionFileContent)"
Overwrite="true" />

<MakeDir Directories="$([System.IO.Path]::GetDirectoryName('$(SourceBuiltTarballName)'))" />
<Exec Command="tar --numeric-owner -czf $(SourceBuiltTarballName) $(SourceBuiltVersionFileName) *.nupkg *.props SourceBuildReferencePackages/"
WorkingDirectory="$(SourceBuildTarballStagingDir)" />
<ItemGroup>
<_AllRepoFiles Include="$(ArtifactsShippingPackagesDir)/**" />
<_AllRepoFiles Include="$(ArtifactsNonShippingPackagesDir)/**" />
<_ReferencePackageFiles Include="$(ReferencePackagesDir)**" />
</ItemGroup>

<Message Importance="High" Text="Packaged source-built artifacts to $(SourceBuiltTarballName)" />
<!-- Create a layout directory for the files that are to be included in the artifacts tarball. Since there are a large number of files, this will use symlinks
instead of copying files to make this execute quickly. -->
<PropertyGroup>
<_SourceBuiltLayoutDir>$([MSBuild]::NormalizeDirectory('$(BaseIntermediateOutputPath)', 'artifacts-layout'))</_SourceBuiltLayoutDir>
<SourceBuildReferencePackagesDestinationDirName>SourceBuildReferencePackages</SourceBuildReferencePackagesDestinationDirName>
</PropertyGroup>
mthalman marked this conversation as resolved.
Show resolved Hide resolved

<Copy SourceFiles="@(_AllRepoFiles)"
DestinationFolder="$(_SourceBuiltLayoutDir)"
UseSymbolicLinksIfPossible="true" />
<Copy SourceFiles="$(BaseIntermediateOutputPath)$(SourceBuiltVersionFileName)"
DestinationFolder="$(_SourceBuiltLayoutDir)"
UseSymbolicLinksIfPossible="true" />
<Copy SourceFiles="$(CurrentSourceBuiltPackageVersionPropsPath)"
DestinationFiles="$(_SourceBuiltLayoutDir)PackageVersions.props"
UseSymbolicLinksIfPossible="true" />
<Copy SourceFiles="@(_ReferencePackageFiles)"
DestinationFolder="$(_SourceBuiltLayoutDir)$(SourceBuildReferencePackagesDestinationDirName)"
UseSymbolicLinksIfPossible="true" />

<RemoveDir Directories="$(SourceBuildTarballStagingDir)" />
<Exec Command="tar --numeric-owner -czhf $(SourceBuiltTarballName) $(SourceBuiltVersionFileName) *"
WorkingDirectory="$(_SourceBuiltLayoutDir)" />

<Message Importance="High" Text="Packaged source-built artifacts to $(SourceBuiltTarballName)" />
</Target>

</Project>
5 changes: 4 additions & 1 deletion src/SourceBuild/content/repo-projects/razor.proj
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
<RepositoryReference Include="runtime" />
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="aspnetcore" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
6 changes: 5 additions & 1 deletion src/SourceBuild/content/repo-projects/roslyn-analyzers.proj
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,12 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="runtime" />
<RepositoryReference Include="roslyn" />
mthalman marked this conversation as resolved.
Show resolved Hide resolved
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

</Project>
7 changes: 5 additions & 2 deletions src/SourceBuild/content/repo-projects/roslyn.proj
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,12 @@

<ItemGroup>
<RepositoryReference Include="arcade" />
<RepositoryReference Include="command-line-api" />
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
</ItemGroup>

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
<RepositoryReference Include="runtime" />
<RepositoryReference Include="source-build-externals" />
<RepositoryReference Include="source-build-reference-packages" />
</ItemGroup>

<ItemGroup>
Expand Down
Loading