Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

.NET Unified Build (Theme) #3926

Open
tkapin opened this issue Jan 5, 2024 · 1 comment
Open

.NET Unified Build (Theme) #3926

tkapin opened this issue Jan 5, 2024 · 1 comment
Assignees
@mmitche @tkapin
Labels
area-unified-build Epic Groups multiple user stories. Can be grouped under a theme.

Comments

@tkapin
Copy link
Member

tkapin commented Jan 5, 2024 •
edited
Loading

Context and Motivation

Currently, the .NET product is organized in multi-repository fashion, with each of its components (runtime, roslyn, SDK, …) living in dedicated git repositories. While this code organization simplifies development of the individual components, construction of the final product requires complex orchestration to flow binary dependencies between the individual product repos, as well as human effort to get a coherent product build. This way of building the .NET product is called Microsoft’s official build.

Source-build, yet another way of building the .NET product, is used by our Linux partners (Red Hat, Canonical) to build and ship .NET in their distros, satisfying strict requirements of building the product without binary dependencies. Maintaining both these methodologies is expensive.

The Unified Build project aims to reduce the costs of the .NET servicing releases as well as the release time by providing a single (virtual) repository that with all the code and assets needed to build the entire product. Focusing on the source-build philosophy (aggregate code base is the source-of-truth) as the way that the community and Microsoft produce the product will drastically improve development operations, servicing, and community interaction.

.NET Unified Build is a theme for the Germanium semester.

Details can be found in the "Unified Build (Theme)" document. The scenarios listed below are pulled from this document and will be represented as linked work items in this project.

Costing

Cost Label Effort Is
S < 4 person weeks
M 4-10 weeks
L 10 – 20 weeks
XL > 20 weeks

Scenarios

  • Microsoft and its partners can develop and build a .NET distribution with minimal orchestration
    • Microsoft can produce a shippable .NET 9 build in (5-6 hours), significantly less than .NET 8 (16 hrs).
    • Release retrospective items classified as “Build” drop by 50% in .NET 9 as compared to .NET 8.
    • A shippable build is produced from a single commit in a single source repo.
  • Developers can efficiently work on the product
    • Cross-stack repository changes can be done in a single commit.
  • Microsoft, its partners, and community developers can validate the complete product
    • Distro partners can decommission their existing smoke tests if they desire, and still have confidence in shipping the product.

Areas

Implement Vertical Build of the .NET product - P1/XL

This effort aims to build the .NET product as a series of vertical builds for each platform from source rather than through a complex graph of binary dependencies resulting from building the individual product repos separately. This build methodology follows the current way of source-building the product, although with less strict requirements on using binaries during the build process.

The Vertical Builds area is the fundamental part of the Unified Build project and by far the largest in scope. It also requires significant expertise in the product repos infrastructure. It's a prerequisite for the main two scenarios of this theme (Building the .NET product with minimal orchestration and Efficient product development).

Source-Build Support for Unified Build - P2/XL

Since the Unified Build aims to adopt the source-build approach to building the .NET product on all platforms, there are certain improvements in the current .NET source-build implementation required to adopt the Unified Build methodology across all supported platforms and provide a good developer experience. Improvements in this area are mainly centered around the "Efficient product development" scenario.

Product Validation - P1/L

Complementary to the individual product repo testing, Product Validation is centered around providing confidence in the final product. The improvements in this area focus on building infrastructure and scenario tests that could be used to validate the installed .NET product.

Product Construction Improvements - P1/XL

Product construction improvements are centered around infrastructure and tooling required to synchronize the code between the individual product repos and the .NET Virtual Monolithic Repository). Similar to the Vertical Builds, improvements in this area are a prerequisite to the main scenarios ("Building the .NET product with minimal orchestration" and "Efficient product development") of this theme.

Release Infrastructure support for the Unified Build - P1/XL

As the Unified Build redefines the methodology of building and assembling the .NET product, the release infrastructure needs to be extended to allow for releases from both the current Microsoft build and the new Unified Build of the .NET product.

  • Release infra investigation & design - Design for changes necessary to enable releases off of the VMR.
  • Identify Repo Dependencies - Identification of the dependencies between product repos and the layout used to stage the product assets for release.
  • Staging / Release Pipeline - Updates to the current release infrastructure, namely the staging and release pipelines to be able to base releases both off the current dependency flow for .NET8 and the VMR for .NET9.
@tkapin tkapin added the Epic Groups multiple user stories. Can be grouped under a theme. label Jan 5, 2024
@tkapin tkapin mentioned this issue Jan 5, 2024
Closed
66 tasks
@tkapin tkapin changed the title [DRAFT] .NET Unified Build (Theme) .NET Unified Build (Theme) Jan 11, 2024
@dotnet-issue-labeler dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@tkapin tkapin transferred this issue from dotnet/arcade-services Jan 11, 2024
@tkapin tkapin removed the untriaged label Jan 11, 2024
@ViktorHofer ViktorHofer mentioned this issue Jan 22, 2024
Merged
ViktorHofer added a commit to dotnet/msbuild that referenced this issue Jan 22, 2024
@ViktorHofer
Condition source-build specific configurations on DotNetBuildFromSouce
93b3da9 
Contributes to VMR work: dotnet/source-build#3926

Changes are from dotnet/dotnet#46

No functional changes introduced.
@ViktorHofer ViktorHofer mentioned this issue Jan 22, 2024
Merged
ViktorHofer added a commit to dotnet/aspnetcore that referenced this issue Jan 22, 2024
@ViktorHofer
Make SourceBuild.props work on Windows
d77955f 
Contributes to VMR work: dotnet/source-build#3926

Changes are from dotnet/dotnet#46
@ViktorHofer ViktorHofer mentioned this issue Jan 22, 2024
Merged
ViktorHofer added a commit to dotnet/aspnetcore that referenced this issue Jan 22, 2024
@ViktorHofer
Make SourceBuild.props work on Windows (#53523)
a278a26 
* Make SourceBuild.props work on Windows

Contributes to VMR work: dotnet/source-build#3926

Changes are from dotnet/dotnet#46

* Update build.ps1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmitche mmitche

@tkapin tkapin

Labels
area-unified-build Epic Groups multiple user stories. Can be grouped under a theme.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MichaelSimons @mmitche @tkapin