Define build dependencies between libraries, runtime, host, installers, native libraries

[ViktorHofer]

Today we have the subset system which sits on top of our build system and allows to only build specific parts of the repository. If a developer working mainly on libraries tries to invoke tests without having the runtime built first, that fails. In an ideal world we would declare dependencies between our components (libraries, runtimes, hosts and installers) with up-to-date checks so that these manual steps aren't necessary and components are kept up-to-date with incremental builds. This might require runtimes and hosts to depend on Ninja.

I imagine that we could compose the runtime pack with all its content purely based on ProjectReferences that would fan out to the individual parts (libraries, host, runtime, native libraries).

cc @dotnet/runtime-infrastructure @jkotas

[ghost]

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

[jkotas]

It means that incremental build of e.g. System.Net.Http would check timestamps of all files in the runtime and lower-level libraries. That is like 10,000 files. How long is it going to take to check time stamps of this many files to make the incremental builds work?

I find myself using --no-dependencies option for inner loop quite often already. The dependency checks take some time and produce a lot build output. Is the idea that you would do the first build without --no-dependencies and then use --no-dependencies for inner loop to make it reasonably fast?

[jkotas]

My preference would be to make dotnet build without extra options work great for the inner loop. I do not see the current scheme where one needs to do build.cmd at top level as significant hassle. I would like to have a bit more granularity for that though - have all the Microsoft.Extensions.* and legacy compat pack libraries to be a separate subset.

[ViktorHofer]

I find myself using --no-dependencies option for inner loop quite often already.

Can you elaborate on what the --no-dependencies option is for? Is that something native specific?

It means that incremental build of e.g. System.Net.Http would check timestamps of all files in the runtime and lower-level libraries. That is like 10,000 files. How long is it going to take to check time stamps of this many files to make the incremental builds work?

Our libraries build dependencies are purely managed, aren't they? The runtime and the host are only required for composing the runtime pack and the shared framework for our testhost. Both are required for packaging and testing the product but not for building the BCL, right?

[jkotas]

Can you elaborate on what the --no-dependencies option is for?

It is dotnet build option.

Compare this:

D:\runtime\src\libraries\System.Runtime.Serialization.Formatters\tests>\runtime\dotnet build System.Runtime.Serialization.Formatters.Tests.csproj
C:\Program Files\dotnet
Microsoft (R) Build Engine version 16.8.0-preview-20451-02+51a1071f8 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  System.Security.Principal.Windows -> D:\runtime\artifacts\bin\System.Security.Principal.Windows\ref\netstandard2.0-Debug\System.Security.Principal.Windows.dll
 ...
100s lines of stuff here
 ...
  System.DirectoryServices -> D:\runtime\artifacts\bin\System.DirectoryServices\net5.0-Windows_NT-Debug\System.DirectoryServices.dll
  System.Drawing.Common -> D:\runtime\artifacts\bin\System.Drawing.Common\net5.0-Windows_NT-Debug\System.Drawing.Common.dll
  System.Diagnostics.EventLog -> D:\runtime\artifacts\bin\System.Diagnostics.EventLog\net5.0-Windows_NT-Debug\System.Diagnostics.EventLog.dll
  System.Security.Cryptography.ProtectedData -> D:\runtime\artifacts\bin\System.Security.Cryptography.ProtectedData\netstandard2.0-Windows_NT-Debug\System.Security.Cryptography.ProtectedData.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-Linux-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-Solaris-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-iOS-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-tvOS-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-illumos-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.DirectoryServices.AccountManagement -> D:\runtime\artifacts\bin\System.DirectoryServices.AccountManagement\net5.0-Windows_NT-Debug\System.DirectoryServices.AccountManagement.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-OSX-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-FreeBSD-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-Windows_NT-Debug\System.Runtime.Serialization.Formatters.Tests.dll

Build succeeded.

vs. this:

D:\runtime\src\libraries\System.Runtime.Serialization.Formatters\tests>dotnet build  System.Runtime.Serialization.Formatters.Tests.csproj --no-dependencies
Microsoft (R) Build Engine version 16.8.0-preview-20451-02+51a1071f8 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net48-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-illumos-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-OSX-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-FreeBSD-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-Solaris-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-Windows_NT-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-tvOS-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-iOS-Debug\System.Runtime.Serialization.Formatters.Tests.dll
  System.Runtime.Serialization.Formatters.Tests -> D:\runtime\artifacts\bin\System.Runtime.Serialization.Formatters.Tests\net5.0-Linux-Debug\System.Runtime.Serialization.Formatters.Tests.dll

Build succeeded.

It is still annoying that it builds 9 versions of System.Runtime.Serialization.Formatters.Tests.dll when I only want one. I did not get bothered enough by this to find the right command line option to workaround that.

[ViktorHofer]

It is still annoying that it builds 9 versions of System.Runtime.Serialization.Formatters.Tests.dll when I only want one. I did not get bothered enough by this to find the right command line option to workaround that.

That would be accomplished by passing in the TargetFramework that you are interested in, ie dotnet build -f net5.0-Linux.

It is dotnet build option.

Awesome, I didn't know that one exists. We should definitely document it as it saves a lot of time when only the library invoked has changed.

System.Runtime.Serialization.Formatters.Tests

It's clear to me why you picked that one, as it has the most dependencies of all the test libraries. We have couple of work items that should make incremental builds faster:

1. Consider collapsing libraries reference and source projects #37961 (perf improvement + less verbose output)
2. Experiment with enabling msbuild static graph build #41975 (incremental perf improvement)

I hope that we can explore even more options to make incremental builds faster to get to a point where --no-dependencies isn't making much of a difference besides for libraries with a ton of dependencies.

[jkotas]

It's clear to me why you picked that one, as it has the most dependencies of all the test libraries.

Right, my point is that once you make the dependencies correctly represented repo-wide, everything is going to look like this - every library is going to have huge dependency graph.

[ViktorHofer]

I would like to have a bit more granularity for that though - have all the Microsoft.Extensions.* and legacy compact pack libraries to be a separate subset.

I think I mentioned on Teams that I plan to enable that but never filed an issue for it: #43118

[ViktorHofer]

Right, my point is that once you make the dependencies correctly represented repo-wide, everything is going to look like this - every library is going to have huge dependency graph.

Right, that's primarily the reason why we are still not representing dependencies for shared framework libraries as ProjectReferences and instead use named references. That said, we know that that would enable scenarios that community members are interested in, i.e. being able to build inside Visual Studio or build an individual library, from a fresh clone.

For libraries we are considering doing that but not before incremental builds are fast enough.

[ericstj]

I'm also concerned with the impact to inner-loop performance.

That is like 10,000 files. How long is it going to take to check time stamps of this many files to make the incremental builds work?

Somehow git status manages to do this in under 1s for the entire repo 🤔

One middle-ground here would be a looser coupling dependency. One that doesn't necessarily support full incremental but would support building an absent dependency.

[agocke]

I think this is a great improvement for people new to the repo, and I hope this could actually be a huge win for inner-loop performance, in that you could eventually only build the stuff you actually need to.

I think rather then preemptively assume we can't do a fast incremental build, we should try it and figure out what the big rocks are. We can wait to merge the change until we're happy with the performance.

[ViktorHofer]

@agocke as we just talked about this in the stand-up, would you mind summarizing your thoughts on this?

[agocke]

Incorporating the higher-level goal and what Jan said, I think the first step is to improve our up-to-date check performance.

Per #47022, building the clr partition after it's already built and nothing has been changed takes roughly 51s on my machine. This appears to be a combination of steps which simply aren't incremental (always rebuilt) and expensive tasks which aren't tracked in cmake at all (high-level build scripts).

I think next steps are to improve this number as much as possible. I've already identified some bigger rocks (crossgen, clretwrc), but as we go we'll need to profile and figure out what's taking so much time.