MSBuild 15 Sdk Design

[AndyGerlicher]

MSBuild will allow third-party-defined and dynamically-delivered extensions of the build process via the new “Sdk” concept. This will extend the experience delivered with RC.2 to include an acquisition process.

Changes to project files

An Sdk can be consumed in two ways: with implicit top and bottom imports at through the Sdk attribute on the Project element:

<Project Sdk="Microsoft.NET.Sdk">
...
</Project>

and through a modified <Import> element with the Sdk attribute:

<Project>
  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/1.0.0" />
...
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk/1.0.0" />
</Project>

The former mechanism is syntactic sugar for the latter, with an Sdk.props import at the top of the project and an Sdk.targets import at its bottom.

User experience

When a user opens an Sdk-using project in an IDE, the IDE will try to evaluate the project. MSBuild will coordinate with the IDE to fetch and integrate all required Sdks. The IDE will probably wish to display a progress bar for any required downloads.

On the command line, a dotnet restore or msbuild.exe /t:Restore invocation will fetch required Sdks before proceeding with the restore process, ensuring that only one user gesture is required to bring the project to a buildable state.

Sdk acquisition

How does MSBuild know how to acquire an Sdk when it needs to? Through a provider model. A caller of the MSBuild APIs can provide an instance of a new interface ISdkResolver which will be used to map the XML from above to files on the local filesystem.

Note: Although the acquisition could differ between builds the resolved SDK would be identical.

struct SdkReference
{
    public string Name;
    public Version? Version;
}

interface ISdkResolver
{
    public Dictionary<SdkReference, Path> Resolve(IEnumerable<SdkReference>);
}

We expect most ISdkResolver implementations to additionally have affordances for reporting progress back to a UI and to log using the MSBuild log interface.

🚧 Exposing logging may require an interface change here.

🚧 Do we need to pass path-to-project, so that something NuGet-like can walk up to find its feed configuration?

MSBuild evaluation changes

MSBuild will collect Sdks needed for a given project early in its evaluation (before pass 1 expands imports). It will then unify Sdks specified within the project with those specified in the (optional) lineup file. This produces a list of Sdks that is then passed to ISdkResolver.Resolve(), producing a lookup table for Sdk imports.

Evaluator.GetCurrentDirectoryForConditionEvaluation should return the Sdk directory if there is an Sdk attribute on an import, so that you can have something like Condition="Exists('Sdk\Sdk.props')" on an Sdk import.

Evaluation pass 1 (properties and imports) then continues as usual. When considering an Import element with an Sdk attribute, the specified Project attribute is treated as a relative path from the base Sdk path looked up in the table generated in the new pre-pass-1 step.

Lineups

After growing past being “small” projects, most repos/solutions will want a simple way to manage the set of Sdks they use--for instance, to unify versions of Sdks available for projects within the repo. This will be accomplished by a lineup file, format TBD, that MSBuild will consider when putting together the list of available Sdks.

🚧 What does this look like on disk?

We expect a few common patterns of use:

• Small-scale “demo projects”
  • Consist of ~1 project.
  • Do not have a lineup file.
  • Specify Sdk version for any “non-core” Sdk referenced.
• Larger-scale projects
  • Have many projects in a solution.
  • Specify a lineup
  • Do not specify versions in the project files themselves, only in the lineup.

Project Load

We'll need to add to ProjectLoadSettings so a user can opt out of SDK resolution. They would need to specify both IgnoreMissingImports and a new DoNotResolveSdks if they wanted to open a project without getting errors. But this would allow minimal property evaluation without resolving anything.

Concrete work items

• Get feedback from VS folks
  • Especially around UI needs for status updating.
• Expose ISdkResolver definition for prototyping.
• Implement Sdk-aware imports (Import from SDKs #1400)
• Implement Sdk-gathering (pass 0.5)
• Augment build-starting methods to accept an ISdkResolver
  • In BuildParameters? BuildRequestData?
• Minimal FilesystemSdkResolver for prototyping and testing
  • Maybe looks in NuGet cache folder? That'd get us a long way toward seeing what the once-everything-is-downloaded scenario looks like.
• Implement NuGetSdkResolver: Proposal: implement a SdkResolver for NuGet packages NuGet/Home#5220
  • Straw man implementation idea: write sdk names + package versions to a minimal project file, restore it out of proc, return paths 🚧 magically.
• Use resolvers in various scenarios
  • Change MSBuild.exe to use a resolver
  • Change dotnet CLI to invoke MSBuild with a resolver
  • Change VS to invoke MSBuild with a resolver
  • Change VS Code to invoke MSBuild with a resolver
  • Change VS Mac to invoke MSBuild with a resolver
  • 🚧 Some of those could be helped by changing the default, if blocking evaluation without progress reporting is ok. Probably is for OmniSharp (out of proc) and CLI (synchronous command line build). UIs probably want advanced progress reporting + cancellability.

[maartenba]

How does this play with VS Code, JetBrains Rider, Xamarin, MonoDevelop, ...?

[jeffkl]

@jviau We're designing a way for a UI to provide to MSBuild some logic that resolves an SDK. The implication for VS is that it would want to present the user with a progress dialog while it restores packages during project load. Our design here means that VS would specify a class to MSBuild that we would create an instance of and call Resolve. If a user loaded a solution with multiple projects, then Resolve would be called multiple times. Would this work okay with VS? Can you review the design here and give any feedback?

[jeffkl]

@maartenba

How does this play with VS Code, JetBrains Rider, Xamarin, MonoDevelop, ...?

We have items near the bottom to involve VS Code, VS Mac and dotnet CLI to ensure they can pass a resolver to give the user a good experience.

JetBrains Rider and MonoDevelop will need to implement the interface for displaying a progress dialog to the user. I'm fairly certain they already have a custom MSBuild logger to handle log output like Visual Studio so we're hoping it will be easy for IDEs to make this happen for the SDK resolution. We can probably pave the way with the ones we ship in VS that they can re-use or take as an example.

[jeffkl]

@AndyGerlicher What do you think about making this more generic and extensible? Right now it's specifically for resolving SDKs but do you think we'll ever want to resolve other project assets? My thought was to have the logic be more centered around resolving assets, one of which would be an SDK. Something like:

public interface AssetResolver
{
    IEnumerable<IResolvedAsset> Resolve(IEnumerable<IAsset> assets);
}

public enum AssetType
{
    None,
    SDK,
}

public interface IAsset
{
    AssetType AssetType { get; }
    string Name { get; }
    Version Version { get; }
}

public interface IResolvedAsset : IAsset
{
    /// <summary>
    /// The path to the resolved asset.
    /// </summary>
    string Path { get; }
}

Thoughts?

[jeffkl]

I also looked at logging and it's going to be tricky. The only thing available during project evaluation is the internal LoggingService so we'd need a new interface to expose to the caller or allow them to throw exceptions. If we want them to be able to log messages and warnings we'll need to come up with something.

[Mike-E-angelo]

This sounds like a very intriguing and valuable feature!

a lineup file, format TBD

Please consider making this an XML file, to keep it consistent with the current ecosystem. It would be confusing to introduce differing and inconsistent data formats, especially after moving from XML to JSON and then back to XML again.

The spirit of the request here isn't to favor a particular format, but rather to favor consistency. I might be alone on this, however. But to me, exploring a given a solution (VS or otherwise) and seeing two different data files in two different formats when they both simply describe data is not only maddeningly inconsistent, but inefficient as well. That means you ultimately have two libraries/concerns in your solution that parse/save data, when it could be done with just one.

Of course, if you provide a strictly POCO-based model whereby the end user could describe their data in the format they prefer, then this is a non-issue. 😄 What strikes me about this feature is that the same sort of magic could eventually be used to download/install data-format preferences/serialization components as well -- something that has been discussed in #613.

[maartenba]

@jeffki Is he resolver ever an IDE-specific extension or is the IDE's only role to call into msbuild and show progress?

[jviau]

@jeffkl I am a bit confused by the VS UI interaction. Is VS supposed to implement its own ISdkResolver that performs both progress UI and the sdk resolution? Or are you going to have a VS-agnostic implementation that performs only sdk resolution (then it can be re-used for the command line) and then CPS hooks up to events or something and is responsible for displaying the progress UI?

[jeffkl]

@jviau Yes Visual Studio's implementation of ISdkResolver would do the resolution and the UI. We didn't think much of the resolvers could be re-used because Visual Studio might use the NuGet object model since it has NuGet assemblies while dotnet CLI might just run NuGet.exe and pipe it's output. Does this sound doable for VS?

[jeffkl]

@maartenba The IDE will new up a Project instance for evaluation properties and items and when it does, MSBuild will call the resolvers to acquire SDKs. The project cannot be evaluated until the SDK exists on disk. From the command-line, you'll see a log message that MSBuild is acquiring SDKs but we want the UIs to be able to present some progress to the user. We were assuming that IDEs are already using custom loggers to get info from MSBuild to present to the user so we wanted to follow that same design for SDK resolvers.

[jviau]

@jeffkl will this be a VS-wide implementation, or is it provided per ProjectCollection, ProjectRootElement, or Project? The design above says MSBuild will construct the object, but I think CPS would prefer to construct it and pass it to MSBuild - can we have that behavior available?

[maartenba]

@jeffki so looking at the various responses the IDE would be responsible for ánd logging ánd resolving SDK's? Does that also imply a build server should perform both these tasks?

[jeffkl]

@jviau We think resolvers would be like loggers so they would have a wide scope like ProjectCollection level or build episode. I think VS should add the resolver to the project collection it uses so the same one is re-used.

MSBuild will be passed an instance of the resolver and call it to resolve any unresolved SDKs. So yes CPS would construct it.

[jeffkl]

@maartenba For command-line based builds, MSBuild will ship with a resolver that logs to the console. So build servers will not need any resolvers. The main issue is that UI applications would want to pop up their customer dialogs and may have their own way of acquiring SDKs.

[shalupov]

@jeffkl I'm concerned that IDE will be able to handle SDK resolving. That way MSBuild will have a different behaviour depending on environment it was started in. It may lead to subtle differences between builds inside IDE and on a build server.

[jeffkl]

@shalupov I'll update the design to note that although different resolvers could run, the full closure of what they resolve would be identical. In the case of an SDK, it's just a NuGet package so when building from the command-line the same NuGet package would be used as the one when building from an IDE. The differences would be in how the end-user application acquires the package (uses its own copy of NuGet, downloads NuGet, uses just the web protocols to get the file) and how it presents progress (a progress dialog or logging to the console).

[eerhardt]

Lineups

@AndyGerlicher can you explain how these files are different than using a .props file to share version numbers across all projects in a repo? For example, in the https://github.com/dotnet/sdk, we have a DependencyVersions.props file that specifies the versions of our dependencies: https://github.com/dotnet/sdk/blob/master/build/DependencyVersions.props. So to update the whole repo to a new NuGet version, we just change a single version number in a single file.

Or are lineups only for managing Sdk versions and not other versions?

[magol]

Will this changes allso apply to c++ project?

[davkean]

From API perspective, you shouldn't be returning a concrete Dictionary<TKey, TValue>, instead, IReadOnlyDictionary<TKey, TValue>.

[dazinator]

Just putting forth an idea for the "lineup" mechanism.

You could make projects in a solution share the same lineup perhaps using the SDK attribute:

Project A:

<Project Sdk="Local">
</Project>

Project B:

<Project Sdk="Local">
</Project>

Then in a well known directory, relative to the project / solution dir, you'd have a matching Local.Targets file for msbuild to discover.

This could use the explicit or implicit form:

<Project>
 <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/1.0.0" /> 
<Import Project="Foo.props" Sdk="Foo/1.0.0" /> 
<Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk/1.0.0" /> 
<Import Project="Foo.targets" Sdk="Foo.Sdk/1.0.0" />
 </Project>

This would act as the lineup file, i.e you can manage sdk version in one place.

If msbuild couldn't find a matching targets file, then it could just attempt to resolve the SDK using normal resolution.

If you needed to special case some projects in your giant solution, to use a different version of some of the SDK's, for those projects you could:

<Project Sdk="Another">
</Project>

And then you add Another.targets alongside your Local.targets..

<Project>
 <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/1.0.1" /> 
<Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk/1.0.1" /> 
 </Project>

[kzu]

@dazinator wouldn't that be exactly the same as the Directory.Build.props/Directory.Build.targets support that's already built-in?

[rainersigwald]

Note on lineups: in addition to version unification, it might be helpful/required to be able to specify what resolver should be used to resolve a specified SDK.

[mkarpuk]

Is it still supposed to keep ability to extend MSBuild process using properties like ResolveReferencesDependsOn? Should I report an issue or is this expected? After reviewing #1392 I was able to extend build process by overriding target AfterResolveReferences using explicit references:

  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk.Web" />

  <Target Name="AfterResolveReferences">
    <Message Text="It Works!" Importance="high"/>
  </Target>

However any changes in the property group didn't make any difference
<ResolveReferencesDependsOn>$(ResolveReferencesDependsOn);ChangesDoesNoteTakeEffect</ResolveReferencesDependsOn>

[rainersigwald]

@mkarpuk Physically where in the project file did you define that property? The defaults are set in Microsoft.Common.CurrentVersion.targets, which is imported through the final import, so you'd have to do the change after an explicit import, much like you did while overriding the target.

Is there a reason you don't want to inject your property using BeforeTargets? I would write your example as

  <Target Name="EmitMessageBeforeResolveReferences"
          BeforeTargets="ResolveReferences">
    <Message Text="It Works!" Importance="high"/>
  </Target>

And then it could be defined anywhere in the file.

[mkarpuk]

I put ResolveReferencesDependsOn in the beginning of the file. Thank you! I didn't realized that property was defined in targets, not props.
BeforeTargets also works, I have just converted from .xproj to new .csproj and explore new format. Everything was nice and clear except for property-based overrides which made me thinking that I don't understand something. Now you solved this, thanks again!

[rainersigwald]

#2850 implemented the NuGet parts of this, and the local-filesystem parts have been long completed (like, .NET Core SDK 1.0).

@jeffkl I think we can close this as completed unless there's outstanding work on the progress-indication stuff.

[tmat]

@rainersigwald I'd like to start using custom SDKs in our repos. Is there a spec (or a sample) I can look at to see how this works end-to-end (covering lineups etc.)?

[jeffkl]

I agree, its time to put a "fork" in this one

[jeffkl]

@tmat please see https://github.com/MicrosoftDocs/visualstudio-docs/blob/master/docs/msbuild/how-to-use-project-sdk.md for some documentation, feedback welcome!

[tmat]

@jeffkl Thanks! This feature has a potential to significantly simplify our repos. Looking forward testing it out.