Reduce the size of the MSBuildWorkspace BuildHost directories

[jasonmalinowski]

As of this writing, this BuildHost directories we deploy as part of MSBuildWorkspace are about 15 MB each, totaling around 30 MB. But the majority of that size is us deploying Microsoft.CodeAnalysis.dll and Microsoft.CodeAnalysis.Workspaces.dll and their dependencies. Of the 15 MB of a single BuildHost, those are at least 10 MB (counting some obvious dependencies).

However, for the most part we're not really using much from those projects. We're using the SourceCodeKind enum from Microsoft.CodeAnalysis.dll and some of our Contract asserts from Workspaces, and not much else. With a few small tweaks (or linked files) we can easily cut these references and save a bunch of space on deployments on disk.

[niltor]

When using packages above version 4.8, buildHost related folders will be generated, which takes up too much space.
My tool uses code analysis. Originally, it would only take up 5mb of space after packaging, but after upgrading, the new buildHost took up a lot of space

I want to know, is there any way to reduce the generated content now?

[jasonmalinowski]

@niltor Which version are you using? We've made some improvements in more recent versions to bring the size back down.

[jasonmalinowski]

(i.e. I think we may be able to close this bug as fixed)

[niltor]

@niltor Which version are you using? We've made some improvements in more recent versions to bring the size back down.

@jasonmalinowski Version 4.9 or 4.10. Now I'm using version 4.8 to avoid this issue!

[jasonmalinowski]

@niltor Looks like the improvements are in 4.11.0-2.final. Give that a shot, the final 4.11 packages should be out in a week or two.

[jasonmalinowski]

As far as what would be left to do:

1. The biggest binary we're deploying is Newtonsoft.Json; the removal there is tracked in Switch the BuildHost to use System.Text.Json #71020.
2. The next biggest is System.Text.Json, but we need some serialization library!
3. The next biggest binary is our actual code, so not much I can do that short of doing evil hacks like making one binary that works for both .NET Core and .NET Framework.
4. The next biggest binary set is the localization resources; it looks like we're only getting resources for System.CommandLine which wouldn't be very valuable to have so we could drop that.
5. Next biggest binary is System.Collections.Immutable at ~250KB. So we could save a half of a megabyte using mutable collections, assuming nothing else is using them which is not a bet I'd be willing to make without investigation.

Beyond that things get much smaller and also much more esoteric to replace. So I'll leave this bug open for the resources removal, but that's going to be about as small as we can get then.

[niltor]

Thank you for what you are doing.

As a user, when using version 4.8, the generated files do not include the two huge directories BuildHost-472 and BuildHost-netcore, and the program can still run normally.

But when using 4.9 or 4.10, there will be these two additional directories, each of which is close to 20mb in size. It is indeed an unacceptable change.

I tried the latest preview version 4.11.0-3.final, and the generated content is much smaller than before, but it is still much larger than 4.8.
Especially I don't need multi-language support, and I only target netcore projects, so BuildHost-472 and other language directories are redundant for me.

I will continue to use the 4.8 version.

[jasonmalinowski]

The size should be down to around 3 MB for each directory now.

[kamronbatman]

I wonder if #73497 is the reason this hangs in azure pipelines windows 2022 CI/CD:

        using var workspace = MSBuildWorkspace.Create();
        workspace.WorkspaceFailed += (_, args) => Console.WriteLine(args.Diagnostic.Message);

        var solution = await workspace.OpenSolutionAsync(solutionPath);

I haven't found a workaround and is happening 4.9+.

[jasonmalinowski]

I'm not aware of issues here. @kamronbatman can you file a new bug with some more details?

[kamronbatman]

I opened an issue here: #75967

[CyrusNajmabadi]

Closing out as complete (as jason says we're down to 3mb).