TypeLoadException when VSINSTALLDIR is set

[frankracis-work]

Operation System:
Windows

DocFX Version Used:
2.43.1.0
or
from source, commit 9a220bd

Template used: default

Steps to Reproduce:
Set VSINSTALLDIR to the Visual Studio location.
Run docfx metadata on the docfx-seed project (or any project)
or
attempt to run the unit tests from the build.

Expected Behavior:
Metadata is built,
or
Unit tests pass

Actual Behavior:
Metadata is not created and warning is logged:

Build succeeded with warning.
[19-06-21 11:22:27.911]Warning:[ExtractMetadata](C:/temp/docfx-seed/src/SampleClass1/SampleClass1.sln)Error opening solution C:/temp/docfx-seed/src/SampleClass1/SampleClass1.sln: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.. Ignored.
[19-06-21 11:22:27.914]Warning:[ExtractMetadata]No metadata is generated for .

Other projects will have the build fail, with the above warning followed by a NullReferenceException

   at Microsoft.DocAsCode.Metadata.ManagedReference.ExtractMetadataWorker.<SaveAllMembersFromCacheAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.DocAsCode.Metadata.ManagedReference.ExtractMetadataWorker.<ExtractMetadataAsync>d__11.MoveNext()

or
Unit tests fail

Running in the debugger, the assemblies that can not be loaded are

Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Workaround
Unset the VSINSTALLDIR variable. docfx metadata succeeds and unit tests pass.

Analysis
The MSBuild assemblies are not in the docfx folder, and must be resolved with an AssemblyResolve event. This is done by MsBuildLocator:
https://github.com/microsoft/MSBuildLocator/blob/8b5c58f5842f58ebf860ece96ac0f6924df839b1/src/MSBuildLocator/MSBuildLocator.cs#L191

Normally, this would be called in MsBuildEnvironmentScope:

docfx/src/Microsoft.DocAsCode.Metadata.ManagedReference/MsBuildEnvironmentScope.cs

Line 99 in af3345c

MSBuildLocator.RegisterInstance(latest);

However, if VSINSTALLDIR is set, MsBuildEnvironmentScope exits early:

docfx/src/Microsoft.DocAsCode.Metadata.ManagedReference/MsBuildEnvironmentScope.cs

Lines 31 to 34 in af3345c

	if (!string.IsNullOrEmpty(vsInstallDirEnv))
	{
	Logger.LogInfo($"Environment variable {VSInstallDirKey} is set to {vsInstallDirEnv}, it is used as the inner compiler.");
	return null;

When this happens, MsBuildLocator is never called, the AssemblyResolve event handler is not added, and the assemblies are not found. It appears that setting MSBUILD_EXE_PATH will cause the same problem.

It seems like MsBuildLocator.RegisterMSBuildPath or similar should be called when the environment variables are set.

[jpenniman]

I'm having the same issue with DocFx 2.43.2.
Windows 10 1903
Visual Studio 2017 and 2019 installed.

[ssolmer]

I am having the same issue with DocFx 2.44.0.0.
Windows 10 Enterprise 17763
Visual Studio Professional 2017 15.9.9

BTW, this is a result of following the workaround for issue #1969

[sgrassie]

Seeing the same issue with DocFX 2.47.0 and from source 4b6abf5
Visual Studio Professional 16.3.7

The workaround with the VSINSTALLDIR environment variable doesn't appear to work for me.

[superyyrrzz]

@sgrassie That should be another issue. This issue fixes that setting VSINSTALLDIR makes docfx not working properly, while unsetting it can work.

[superyyrrzz]

fixed since 2.47