Starting CLI Commands no longer works via the dotnet-runtime-host ".exe"

[matthid]

Since 0,14 and the new build system it seems like you can no longer start other referenced cli projects via Process.Start in BenchmarkDotNet.

Background: It seems like the embedded dll is always the one from BenchmarkDotNet now.

This is a COREHOST_TRACE=1 capture of the started project

Tracing enabled @ Wed Nov 13 14:41:27 2024 GMT
--- Invoked apphost [version: 8.0.11 @Commit: 9cb3b725e3ad2b57ddc9fb2dd48d2d170563a8f5] main = {
C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\bin\Release\net8.0-Windows7.0\OtherProject.exe
config=C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\bin\Release\net8.0-Windows7.0\epsSingle.config
}
Redirecting errors to custom writer.
The managed DLL bound to this executable is: 'f4d0ff40-a7b1-4376-b14d-87ef1f70a014.dll'

The problem is it should be 'OtherProject.dll' instead since that is a completely other project.

The problem only materializes when you use a command line which can not be parsed by the BDN dll and your output will be:

// BeforeAnythingElse

System.FormatException: The input string 'config=C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\bin\Release\net8.0-Windows7.0\epsSingle.config' was not in a correct format.
   at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
   at System.Int32.Parse(String s)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\f4d0ff40-a7b1-4376-b14d-87ef1f70a014.notcs:line 45
// AfterAll

Instead of running the correct entry point.

Workaround:

we now start dotnet <OtherProject.dll> instead but it was quite the pain to find and debug this problem, since if you open the .exe or the .dll in DotPeek everything looks nice, and if you don't have actual parameters it will forward to the 'correct' entrypoint.

[timcassell]

@matthid I don't really follow your usage explanation. Could you share a repro project?

[matthid]

Sure: https://github.com/matthid/BNDIssue2664 here is a repro to reproduce the issue. Like I said it basically is just Process.Start with the .exe of an unrelated project.

[timcassell]

I see the exe in the binary output directory as expected. However, when I run it directly, it prints:

Details

// Benchmark Process Environment Information:
// BenchmarkDotNet v0.13.13-develop (2024-11-22)
// Runtime=.NET 8.0.10 (8.0.1024.46610), X64 RyuJIT SSE3
// GC=Concurrent Workstation
// HardwareIntrinsics=SSE3,LZCNT VectorSize=128
// Job: DefaultJob

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.InvalidOperationException: External process failed (-1): windir: C:\WINDOWS
LOCALAPPDATA: C:\Users\Tim\AppData\Local
CommonProgramFiles: C:\Program Files\Common Files
PROCESSOR_REVISION: 0a00
FP_NO_HOST_CHECK: NO
OS: Windows_NT
PROCESSOR_ARCHITECTURE: AMD64
SystemDrive: C:
TMP: C:\Users\Tim\AppData\Local\Temp
LOGONSERVER: \\TIM-PC
OneDrive: C:\Users\Tim\OneDrive
USERDOMAIN: Tim-PC
ProgramFiles: C:\Program Files
ProgramW6432: C:\Program Files
HOMEPATH: \Users\Tim
USERDOMAIN_ROAMINGPROFILE: Tim-PC
DriverData: C:\Windows\System32\Drivers\DriverData
ProgramData: C:\ProgramData
SESSIONNAME: Console
CommonProgramW6432: C:\Program Files\Common Files
USERPROFILE: C:\Users\Tim
PROCESSOR_LEVEL: 16
ADSK_3DSMAX_x64_2018: C:\Program Files\Autodesk\3ds Max 2018\
GIT_LFS_PATH: C:\Program Files\Git LFS
PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL
PUBLIC: C:\Users\Public
CommonProgramFiles(x86): C:\Program Files (x86)\Common Files
TEMP: C:\Users\Tim\AppData\Local\Temp
ComSpec: C:\WINDOWS\system32\cmd.exe
SystemRoot: C:\WINDOWS
DEPOT_TOOLS_WIN_TOOLCHAIN: 0
ProgramFiles(x86): C:\Program Files (x86)
OneDriveConsumer: C:\Users\Tim\OneDrive
ChocolateyLastPathUpdate: 133254941067942229
USERNAME: Tim
COMPUTERNAME: TIM-PC
APPDATA: C:\Users\Tim\AppData\Roaming
PROCESSOR_IDENTIFIER: AMD64 Family 16 Model 10 Stepping 0, AuthenticAMD
ALLUSERSPROFILE: C:\ProgramData
HOMEDRIVE: C:
NUMBER_OF_PROCESSORS: 6
ChocolateyInstall: C:\ProgramData\chocolatey
PSModulePath: C:\Users\Tim\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\
GTK_BASEPATH: C:\Program Files (x86)\GtkSharp\2.12\
Path: C:\Users\Tim\source\depot_tools;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Program Files\Common Files\Autodesk Shared\;C:\Program Files (x86)\Autodesk\Backburner\;C:\Program Files\Git LFS;C:\Program Files\Mono\bin;C:\Program Files (x86)\GtkSharp\2.12\bin;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Users\Tim\.dotnet\tools;C:\Program Files (x86)\NUnit.org\nunit-console;C:\Python27\Scripts;C:\Program Files\Perforce\;C:\Program Files (x86)\dotnet-core-uninstall\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;C:\Program Files\CMake\bin;C:\Users\Tim\.jsvu\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Mono\bin;C:\Users\Tim\.dotnet\tools;C:\Program Files (x86)\NUnit.org\nunit-console;C:\Users\Tim\AppData\Local\Microsoft\WindowsApps;C:\Users\Tim\AppData\Roaming\npm;C:\Users\Tim\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Tim\.dotnet\tools;C:\Users\Tim\.dotnet\tools;C:\Users\Tim\.dotnet\tools;C:\Users\Tim\.jsvu\bin;C:\Users\Tim\.dotnet\tools
OUT:// BeforeAnythingElse
OUT:
OUT:System.FormatException: The input string 'config=test.config' was not in a correct format.
OUT:   at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
OUT:   at System.Int32.Parse(String s)
OUT:   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\bin\Release\net8.0\d2dbb57d-d694-4e59-9f73-9f7e5480c263\d2dbb57d-d694-4e59-9f73-9f7e5480c263.notcs:line 45
OUT:// AfterAll

   at Benchmark.TestBenchmarkWithExternalProc.GlobalSetup() in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\TestBenchmarkWithExternalProc.cs:line 57
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters) in C:\BenchmarkDotNet\src\BenchmarkDotNet\Engines\EngineFactory.cs:line 28
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName) in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\bin\Release\net8.0\d2dbb57d-d694-4e59-9f73-9f7e5480c263\d2dbb57d-d694-4e59-9f73-9f7e5480c263.notcs:line 176
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\bin\Release\net8.0\d2dbb57d-d694-4e59-9f73-9f7e5480c263\d2dbb57d-d694-4e59-9f73-9f7e5480c263.notcs:line 57
// AfterAll

It looks like it's being overwritten with the benchmark exe. So it's trying to process spawn itself recursively. And since it's actually the benchmark code running instead of the expected process, it doesn't recognize the arguments and throws.

@rainersigwald Any idea why passing the output path build arguments would cause this?

[matthid]

Just to add something which might be a hint:
Without any parameter it actually starts the correct program (you can add a Console.WriteLine to see the correct output). So the generated code later starts the correct dll, so one workaround/hack might be to change what BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached does.

But indeed the correct way would be to have it emit a valid exe host with the correct dll binding (as you can see in my trace output above)