Add support for running xUnit tests using .NET Core

[jcansdale]

No description provided.

[jcansdale]

[jcansdale]

TestDriven.NET-4.0.3258_Extraterrestrial_Alpha (1).zip

I haven't updated the release notes yet, but this versions supports running xUnit, NUnit and "Ad hoc" tests using .NET core.

The .NET Core test runner requires that unit tests be inside a Console Application project, but any method/property that doesn't have a test attribute can be executed as an "Ad hoc" test (even if it's inside a Class Library project). Anything returned from the method/property will be expanded on the output window.

Please let me know how you get on and report any issues here. 😃

[jeremydmiller]

@jcansdale,

I'm trying it out on a small solution, but it isn't able to resolve the xunit assembly. Should we be copying the xUnit assemblies to wherever TestDriven.Net is testing it, or something about how files are being copied?

Test 'M:Baseline.Testing.LightweightCacheTests.can_remove' failed: Could not load file or assembly 'xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
    System.IO.FileNotFoundException: Could not load file or assembly 'xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
    File name: 'xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
    at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
    at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
    at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
    at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
    at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent)
    at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeAssembly assembly, RuntimeType caType)
    at System.Reflection.RuntimeAssembly.GetCustomAttributes(Type attributeType, Boolean inherit)
    at System.Attribute.GetCustomAttributes(Assembly element, Type attributeType, Boolean inherit)
    at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit)
    at System.Reflection.CustomAttributeExtensions.GetCustomAttribute[T](Assembly element)
    at TestDriven.TestRunner.TargetFrameworkUtilities.TargetFrameworkAgent.GetTargetFrameworkName(String assemblyPath)
    at TestDriven.TestRunner.TargetFrameworkUtilities.TargetFrameworkAgent.GetTargetFrameworkName(String assemblyPath)
    at TestDriven.TestRunner.TargetFrameworkUtilities.FindTargetFrameworkName(String assemblyPath)
    at TestDriven.NetCore.TestRunner.NetCoreTestRunner.isNetCoreApp(String assemblyFile)
    at TestDriven.NetCore.TestRunner.NetCoreTestRunner.Run(ITestListener testListener, String assemblyFile, String cref)
    at TestDriven.TestRunner.AdaptorTestRunner.Run(ITestListener testListener, ITraceListener traceListener, String assemblyPath, String testPath)
    at TestDriven.TestRunner.ThreadTestRunner.Runner.Run()

    Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
    Running under executable  C:\Program Files (x86)\TestDriven.NET 4\ProcessInvocation86.exe
    --- A detailed error log follows.

    === Pre-bind state information ===
    LOG: DisplayName = xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
     (Fully-specified)
    LOG: Appbase = file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore
    LOG: Initial PrivatePath = NULL
    Calling assembly : Baseline.Testing, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
    ===
    LOG: This bind starts in LoadFrom load context.
    WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
    LOG: Using application configuration file: C:\Program Files (x86)\TestDriven.NET 4\ProcessInvocation86.exe.Config
    LOG: Using host configuration file:
    LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
    LOG: Post-policy reference: xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core.EXE.
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core/xunit.core.EXE.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core.EXE.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core/xunit.core.EXE.

[jeremydmiller]

@jcansdale It does work when I put the xunit assemblies under the /NetCore directory, so there's that.

In my project configuration, I did notice that the xunit assemblies are not being copied into bin/Debug/netcoreapp1.0. They do show up in bin\Debug\net451\win7-x64 if that helps. Is there another setting in the project.json to force it to copy the dependency assemblies there?

[jcansdale]

@jeremydmiller It should just work, you shouldn't have to copy anything! I'll investigate and get back to you. It works on my machine. ;)

[jeremydmiller]

I blame VS/dotnet anyway;)

[jcansdale]

Are you using any assembly level custom attributes? I.e. [assembly: FooBar(....)]

[jeremydmiller]

Other than the normal versioning stuff, just this:

[assembly: CollectionBehavior(DisableTestParallelization = true)]

Which I think we could get away w/ dropping on this project.

[jcansdale]

Ahhhh, that makes sense. Didn't expect that though. I expect it would have worked for most people. Thanks for the bug report!

[jeremydmiller]

A little more feedback, I can trigger tests via the actual test, test classes, and namespaces. It's throwing this against the Assembly or by the option to run all tests in the solution:

Error: 
Error: Unhandled Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
Error:    at TestDriven.NetCore.AdHoc.XUnitNetCoreTestRunner.toArgs(ITestListener testListener, String assemblyFile, String cref)
Error:    at TestDriven.NetCore.AdHoc.XUnitNetCoreTestRunner.Run(ITestListener testListener, String assemblyFile, String cref)
Error:    at TestDriven.NetCore.AdHoc.Program.run(String assemblyFile, String cref)
Error:    at TestDriven.NetCore.AdHoc.Program.Main(String[] args)

[jcansdale]

@jeremydmiller Here's a new build with those issues resolved.

TestDriven.NET-4.0.3264_Extraterrestrial_Alpha.zip

The first issue was to do with how it detects a .NET Core assembly from .NET code. It needed to read the [assembly: TargetFramework(...)] attribute an look for ".NETCoreApp" or ".NETStandard". The issue was it also found your xUnit assembly level attribute, but couldn't resolve the assembly it was defined in. I'm now detecting .NET Core targets in a different way.

The other issue was an oversight. I'd already fixed it in the code that supports NUnit. When you execute all tests in a solution, there will still be some output for non-test projects (I'll fix this later).

Please keep the bug reports coming!

[jcansdale]

@plioi @bradwilson, I thought you might be interested to try the build I posted above...

[jeremydmiller]

@jcansdale I'll pull it down tomorrow and try again

[bradwilson]

@jcansdale This is well timed, as I'm doing some xUnit.net demos next week. 👍

[jcansdale]

@bradwilson Excellent, I appreciate the exposure. I'm trying to breathe some life back into TD.Net and make it useful to people. 😄

I noticed a couple of issues with the error reporting when executed with TD.Net.

1. Stack traces reported with -designtime flag aren't pruned and contain an Assert method stack frame.
2. Failed tests are still being sent to stderr when the -designtime flag is enabled
   (user console out/err is fine).

Could you disable reporting to stderr when the -designtime flag is set and clean up the stack trace?

[jcansdale]

In the last preview version I uploaded, the output when a test failed was a bit of a train wreck. The problem is that xUnit reports the failed test to stdout, stderr and as a test-result message. Ideally this would be resolved at the xUnit runner end, but because versions that do this are already in the wild, I've decided to add a workaround for it at the TestDriven.Net end.

This makes the output a lot more readable. Ctrl+Alt+O will take you to the relevant stack frame, then Enter will take you to the failing assert.

Here is what the above code looks like now:

@bradwilson It would be even better if we could filter out the at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer1 comparer)` stack frame. There must be code that does this, but it isn't being run before the stack trace is passed to the test-result event.

@jeremydmiller I'll let you know when this has been uploaded to the http://testdriven.net site.

[bradwilson]

@jcansdale We've run into a problem with the Alpha build, wherein you seem to assume that all project.json / .xproj-based projects are actually .NET Core projects. The xUnit.net source is a good example where this is not the case: we use .xproj to build desktop and netstandard DLLs only, not .NET core.

[jcansdale]

@bradwilson,

I've been working on support for all framework versions. It will now execute your tests using the same framework they were built for (including quirks mode in .NET 4+).

There are two new commands, Test With > .NET Core and Test With > .NET Framework. They will execute using the first framework of that type in the list. Run Test(s) and Test With > Debugger will always use the first framework in the list.

Here is a fresh build that includes this feature:

TestDriven.NET-4.0.0_Extraterrestrial_Alpha.zip 🚀

Let me know how you get on and if you spot an weirdness!

[bradwilson]

I think maybe you're missing my point.

These are desktop assemblies. They are not .NET core assemblies. There's no reason why we would want to register the .NET Core runner for them, so they do not (and never will) reference dotnet-test-xunit.

[jcansdale]

Ah, your running all tests in a solution? I need to work on when to offer a (hopefully) helpful message and when to ignore the project. It currently treats all projects as potential test projects. 😞

[jcansdale]

There's no reason why we would want to register the .NET Core runner for them, so they do not (and never will) reference dotnet-test-xunit.

Okay, I finally get it. 😉

Could you give this version a try:
#40