Discussion: Overview of scenario-level parallel execution issues

[gasparnagy]

There are a couple of issues related to parallel execution with different test runners.

Many of them related to the scenario-level parallel execution (scenarios or outline examples within the same feature can run in parallel). I made a bit of investigation with v3, here are my results.

xUnit

I think xUnit does not support method-level parallel execution or at least I could not figure out how, therefore the problem does not exist here.

You can run scenarios across features parallel, that works fine. The parallel behavior is enabled by default and can be disabled with:

[assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)]

MsTest V2

MsTest supports Class-level (feature-level) and Method-level (scenario-level) parallelization. It is disabled by default but can be enabled/configured with:

using Microsoft.VisualStudio.TestTools.UnitTesting;
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]

With SpecFlow v2.4 or earlier, none of these are supported, because in the generated code, SpecFlow uses the fixed 0 thread id for all scenarios.

With SpecFlow v3, the Class-level (feature-level) parallel execution is supported and seems to be working.

Enabling Method-level (scenario-level) parallelization causes various errors. MsTest generally creates a new instance of the class for each method execution, however, the generated code:

• stores the testRunner in a static field
• initializes the test runner in the static FeatureSetup method, that is called only once by MsTest -- the test runner will "stick" to the thread that was used to call the FeatureSetup

A solution would be to make testRunner an instance field and call FeatureSetup / FeatureTearDown from the TestInitialize / ScenarioTearDown methods. I have tried that with a manually modified generated code and it seems to be working.
(Of course this kills the meaning of Before/After hooks, so we cannot do this by default...)

NUnit

NUnit has a pretty strange concept for parallel execution that is confusing for people anyway (see e.g. nunit/nunit#2574), because NUnit uses the same test class instance for parallel execution, so you cannot have muted state in the test classes.

There are multiple test scopes possible for parallel execution. ParallelScope.Fixtures provides class(feature)-level and ParallelScope.Children method(scenario)-level parallelization. The scope can be changed with:

[assembly: NUnit.Framework.Parallelizable(NUnit.Framework.ParallelScope.Children)]

SpecFlow v3 supports ParallelScope.Fixtures, but shows errors with ParallelScope.Children. This is because

• we store the testRunner in an instance field (so it will be shared across threads)
• we initialize the test runner in an [OneTimeSetUp] method, that is running on a different thread anyway.

Here the solution could be to get rid of the state and put everything in the test method:

• calculates test runner as TechTalk.SpecFlow.TestRunnerManager.GetTestRunner()
• calls FeatureSetup
• calls TestInitialize
• calls what we usually have in the test methods
• in finally: call ScenarioTearDown
• in finally: call FeatureTearDown

Just like with MsTest, this would kill the meaning of Before/After hooks, so we cannot do this by default...

[SabotageAndi]

Interesting, that NUnit didn't work for you. We have scenarios for it https://github.com/techtalk/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Execution/InAppDomainParallelExecution.feature#L74 and they run through.

[gasparnagy]

@SabotageAndi you are right. my test was wrong at that point. I correct the summary.

[bhushansona]

Hi @gasparnagy , I am trying to apply class level parallelism using MSTestV2. However, whenever I generate .feature.cs file I can not see [assembly: Parallelize(Workers = 4, Scope = ExecutionScope.ClassLevel)] applied to class. My requirement is apply above attribute (using some setting, may be from specflow.json), so that code behind file will be generated with this attribute appended.

[gasparnagy]

@bhushansona You don't need to apply that on class level, you can set it as an assembly-level configuration with the code snippet I have included in the base description.

[bhushansona]

@gasparnagy sorry for not understanding... could you please explore where to make changes in VS 2017. I am using .Net core 2.1 project.

[gasparnagy]

@bhushansona no prob. In the project, add a new C# class file, delete the entire file content and replace it with this two lines.

using Microsoft.VisualStudio.TestTools.UnitTesting;
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]

I have also attached my test project as zip: SpecFlowParallelTest_MsTest_NetCore.zip. In this project, the ParallelConfiguration.cs file does this setting.

[jbeasley]

Hi @gasparnagy I think you need

assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)]

to disable the parallelization behaviour in Xunit

[gasparnagy]

@jbeasley fixed, thx

[Planche95]

So for now, there is no way to run test with scenario-level parallel with any of test runner?

[gotgenes]

I think you need

[assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)]

to disable the parallelization behaviour in Xunit

I am new to C# and .NET Core, let alone SpecFlow. None of our tests are safe to run in parallel, and so we need to disable the parallelism. Can someone please say explicitly where this annotation needs to go? Does it go in a hooks file, a steps file, or a feature file? Does it need to appear in one file, or in many?

[david1995]

Hi @gotgenes, it does not matter in what file you place this attribute. This attribute must be set only once per project / assembly (see CollectionBehaviorAttribute.cs, line 8). I'd recommend you to create a .cs file with the single purpose of having all assembly-level attributes in one place.

[gotgenes]

I'd recommend you to create a .cs file with the single purpose of having all assembly-level attributes in one place.

@david1995 This is excellent advice. Thanks!

[Oliver-S-Zheng]

Hi @gasparnagy Do you have an example of your manually modified generated code for MsTest to get it to run testcases in parallel? Or a sample solution like what you had above but for .Net rather than .Net Core

[markvincze]

Hi @gasparnagy,

I've been using Specflow with xUnit, and I didn't have an explicit Xunit.CollectionBehavior configuration, and I was running my test suite like this:

dotnet vstest ./MyTests.dll --Parallel

Based on what you described above, this should work fine with xUnit, right?

What I see, is that during the test suite execution, I'm randomly getting warnings in the output saying either

-> #12: warning: The previous FeatureContext was already disposed.

or

 -> warning: The previous FeatureContext was not disposed.
 -> warning: The previous ScenarioContext was not disposed.

Do you think this can be caused by the parallel execution?

[NaZaRKIN123]

Hi,
Any plans to fix it?

[psphanireddy]

Hi, Any update on using NUnit specflow for scenarios parallelism within same feature..?

[LirazShay]

Hi,
NUnit issue was resolved in the latest version (3.13): nunit/nunit#2574
So we can use FixtureLifeCycle - LifeCycle.InstancePerTestCase in assemby level
https://docs.nunit.org/articles/nunit/writing-tests/attributes/fixturelifecycle.html
So I tried the combination of the two:
[assembly: Parallelizable(ParallelScope.Children)]
[assembly: FixtureLifeCycle(LifeCycle.InstancePerTestCase)]
And I still got errors
So I guess that the issues that related to mstest will happen also hear
@gasparnagy Can you please the description in the nunit part?
@SabotageAndi Can I open new issue to support nunit parallel execution in scenario level?
SpecFlowParallelScenariosRunTests.zip

[SabotageAndi]

@LirazShay There is already one open at #2225
Adding support for scenario level parallel execution is not an easy fix. For that a lot of internals of SpecFlow has to be changed, because of the way currently state is managed. Some details are in the linked issue above.