Summary issue to fix Parallel Execution on Scenario-Level

[SabotageAndi]

What get's executed parallel: multiple Scenarios in one Feature file

This mode of execution doesn't work with SpecFlow at the moment.

Reason:

In the generated code-behind classes we are saving the TestRunner in a static field. The TestRunner is currently "pinned" to a thread. The TestRunner is responsible for executing the Before/After Feature Hooks.
Having this static field is a problem when it is called via multiple threads.

So the whole state management of SpecFlow has to be rethink when we want to change this.
One challenge is, that there are different behaviors in the unit test runner. E.g. NUnit uses the same instance on multiple threads, as the others create an instance per thread.

Supporting Test Runners:

• MSTest
• NUnit
• SpecFlow+ Runner

Known Issues:

• Parallel execution of test scenarios within a feature with NUnit 3 #894
• Issue with injecting ScenarioContext into Before Hooks with multithreading (MsTest) #1647

Open Questions:

• Should we implement support for this at all? It will only work in 2 of 4 Unit Test Runners.
  • If not, can we detect the wrong configuration and provide a good error message?

[JontyMC]

Are the docs incorrect when stating that Specflow+Runner can run scenarios in parallel? Understand this is hard, but I'm now having to decide between either single scenario per feature or not using specflow.

[SabotageAndi]

@JontyMC No, docs are correct. This issue was wrong.
It works with the SpecFlow+ Runner, but only in AppDomain or Process test thread isolation mode.

[nvborisenko]

I believe it's easy implementation. I just tried to play within modifying auto-generated code in nunit provider, and discovered that it's achievable. There is only one problem we should resolve: ITestRunner interface should consider a fact that scenarios can be run in parallel (right now it holds only single Scenario inside). If we resolve this in design level (probably new interface/implementation), then it will be easy to adopt unit generators.

[SabotageAndi]

I am always happy to review PRs.

[VitaliiDolotov]

Hi there, since SpecFlow+ Runner end of support the problem of executing tests within one feature in parallel becomes more urgent. Unfortunately non of the exciting test runners doesn't support scenario level parallelization. Could you please give some update is there any plans to implement such functionality for n/xUnit test runners?

Related #894 and #1535

TEMPORARY SOLUTION
As a quick workaround before running test cases on CI we split all feature files into temporary feature files with one test per feature using powershell script. In this case all tests will be executed in parallels

[JagadeesanV]

Hi there, since SpecFlow+ Runner end of support the problem of executing tests within one feature in parallel becomes more urgent. Unfortunately non of the exciting test runners doesn't support scenario level parallelization. Could you please give some update is there any plans to implement such functionality for n/xUnit test runners?

Related #894 and #1535

TEMPORARY SOLUTION As a quick workaround before running test cases on CI we split all feature files into temporary feature files with one test per feature using powershell script. In this case all tests will be executed in parallels

@VitaliiDolotov - Is it possible to share the powershell script you have created to achieve this. Thanks in advance.

[VitaliiDolotov]

Hi there, since SpecFlow+ Runner end of support the problem of executing tests within one feature in parallel becomes more urgent. Unfortunately non of the exciting test runners doesn't support scenario level parallelization. Could you please give some update is there any plans to implement such functionality for n/xUnit test runners?
Related #894 and #1535
TEMPORARY SOLUTION As a quick workaround before running test cases on CI we split all feature files into temporary feature files with one test per feature using powershell script. In this case all tests will be executed in parallels

@VitaliiDolotov - Is it possible to share the powershell script you have created to achieve this. Thanks in advance.

Hi @JagadeesanV, here is the example of the script. Hope it will help you

$code = @"
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace App
{
public class FeatureFilesSplitter
{
public static string root = @".\SOLUTION_FOLDER";
public static string specFlowFolderPath = Path.Combine(root, "FOLDER_WITH_SPECFLOW_TESTS");
public static string tempDirectoryPath = Path.Combine(root, "TEMP_FOLDER_WITH_SPECFLOW_TESTS");

    public static void Do()
    {
        var allFiles = Directory.GetFiles(specFlowFolderPath, "*.*", SearchOption.AllDirectories);

        // Create temp directory with specflow tests
        Directory.CreateDirectory(tempDirectoryPath);

        foreach (var file in allFiles)
        {
            // Skip all not feature files
            if (!file.EndsWith(".feature"))
                continue;

            var fileLines = FeatureFileLines(file);
            var featureName = fileLines.First();

            // Get background
            var background = new List<string>() { };

            int testLine = 0;
            for (int i = 1; i < fileLines.Count; i++)
            {
                // Collect all lines before first test tags
                if (fileLines[i].TrimStart().StartsWith("@"))
                {
                    testLine = i;
                    break;
                }

                background.Add(fileLines[i]);
            }

            var testLines = new List<string>();
            var isExamples = false;
            var isExamplesHeadersAdded = false;
            for (int i = testLine; i < fileLines.Count; i++)
            {
                // Last line
                if (i + 1 >= fileLines.Count)
                {
                    testLines.Add(fileLines[i]);
                    WriteTest(featureName, background, testLines);
                    break;
                }

                if (fileLines[i].TrimStart().StartsWith("@"))
                {
                    // If previous test was Outline
                    // Then just drop test as it was saved before
                    if (isExamples)
                    {
                        testLines = new List<string>();
                    }
                    // If we have stored test
                    // And this is not the first found test
                    if (testLines.Any())
                    {
                        WriteTest(featureName, background, testLines);
                        testLines = new List<string>();
                    }

                    isExamples = false;
                    isExamplesHeadersAdded = false;
                }

                if (fileLines[i].TrimStart().StartsWith("Examples:"))
                {
                    isExamples = true;
                }

                if (fileLines[i].TrimStart().StartsWith("|") && isExamples)
                {
                    if (isExamplesHeadersAdded)
                    {
                        var examplesTest = new List<string>();
                        examplesTest.AddRange(testLines);
                        examplesTest.Add(fileLines[i]);
                        WriteTest(featureName, background, examplesTest);
                        continue;
                    }

                    // Finished saving Exmaples tests
                    isExamplesHeadersAdded = true;
                }

                testLines.Add(fileLines[i]);
            }
        }

        Directory.Delete(specFlowFolderPath, true);
        Directory.Move(tempDirectoryPath, specFlowFolderPath);
    }
	
	public static List<string> FeatureFileLines(string filePath)
    {
        var allLines = File
             .ReadAllLines(filePath)
             // Do not get comments
             .Where(x => !x.TrimStart().StartsWith("#"))
             .ToList();

        var withoutEmptyLines = new List<string>();

        // Remove all empty lines
        // But leave multiline text
        var isMultiline = false;
        foreach (var line in allLines)
        {
            if (line.TrimStart().StartsWith("\"\"\""))
            {
                isMultiline = !isMultiline;
            }

            if (string.IsNullOrEmpty(line) && !isMultiline)
                continue;

            withoutEmptyLines.Add(line);
        }

        return withoutEmptyLines;
    }
	
	public static void WriteTest(string featureName, List<string> background, List<string> testLines)
	{
        var refinedTestLines = testLines.Where(line => !string.IsNullOrEmpty(line.Trim())).ToList();
        if (!refinedTestLines.Any())
        {
            return;
        }
        var test = new List<string>();
		// Add retries to the tests
        // test.Add("@retry(2)");
        test.Add(string.Format("{0}_{1}", featureName, Guid.NewGuid()));
        test.AddRange(background);
        test.AddRange(testLines);
        File.WriteAllLines(Path.Combine(tempDirectoryPath, string.Format("{0}.feature", Guid.NewGuid())), test);
	}
}

}
"@

Add-Type -TypeDefinition $code -Language CSharp
iex "[App.FeatureFilesSplitter]::Do()"

[ghost]

Any movement on this please? we are stuck between using a supported product with slower execution times or sticking to specrun+ which wont be suitable in the longer term.

[SabotageAndi]

@savagerob No movement currently and planned to do. This would need a major part of SpecFlow changed and we are only a small team. This is out of scope for us at the moment.

[jrod567]

Throwing out my support/need for scenario level parallel execution. This is quickly becoming a problem for us from an execution time perspective.

[jrod567]

Any chance this will make it in the upcoming SpecFlow v4 release? Still have a very large need for this on our end.

[joeschmidtJHA]

This is our team's biggest need. We're currently having to split feature files in to multiple parts just so we can utilize all machines/threads in the Selenium Grid. It's not a very clean way to design a test suite. We need scenarios in a single feature file to be able to run parallel. We're all hoping this will be added in the next release.

[CRohm42]

Yep, would like to see this in SF v4.

[JagadeesanV]

Same from us as well.

[Tiberriver256]

This might be a relevant note for tackling this in xUnit:
https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm

PR to use as reference: #2711

*NOTE: It runs into the same problems described here with state. Included only to show that scenario level parallelization should be doable on all three runners.

[Tiberriver256]

My take on the feature file splitting @VitaliiDolotov introduced. This puts each scenario into its folder so they can keep the same feature name.

https://gist.github.com/Tiberriver256/9ff2bf300873f70ff499255706743201

[sampath-ganesan]

Hi @SabotageAndi, Any update on this?

[SabotageAndi]

@sampath-ganesan Sorry, I can't help you. I don't work for Tricentis and SpecFlow anymore.