Bugfix/fix reqnroll.verify parallelization (#255)

* Allow Reqnroll.Verify to run in parallel - Allow Reqnroll.Verify to run in parallel by registering a per-scenario VerifySettings - Added documentation on how to use Reqnroll.Verify. * Verify the code works when tests are executed in parallel * revert some of the formatting changes done to available-plugins.md * - used CustomizeScenarioDependencies in VerifyRuntimePlugin - Removed example for using verify in a single-threaded context as we would like users to not use that approach.
reqnroll · Sep 17, 2024 · 03ecd34 · 03ecd34
1 parent b8988c7
commit 03ecd34
Show file tree

Hide file tree

Showing 17 changed files with 220 additions and 42 deletions.
diff --git a/.gitignore b/.gitignore
@@ -221,7 +221,7 @@ ClientBin/
 *.publishsettings
 orleans.codegen.cs
 
-# Including strong name files can present a security risk 
+# Including strong name files can present a security risk
 # (https://github.com/github/gitignore/pull/2483#issue-259490424)
 #*.snk
 
@@ -315,7 +315,7 @@ __pycache__/
 # OpenCover UI analysis results
 OpenCover/
 
-# Azure Stream Analytics local run output 
+# Azure Stream Analytics local run output
 ASALocalRun/
 
 # MSBuild Binary and Structured Log
@@ -384,3 +384,5 @@ docs/_build/*
 /docs/Scripts/
 /docs/pyvenv.cfg
 nCrunchTemp*.csproj
+
+*.received.*
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,8 +3,9 @@
 ## Bug fixes:
 * Modified VersionInfo class to force it to pull version information from the Reqnroll assembly
 * Fix: Reqnroll.CustomPlugin NuGet package has a version mismatch for the System.CodeDom dependency (#244)
+* Reqnroll.Verify fails to run parallel tests determinately (#254). See our [verify documentation](docs/integrations/verify.md) on how to set up your test code to enable parallel testing.
 
-*Contributors of this release (in alphabetical order):* @clrudolphi, @UL-ChrisGlew
+*Contributors of this release (in alphabetical order):* @ajeckmans, @clrudolphi, @UL-ChrisGlew
 
 # v2.1.0 - 2024-08-30
 

diff --git a/...s/Reqnroll.Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/Features/Feature.feature b/...s/Reqnroll.Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/Features/Feature.feature
@@ -1,14 +1,28 @@
+@DoNotParallelize
 Feature: Verify Test
 
-Scenario: Check if Verify is working
-	When I try Verify with Reqnroll
-	Then it works
-
-Scenario Outline: Check if Verify is working with Example Tables
-	When I try Verify with Reqnroll for Parameter '<Parameter>'
-	Then it works
-
-Examples: 
-	| Parameter |
-	| 1         |
-	| 2         |
+    Scenario: Check if Verify is working
+        When I try Verify with Reqnroll
+        Then it works
+
+    Scenario Outline: Check if Verify is working with Example Tables
+        When I try Verify with Reqnroll for Parameter '<Parameter>'
+        Then it works
+
+        Examples:
+          | Parameter |
+          | 1         |
+          | 2         |
+
+    Scenario: Check if Verify is working with multiple scenario parameters
+        When I try Verify with Reqnroll for Parameter '<Parameter>' and some additional parameter '<Additional Parameter>'
+        Then it works
+
+    Examples:
+      | Parameter | Additional Parameter |
+      | 1         | a                    |
+      | 2         | b                    |
+
+    Scenario: Check if Verify is working with global registered path info
+        When I try Verify with Reqnroll with global registered path info
+        Then it works
diff --git a/...Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/Features/Parallel Feature 1.feature b/...Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/Features/Parallel Feature 1.feature
@@ -0,0 +1,6 @@
+Feature: Verify Parallel feature #1
+
+    Scenario: Check if Verify uses the correct paths when ran in parallel 1
+        When I try Verify with Reqnroll in parallel
+        Then it works in parallel with contents `Verify Parallel feature #1`
+        And the verified file is `Verify Parallel feature #1.Check if Verify uses the correct paths when ran in parallel 1.verified.txt` with contents `Verify Parallel feature #1`
diff --git a/...Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/Features/Parallel Feature 2.feature b/...Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/Features/Parallel Feature 2.feature
@@ -0,0 +1,6 @@
+Feature: Verify Parallel feature #2
+
+    Scenario: Check if Verify uses the correct paths when ran in parallel 2
+        When I try Verify with Reqnroll in parallel
+        Then it works in parallel with contents `Verify Parallel feature #2`
+        And the verified file is `Verify Parallel feature #2.Check if Verify uses the correct paths when ran in parallel 2.verified.txt` with contents `Verify Parallel feature #2`
diff --git a/...lel feature #1.Check if Verify uses the correct paths when ran in parallel 1.verified.txt b/...lel feature #1.Check if Verify uses the correct paths when ran in parallel 1.verified.txt
@@ -0,0 +1 @@
+Verify Parallel feature #1
diff --git a/...lel feature #2.Check if Verify uses the correct paths when ran in parallel 2.verified.txt b/...lel feature #2.Check if Verify uses the correct paths when ran in parallel 2.verified.txt
@@ -0,0 +1 @@
+Verify Parallel feature #2
diff --git a/...ures/Verify Test.Check if Verify is working with global registered path info.verified.txt b/...ures/Verify Test.Check if Verify is working with global registered path info.verified.txt
@@ -0,0 +1 @@
+value
diff --git a/...Verify Test.Check if Verify is working with multiple scenario parameters_1_a.verified.txt b/...Verify Test.Check if Verify is working with multiple scenario parameters_1_a.verified.txt
@@ -0,0 +1 @@
+1 .. a
diff --git a/...Verify Test.Check if Verify is working with multiple scenario parameters_2_b.verified.txt b/...Verify Test.Check if Verify is working with multiple scenario parameters_2_b.verified.txt
@@ -0,0 +1 @@
+2 .. b
diff --git a/...rify.ReqnrollPlugin.IntegrationTest/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest.csproj b/...rify.ReqnrollPlugin.IntegrationTest/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest.csproj
@@ -34,10 +34,6 @@
     <ReqnrollGeneratorPlugins Include="$(MSBuildThisFileDirectory)..\..\Reqnroll.xUnit.Generator.ReqnrollPlugin\bin\$(Configuration)\netstandard2.0\Reqnroll.xUnit.Generator.ReqnrollPlugin.dll" />
   </ItemGroup>
 
-  <ItemGroup>
-    <Folder Include="Features\" />
-  </ItemGroup>
-
   <Import Project="..\..\..\Reqnroll.Tools.MsBuild.Generation\build\Reqnroll.Tools.MsBuild.Generation.targets" />
   <!-- has to be before the PropertyGroup change-->
   <PropertyGroup>

diff --git a/....Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/StepDefinitions/StepDefinitions.cs b/....Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/StepDefinitions/StepDefinitions.cs
@@ -3,20 +3,79 @@ namespace Reqnroll.Verify.ReqnrollPlugin.IntegrationTest.StepDefinitions;
 [Binding]
 internal class StepDefinitions
 {
+    private readonly VerifySettings _settings;
+    private VerifyResult _verifyResult;
+
+    public StepDefinitions(VerifySettings settings)
+    {
+        _settings = settings;
+    }
+
     [When("I try Verify with Reqnroll")]
-    public async Task ITryVerifyWithReqnroll()
+    public async Task WhenITryVerifyWithReqnroll()
     {
-        await Verifier.Verify("value");
+        await Verifier.Verify("value", _settings);
     }
 
     [When(@"I try Verify with Reqnroll for Parameter '([^']*)'")]
     public async Task WhenITryVerifyWithReqnrollForParameter(string p0)
+    {
+        await Verifier.Verify("value", _settings);
+    }
+
+    [When(@"I try Verify with Reqnroll for Parameter '(.*)' and some additional parameter '(.*)'")]
+    public async Task WhenITryVerifyWithReqnrollForParameterAndSomeAdditionalParameter(string p0, string p1)
+    {
+        await Verifier.Verify($"{p0} .. {p1}", _settings);
+    }
+
+    [When(@"I try Verify with Reqnroll with global registered path info")]
+    public async Task WhenITryVerifyWithReqnrollWithGlobalRegisteredPathInfo()
     {
         await Verifier.Verify("value");
     }
 
     [Then("it works")]
-    public void ItWorks()
+    public void ThenItWorks()
     {
+        // no-op
+    }
+
+    private static readonly object Lock = new();
+    private static bool _isFirstCallComplete;
+
+    [When(@"I try Verify with Reqnroll in parallel")]
+    public static void WhenITryVerifyWithReqnrollInParallel()
+    {
+        // lock so the first execution will always finish after the second execution
+
+        if (!_isFirstCallComplete)
+        {
+            lock (Lock)
+            {
+                if (!_isFirstCallComplete)
+                {
+                    Thread.Sleep(500);
+                    _isFirstCallComplete = true;
+                }
+            }
+        }
+    }
+
+    [Then(@"it works in parallel with contents `(.*)`")]
+    public async Task ThenItWorksInParallelWithVerifyContents(string contents)
+    {
+        _verifyResult = await Verifier.Verify(contents, _settings);
+    }
+
+    [Then(@"the verified file is `(.*)` with contents `(.*)`")]
+    public void ThenTheVerifiedFileIsWithContents(string fileName, string contents)
+    {
+        var actualFilePath = _verifyResult.Files.First();
+        var actualFileName = Path.GetFileName(actualFilePath);
+        Assert.Equal(fileName, actualFileName);
+
+        var actualContents = File.ReadAllText(actualFilePath);
+        Assert.Equal(contents, actualContents);
     }
 }
diff --git a/Plugins/Reqnroll.Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/reqnroll.json b/Plugins/Reqnroll.Verify/Reqnroll.Verify.ReqnrollPlugin.IntegrationTest/reqnroll.json
@@ -0,0 +1,8 @@
+{
+    "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
+    "generator": {
+        "addNonParallelizableMarkerForTags": [
+            "DoNotParallelize"
+        ]
+    }
+}
diff --git a/Plugins/Reqnroll.Verify/Reqnroll.Verify.ReqnrollPlugin/VerifyRuntimePlugin.cs b/Plugins/Reqnroll.Verify/Reqnroll.Verify.ReqnrollPlugin/VerifyRuntimePlugin.cs
@@ -1,5 +1,9 @@
+using System;
 using System.Collections;
 using System.IO;
+using System.Linq;
+using System.Text;
+using System.Threading;
 using Reqnroll.Plugins;
 using Reqnroll.UnitTestProvider;
 using Reqnroll.Verify.ReqnrollPlugin;
@@ -15,33 +19,59 @@ public class VerifyRuntimePlugin : IRuntimePlugin
     public void Initialize(RuntimePluginEvents runtimePluginEvents, RuntimePluginParameters runtimePluginParameters, UnitTestProviderConfiguration unitTestProviderConfiguration)
     {
         runtimePluginEvents.CustomizeGlobalDependencies += RuntimePluginEvents_CustomizeGlobalDependencies;
+        runtimePluginEvents.CustomizeScenarioDependencies += RuntimePluginEvents_CustomizeScenarioDependencies;
     }
 
     private void RuntimePluginEvents_CustomizeGlobalDependencies(object sender, CustomizeGlobalDependenciesEventArgs e)
     {
         var runtimePluginTestExecutionLifecycleEvents = e.ObjectContainer.Resolve<RuntimePluginTestExecutionLifecycleEvents>();
-        runtimePluginTestExecutionLifecycleEvents.BeforeScenario += RuntimePluginTestExecutionLifecycleEvents_BeforeScenario;
+        runtimePluginTestExecutionLifecycleEvents.BeforeScenario += (_, runtimePluginBeforeScenarioEventArgs) =>
+        {
+            var scenarioContext = runtimePluginBeforeScenarioEventArgs.ObjectContainer.Resolve<ScenarioContext>();
+            var featureContext = runtimePluginBeforeScenarioEventArgs.ObjectContainer.Resolve<FeatureContext>();
+
+            Verifier.DerivePathInfo(
+                (_, projectDirectory, _, _) =>
+                {
+                    string scenarioInfoTitle = scenarioContext.ScenarioInfo.Title;
+
+                    foreach (DictionaryEntry scenarioInfoArgument in scenarioContext.ScenarioInfo.Arguments)
+                    {
+                        scenarioInfoTitle += "_" + scenarioInfoArgument.Value;
+                    }
+
+                    return new PathInfo(
+                        Path.Combine(projectDirectory, featureContext.FeatureInfo.FolderPath),
+                        featureContext.FeatureInfo.Title,
+                        scenarioInfoTitle);
+                });
+        };
     }
 
-    private void RuntimePluginTestExecutionLifecycleEvents_BeforeScenario(object sender, RuntimePluginBeforeScenarioEventArgs e)
+    private void RuntimePluginEvents_CustomizeScenarioDependencies(object sender, CustomizeScenarioDependenciesEventArgs e)
     {
-        var scenarioContext = e.ObjectContainer.Resolve<ScenarioContext>();
-        var featureContext = e.ObjectContainer.Resolve<FeatureContext>();
-
-        Verifier.DerivePathInfo(
-            (sourceFile, projectDirectory, type, method) =>
+        e.ObjectContainer.RegisterFactoryAs(
+            container =>
             {
-                string scenarioInfoTitle = scenarioContext.ScenarioInfo.Title;
+                var featureContext = container.Resolve<FeatureContext>();
+                var scenarioContext = container.Resolve<ScenarioContext>();
 
-                foreach (DictionaryEntry scenarioInfoArgument in scenarioContext.ScenarioInfo.Arguments)
+                var settings = new VerifySettings();
+                string projectDirectory = Directory.GetCurrentDirectory().Split([@"\bin\"], StringSplitOptions.RemoveEmptyEntries).First();
+
+                settings.UseDirectory(Path.Combine(projectDirectory, featureContext.FeatureInfo.FolderPath));
+                settings.UseTypeName(featureContext.FeatureInfo.Title);
+
+                var methodNameBuilder = new StringBuilder(scenarioContext.ScenarioInfo.Title);
+
+                foreach (DictionaryEntry entry in scenarioContext.ScenarioInfo.Arguments)
                 {
-                    scenarioInfoTitle += "_" + scenarioInfoArgument.Value;
+                    methodNameBuilder.AppendFormat("_{0}", entry.Value);
                 }
 
-                return new PathInfo(
-                    Path.Combine(projectDirectory, featureContext.FeatureInfo.FolderPath),
-                    featureContext.FeatureInfo.Title,
-                    scenarioInfoTitle);
+                settings.UseMethodName(methodNameBuilder.ToString());
+
+                return settings;
             });
     }
 }
diff --git a/docs/integrations/available-plugins.md b/docs/integrations/available-plugins.md
@@ -4,12 +4,13 @@
 
 | Name | Description | Download |
 |---|---|---|
-|[Reqnroll.Autofac](https://github.com/reqnroll/Reqnroll)|Reqnroll plugin for using Autofac as a dependency injection framework for step definitions. [Read more...](autofac.md)|<a href="https://www.nuget.org/packages/Reqnroll.Autofac/">![](https://img.shields.io/nuget/v/Reqnroll.Autofac.svg)</a>|
-|[Reqnroll.Windsor](https://github.com/reqnroll/Reqnroll)|Reqnroll plugin for using Castle Windsor as a dependency injection framework for step definitions. [Read more...](windsor.md)|<a href="https://www.nuget.org/packages/Reqnroll.Windsor/">![](https://img.shields.io/nuget/v/Reqnroll.Windsor.svg)</a>|
-|[Reqnroll.Microsoft.Extensions.DependencyInjection](https://github.com/reqnroll/Reqnroll)|Reqnroll plugin for using Microsoft.Extensions.DependencyInjection as a dependency injection framework for step definitions. [Read more...](https://github.com/reqnroll/Reqnroll)|<a href="https://www.nuget.org/packages/Reqnroll.Microsoft.Extensions.DependencyInjection/">![](https://img.shields.io/nuget/v/Reqnroll.Microsoft.Extensions.DependencyInjection.svg)</a>|
+| [Reqnroll.Autofac](https://github.com/reqnroll/Reqnroll) | Reqnroll plugin for using Autofac as a dependency injection framework for step definitions. [Read more...](autofac.md)| <a href="https://www.nuget.org/packages/Reqnroll.Autofac/">![](https://img.shields.io/nuget/v/Reqnroll.Autofac.svg)</a> |
+| [Reqnroll.Microsoft.Extensions.DependencyInjection](https://github.com/reqnroll/Reqnroll) | Reqnroll plugin for using Microsoft.Extensions.DependencyInjection as a dependency injection framework for step definitions. [Read more...](dependency-injection.md) | <a href="https://www.nuget.org/packages/Reqnroll.Microsoft.Extensions.DependencyInjection/">![](https://img.shields.io/nuget/v/Reqnroll.Microsoft.Extensions.DependencyInjection.svg)</a> |
+| [Reqnroll.Windsor](https://github.com/reqnroll/Reqnroll) | Reqnroll plugin for using Castle Windsor as a dependency injection framework for step definitions. [Read more...](windsor.md) | <a href="https://www.nuget.org/packages/Reqnroll.Windsor/">![](https://img.shields.io/nuget/v/Reqnroll.Windsor.svg)</a> |
 
 ## Other Plugins
 
 | Name | Description | Download |
 |---|---|---|
-| [Reqnroll.External Data](https://www.nuget.org/packages/Reqnroll.ExternalData/) | Package to use external data in Gherkin scenarios. [Read more...](https://go.reqnroll.net/doc-externaldata) | <a href="https://www.nuget.org/packages/Reqnroll.ExternalData/">![](https://img.shields.io/nuget/vpre/Reqnroll.ExternalData.svg)</a>|
+| [Reqnroll.External Data](https://www.nuget.org/packages/Reqnroll.ExternalData/) | Package to use external data in Gherkin scenarios. [Read more...](https://go.reqnroll.net/doc-externaldata) | <a href="https://www.nuget.org/packages/Reqnroll.ExternalData/">![](https://img.shields.io/nuget/vpre/Reqnroll.ExternalData.svg)</a> |
+| [Reqnroll.Verify](https://github.com/reqnroll/Reqnroll/tree/main/Plugins/Reqnroll.Verify) | Reqnroll plugin for using Verify in scenario. [Read more...](verify.md) | <a href="https://www.nuget.org/packages/Reqnroll.Verify/">![](https://img.shields.io/nuget/v/Reqnroll.Verify.svg)</a> |
diff --git a/docs/integrations/index.md b/docs/integrations/index.md
@@ -8,10 +8,10 @@ This part contains details of the following topics.
 available-plugins
 autofac
 dependency-injection
-windsor
-fsharp
 externaldata
+fsharp
 mstest
 nunit
+windsor
 xunit
 ```
diff --git a/docs/integrations/verify.md b/docs/integrations/verify.md
@@ -0,0 +1,50 @@
+# Verify
+
+Reqnroll supports Verify 24.2.0 or later.  
+
+Documentation for Verify can be found [here](https://github.com/VerifyTests/Verify).
+
+## Needed NuGet Packages
+
+* [Reqnroll.xUnit](https://www.nuget.org/packages/Reqnroll.xUnit/) and its [dependencies](xunit.md#Needed%20NuGet%20Packages)
+* [Reqnroll.Verify](https://www.nuget.org/packages/Reqnroll.Verify/)
+
+## How it works
+
+This plugin adds a VerifySettings instance to Reqnroll's scenario container, which can be used to set the correct path for the tests' verified files.
+
+### Example
+
+```Gherkin
+Feature: Verify feature
+
+    Scenario: Verify scenario
+        When I calculate 1 + 2
+        Then I expect the result is correct
+```
+```csharp
+[Binding]
+internal class StepDefinitions
+{
+    private readonly VerifySettings _settings;
+
+    public StepDefinitions(VerifySettings settings)
+    {
+        _settings = settings;
+    }
+
+    [When("I calculate (\d+) + (\d+)")]
+    public void WhenICalculate(int value, int value2)
+    {
+        _settings.Verify(value + value, _settings);
+    }
+
+    [Then("I expect the result is correct")]
+    public void ThenIExpectTheResultIsCorrect()
+    {
+    }
+}
+
+```
+
+**Note:** in a single-threaded environment, the plugin will work without the injected VerifySettings instance. However, in a multithreaded environment, the VerifySettings instance must be injected into the step definition class for a deterministic outcome.