Deadlock in integration tests after upgrading to ASP.NET Core 6

[angularsen]

Describe the bug

After upgrading from ASP.NET Core 5 to 6, our integration tests would hang indefinitely on Azure Pipelines with ubuntu-20.04.

After analyzing dumps with dotnet test --blame-hang-timeout, we found a deadlock on WebApplicationFactory.Dispose() and concurrent tests calling the API.

The root cause

• Single WebApplicationFactory instance per test class, reused by its test methods to call the API.
• Hosted service with Task.Delay(TimeSpan.FromSeconds(5), ct) in its IHostedService.StopAsync(), which is used to flush logs and telemetry before the API is allowed to shut down.
• The delay seems to create a deadlock for shared factories, only on Linux/Ubuntu and only on .NET6.

Workaround
Do not delay IHostedService.StopAsync() for "Test" environment.

Investigation

• dotnet test on Azure Pipelines ubuntu-20.04 would hang indefinitely after upgrading to ASP.NET Core 6
• Could not reproduce on Azure Pipelines windows-2019
• Could not reproduce on Windows, with Rider or Visual Studio 2022
• Could not reproduce on Windows with dotnet test
• Could not reproduce on ASP.NET Core 5
• Could not reproduce on ASP.NET Core 6 when creating individual factories per test method

To Reproduce

Detailed description and repro integration tests here:
Repro hang of integration tests after upgrading to ASP.NET Core 6 - Pull Request #1 · angularsen/NET6-Testing

Exceptions (if any)

On Ubuntu, deadlock.
On Windows (as expected):

[Test Class Cleanup Failure (Test.Net6WebApi.Repro_RED_ConcurrentTestsOnSharedFactory+TestClass3)]: System.AggregateException : One or more hosted services failed to stop. (A task was canceled.)
---- System.Threading.Tasks.TaskCanceledException : A task was canceled.
  Stack Trace:
     at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.DisposeAsync()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.Dispose(Boolean disposing)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.Dispose()
----- Inner Stack Trace -----
   at Test.Net6WebApi.Setup.MySlowStoppingHostedService.StopAsync(CancellationToken cancellationToken) in C:\dev\digma\NET6-Testing\src\Test.Net6WebApi\Setup\MySlowStoppingHostedService.cs:line 30
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)

Further technical details

• ASP.NET Core version: 6.0.0
• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version: Rider 2021.2.2, VS 2022 17.1.0 Preview 1.0
• Include the output of dotnet --info:

dotnet --info Output

WSL Ubuntu 20.10:

.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  21.10
 OS Platform: Linux
 RID:         ubuntu.21.10-x64
 Base Path:   /usr/share/dotnet/sdk/6.0.100/

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  3.1.415 [/usr/share/dotnet/sdk]
  5.0.403 [/usr/share/dotnet/sdk]
  6.0.100 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.21 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.12 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.21 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Windows:

.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19043
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.100\

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  3.1.415 [C:\Program Files\dotnet\sdk]
  5.0.209 [C:\Program Files\dotnet\sdk]
  5.0.301 [C:\Program Files\dotnet\sdk]
  5.0.302 [C:\Program Files\dotnet\sdk]
  6.0.100-rc.2.21505.57 [C:\Program Files\dotnet\sdk]
  6.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0-rc.2.21480.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-rc.2.21480.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0-rc.2.21501.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[halter73]

Hosted service with Task.Delay(TimeSpan.FromSeconds(5), ct) in its IHostedService.StopAsync(), which is used to flush logs and telemetry before the API is allowed to shut down.

I'm surprised this doesn't hang in .NET 5. Why should WebApplicationFactory.Dispose() not wait for StopAsync() to complete on the IHost and all its IHostedServices? if the issue is just the blocking, WebApplicationFactory implements IAsyncDisposable as well.

[angularsen]

@halter73 It is not the 5 second hang that is the problem, that is naturally expected as shown in Windows logs above. What is unexpected is deadlock and infinite hang on Ubuntu, only on .NET6.

[BrennanConroy]

Could you try using a Noop IHostLifetime implementation and see if the hang still occurs.
See https://github.com/dotnet/aspnetcore/pull/23761/files for reference.

[ghost]

Hi @angularsen. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[angularsen]

@BrennanConroy I am certain that a Noop IHostLifetime works, because the workaround I described was to remove the Task.Delay in StopAsync of our IHostedService implementation. This fixed our tests. I believe this is effectively the same as the Noop implementation, because our implementation did little beyond logging and waiting on stop.

[ghost]

Thanks for contacting us.
We're moving this issue to the .NET 7 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[ghost]

Thanks for contacting us.
We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[mitchdenny]

Looks like @BrennanConroy's change merged in, I'm assuming this is the long term fix for this issue. Re-open if it isn't Brennan.

[gdoron]

@mitchdenny which change were you referring to? Can you kindly add a link so I could see if it would fix our issue as well?