Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"SqlException: The transaction operation cannot be performed because there are pending requests..." during setup with SQL Server #15794

Closed
MikeAlhayek opened this issue Apr 19, 2024 · 20 comments · Fixed by #15942
Labels
Milestone

Comments

@MikeAlhayek
Copy link
Member

Describe the bug

To Reproduce

Steps to reproduce the behavior:

  1. Create a new tenant using SQL Server (I tried with SQL Server in Azure) and the Blog recipe.
  2. Setup the site.
  3. See error

Expected behavior

There should be no exceptions

Screenshots

image

@MikeAlhayek MikeAlhayek added this to the 1.9 milestone Apr 19, 2024
@MikeAlhayek
Copy link
Member Author

This maybe related to #15628

@infofromca
Copy link
Contributor

infofromca commented Apr 19, 2024

An unhandled exception occurred while processing the request.

NpgsqlOperationInProgressException: A command is already in progress: SELECT "Document".* FROM "Document" WHERE "Document"."Type" = @Type limit 1
Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|279_0(ConnectorState newState, NpgsqlCommand command, ref <>c__DisplayClass279_0 )

Stack Query Cookies Headers Routing
NpgsqlOperationInProgressException: A command is already in progress: SELECT "Document".* FROM "Document" WHERE "Document"."Type" = @Type limit 1
Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|279_0(ConnectorState newState, NpgsqlCommand command, ref <>c__DisplayClass279_0 )
Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, bool attemptPgCancellation)
Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken)
Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken)
System.Threading.Tasks.ValueTask<TResult>.get_Result()
Npgsql.NpgsqlCommand.ExecuteNonQuery(bool async, CancellationToken cancellationToken)
Dapper.SqlMapper.ExecuteImplAsync(IDbConnection cnn, CommandDefinition command, object param) in SqlMapper.Async.cs
YesSql.Commands.UpdateDocumentCommand.ExecuteAsync(DbConnection connection, DbTransaction transaction, ISqlDialect dialect, ILogger logger)
YesSql.Session.FlushAsync()
YesSql.Session.FlushAsync()
YesSql.Session.SaveChangesAsync()
YesSql.Session.SaveChangesAsync()
OrchardCore.Data.Documents.DocumentStore.CommitAsync() in DocumentStore.cs
OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync() in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteStepAsync(RecipeExecutionContext recipeStep) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Setup.Services.SetupService.SetupInternalAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupInternalAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Controllers.SetupController.IndexPOST(SetupViewModel model) in SetupController.cs
lambda_method118(Closure , object )
Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor+TaskOfActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
System.Threading.Tasks.ValueTask<TResult>.get_Result()
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask<IActionResult> actionResultValueTask)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
OrchardCore.Modules.ModularTenantRouterMiddleware.Invoke(HttpContext httpContext) in ModularTenantRouterMiddleware.cs
OrchardCore.Modules.ModularTenantContainerMiddleware+<>c__DisplayClass4_0+<<Invoke>b__0>d.MoveNext() in ModularTenantContainerMiddleware.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Modules.ModularTenantContainerMiddleware.Invoke(HttpContext httpContext) in ModularTenantContainerMiddleware.cs
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

@infofromca
Copy link
Contributor

infofromca commented Apr 19, 2024

An unhandled exception occurred while processing the request.

ObjectDisposedException: Cannot access a disposed object.
Object name: 'MySqlConnector.MySqlTransaction'.
MySqlConnector.MySqlTransaction.VerifyValid() in MySqlTransaction.cs, line 281

Stack Query Cookies Headers Routing
ObjectDisposedException: Cannot access a disposed object. Object name: 'MySqlConnector.MySqlTransaction'.
MySqlConnector.MySqlTransaction.VerifyValid() in MySqlTransaction.cs
MySqlConnector.MySqlTransaction.RollbackAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in MySqlTransaction.cs
YesSql.Session.CommitOrRollbackTransactionAsync()
YesSql.Session.CommitOrRollbackTransactionAsync()
YesSql.Session.SaveChangesAsync()
OrchardCore.Data.Documents.DocumentStore.CommitAsync() in DocumentStore.cs
OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync() in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteStepAsync(RecipeExecutionContext recipeStep) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Setup.Services.SetupService.SetupInternalAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupInternalAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Controllers.SetupController.IndexPOST(SetupViewModel model) in SetupController.cs
lambda_method63(Closure , object )
Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor+TaskOfActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
System.Threading.Tasks.ValueTask<TResult>.get_Result()
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask<IActionResult> actionResultValueTask)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
OrchardCore.Modules.ModularTenantRouterMiddleware.Invoke(HttpContext httpContext) in ModularTenantRouterMiddleware.cs
OrchardCore.Modules.ModularTenantContainerMiddleware+<>c__DisplayClass4_0+<<Invoke>b__0>d.MoveNext() in ModularTenantContainerMiddleware.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Modules.ModularTenantContainerMiddleware.Invoke(HttpContext httpContext) in ModularTenantContainerMiddleware.cs
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

@infofromca
Copy link
Contributor

infofromca commented Apr 19, 2024

An unhandled exception occurred while processing the request.

InvalidOperationException: Collection was modified; enumeration operation may not execute.
YesSql.Session.FlushAsync()

Stack Query Cookies Headers Routing
InvalidOperationException: Collection was modified; enumeration operation may not execute.
YesSql.Session.FlushAsync()
YesSql.Session.FlushAsync()
YesSql.Session.SaveChangesAsync()
YesSql.Session.SaveChangesAsync()
OrchardCore.Data.Documents.DocumentStore.CommitAsync() in DocumentStore.cs
OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync() in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteStepAsync(RecipeExecutionContext recipeStep) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(string executionId, RecipeDescriptor recipeDescriptor, IDictionary<string, object> environment, CancellationToken cancellationToken) in RecipeExecutor.cs
OrchardCore.Setup.Services.SetupService.SetupInternalAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupInternalAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Services.SetupService.SetupAsync(SetupContext context) in SetupService.cs
OrchardCore.Setup.Controllers.SetupController.IndexPOST(SetupViewModel model) in SetupController.cs
lambda_method173(Closure , object )
Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor+TaskOfActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
System.Threading.Tasks.ValueTask<TResult>.get_Result()
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask<IActionResult> actionResultValueTask)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
OrchardCore.Modules.ModularTenantRouterMiddleware.Invoke(HttpContext httpContext) in ModularTenantRouterMiddleware.cs
OrchardCore.Modules.ModularTenantContainerMiddleware+<>c__DisplayClass4_0+<<Invoke>b__0>d.MoveNext() in ModularTenantContainerMiddleware.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func<ShellScope, Task> execute, bool activateShell) in ShellScope.cs
OrchardCore.Modules.ModularTenantContainerMiddleware.Invoke(HttpContext httpContext) in ModularTenantContainerMiddleware.cs
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

@infofromca
Copy link
Contributor

infofromca commented Apr 19, 2024

just met the above 3 errors when i keep the old volume of the db
when i empty all, and compose up from blank, it got success.
it seems happen occationally.

use the same ways, i did not met the errors using the commit of a long time ago

@Piedone
Copy link
Member

Piedone commented Apr 21, 2024

All of these exceptions seem to have something to do with parallel SQL queries happening with the same ISession. I.e, mistaken multi-threading, Task.WhenAll(), or something like that. #15628 definitely seems related, as does #15224.

@Piedone
Copy link
Member

Piedone commented Apr 21, 2024

Hmm, we actually have quite a lot of Task.WhenAll(), especially related to recipes.

@MikeAlhayek
Copy link
Member Author

@Piedone yes I agree with your assessment. Recipes and sub recipes are executed in a new tenant scope and should be using a new instance of the ISession.

Maybe there is an issue with creating a new scope where we are reusing same service across scope

@Piedone
Copy link
Member

Piedone commented Apr 22, 2024

Hmm, do we need new scopes, to begin with? Is that for when recipes enable/disable features, right?

If this is done before #15628 (if the two are different issues, to begin with; I'm not sure) then here also we should also change the trigger of functional_all_db so it runs at least on PR reviews, so we can catch such issues earlier:

on:
  # Manual trigger.
  workflow_dispatch:
  pull_request_review:
    types: [submitted]

@MikeAlhayek
Copy link
Member Author

Hmm, we actually have quite a lot of Task.WhenAll(), especially related to recipes.

We mainly use Task.WhenAll() to harvest recipes. None of the Recipe harvesters depends on ISession or make a call to a database so it is fine to use it there.

Hmm, do we need new scopes, to begin with?

When we execute sub recipes from a main recipe, each recipe should run independently with a new scope. I am guessing there is a reason why it's done this way but not sure I know the exact reason.

@Piedone
Copy link
Member

Piedone commented Apr 22, 2024

Yeah, I guess that's to allow reloading the shell for feature state changes.

@Piedone Piedone changed the title An unhandled Exception occured while processing the request "SqlException: The transaction operation cannot be performed because there are pending requests..." during setup with SQL Server Apr 28, 2024
@MikeAlhayek
Copy link
Member Author

The issue here is related to UpdateRole

It is what is causing this issue. It has to be a concurrency issue because if you step through the code everything works.

@sebastienros
Copy link
Member

If this is something that stopped working, maybe going back in time could help isolate the problem to a few commits?

My instinct would say it's about the DocumentManager usage of ShellScope.Current and other mixes of direct resolution of IDocumentStore. Though as mentioned many times, if everything was serial that there shouldn't be any concurrent usage of sessions, so this should not be an issue.

@MikeAlhayek
Copy link
Member Author

Yeah very strange. I tried to locate all files that are involved and looking at their change history. I don't see anything that stood out. Unless there is an issue with the shell scope. I don't recall seeing any change in the infrastructure. But assigning an incorrect current scope could cause this odd behavior

@MikeAlhayek
Copy link
Member Author

@sebastienros I think I see the problem now. But I am still not sure what changes for this to break now and work before.

The method RoleUpdater.UpdateRolesForInstalledFeatureAsync is called when the features are installed and this is called using a deferred task.

Next, the RoleStep "which is executed during the setup" uses the RoleManager to create or update roles. When the role manager is called, the RoleUpdater.UpdateRoleForInstalledFeaturesAsync() or RoleUpdater.RemoveRoleForMissingFeaturesAsync() are called from a different scope since this would be triggered by the RoleStep.

Since the RoleUpdater and The RoleStep both call the DocumentManager, this seems to be causing the issue.
What I still don't understand is why would this cause parallel execution? Thoughts?

To confirm the finding, I removed the RolesStep from the IoC and the site was setup with no issue.

@Piedone
Copy link
Member

Piedone commented May 1, 2024

Isn't this called by chance, parallelizing steps? Or, perhaps, isn't one of the services resolved or reused across scope boundaries?

@MikeAlhayek
Copy link
Member Author

MikeAlhayek commented May 1, 2024

@Piedone this is only called by ModularBackgroundService and in

await modules.ForEachAsync((module) =>
{
if (!module.ModuleInfo.Exists)
{
return Task.CompletedTask;
}
var manifestInfo = new ManifestInfo(module.ModuleInfo);
var extensionInfo = new ExtensionInfo(module.SubPath, manifestInfo, (mi, ei) =>
{
return _featuresProvider.GetFeatures(ei, mi);
});
var entry = new ExtensionEntry
{
ExtensionInfo = extensionInfo,
Assembly = module.Assembly,
ExportedTypes = module.Assembly.ExportedTypes
};
loadedExtensions.TryAdd(module.Name, entry);
return Task.CompletedTask;
});

@MikeAlhayek
Copy link
Member Author

I think I have a good theory about what is happening. I'll try it out tomorrow.

@sebastienros
Copy link
Member

If that helps, I test locally by setting up the autosetup module in appsettings.json

    // WARNING: AutoSetup section given as an example for Development only, for Production use "Environment Variables" instead
    "OrchardCore_AutoSetup": {
      "AutoSetupPath": "",
      "Tenants": [
        {
          "ShellName": "Default",
          "SiteName": "AutoSetup Example",
          "SiteTimeZone": "Europe/Amsterdam",
          "AdminUsername": "admin",
          "AdminEmail": "[email protected]",
          "AdminPassword": "Demo123!",
          "DatabaseProvider": "SqlConnection",
          "DatabaseConnectionString": "Server=localhost;Database=tempdb;User Id=sa;Password=Password12!;TrustServerCertificate=True",
          "DatabaseTablePrefix": "test39",
          "DatabaseSchema": "",
          "RecipeName": "Blog"
        },

This way I only need to start the app with the changes and update the DatabaseTablePrefix to see it crashing. Besides I also have changed my local yessql projects to detect concurrent usages of sessions (read or writes) and show both stacktraces when this happens. Maybe something to add as an option in yessql for later occurences, the same way DI has this options to let us know when a Singleton resolves a Scoped.

@sebastienros
Copy link
Member

These are the two concurrent stacktraces that trigger the issue consistently for me (now even with my changes, I assume it worked once and didn't recrete the tenant after)

Current:
   at System.Environment.get_StackTrace()
         at YesSql.Session.SaveChangesAsync() in D:\yessql\src\YesSql.Core\Session.cs:line 897
         at YesSql.Session.SaveChangesAsync()
         at OrchardCore.Data.Documents.DocumentStore.CommitAsync() in D:\orchardcore\src\OrchardCore\OrchardCore.Data.YesSql\Documents\DocumentStore.cs:line 121
         at OrchardCore.Data.Documents.DocumentStore.CommitAsync()
         at Microsoft.Extensions.DependencyInjection.OrchardCoreBuilderExtensions.<>c.<AddDataAccess>b__0_4(ShellScope scope) in D:\orchardcore\src\OrchardCore\OrchardCore.Data.YesSql\OrchardCoreBuilderExtensions.cs:line 156
         at OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync() in D:\orchardcore\src\OrchardCore\OrchardCore.Abstractions\Shell\Scope\ShellScope.cs:line 442
         at OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync()
         at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute, Boolean activateShell) in D:\orchardcore\src\OrchardCore\OrchardCore.Abstractions\Shell\Scope\ShellScope.cs:line 305
         at OrchardCore.Recipes.Services.RecipeExecutor.<>c__DisplayClass7_0.<<ExecuteStepAsync>b__0>d.MoveNext() in D:\orchardcore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 182
         at OrchardCore.Roles.Recipes.RolesStep.ExecuteAsync(RecipeExecutionContext context) in D:\orchardcore\src\OrchardCore.Modules\OrchardCore.Roles\Recipes\RolesStep.cs:line 62
         at Microsoft.AspNetCore.Identity.RoleManager`1.CreateAsync(TRole role)
         at OrchardCore.Roles.Services.RoleStore.CreateAsync(IRole role, CancellationToken cancellationToken) in D:\orchardcore\src\OrchardCore.Modules\OrchardCore.Roles\Services\RoleStore.cs:line 79
         at OrchardCore.Documents.DocumentManager`1.GetOrCreateMutableAsync(Func`1 factoryAsync) in D:\orchardcore\src\OrchardCore\OrchardCore.Infrastructure\Documents\DocumentManager.cs:line 99
         at OrchardCore.Data.Documents.DocumentStore.GetOrCreateMutableAsync[T](Func`1 factoryAsync) in D:\orchardcore\src\OrchardCore\OrchardCore.Data.YesSql\Documents\DocumentStore.cs:line 45
         at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl() in D:\yessql\src\YesSql.Core\Services\DefaultQuery.cs:line 1275
         at YesSql.Store.ProduceAsync[T,TState](WorkerQueryKey key, Func`2 work, TState state) in D:\yessql\src\YesSql.Core\Store.cs:line 343
         at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 475
---
Other:
   at System.Environment.get_StackTrace()
         at YesSql.Session.FlushAsync(Boolean saving) in D:\yessql\src\YesSql.Core\Session.cs:line 695
         at YesSql.Session.FlushAsync(Boolean saving)
         at YesSql.Session.FlushAsync() in D:\yessql\src\YesSql.Core\Session.cs:line 667
         at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl() in D:\yessql\src\YesSql.Core\Services\DefaultQuery.cs:line 1198
         at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
         at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultAsync() in D:\yessql\src\YesSql.Core\Services\DefaultQuery.cs:line 1192
         at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync) in D:\orchardcore\src\OrchardCore\OrchardCore.Data.YesSql\Documents\DocumentStore.cs:line 56
         at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
         at OrchardCore.Documents.DocumentManager`1.SetInternalAsync(TDocument document, Boolean failover) in D:\orchardcore\src\OrchardCore\OrchardCore.Infrastructure\Documents\DocumentManager.cs:line 344
         at OrchardCore.Documents.DocumentManager`1.SetInternalAsync(TDocument document, Boolean failover)
         at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync) in D:\orchardcore\src\OrchardCore\OrchardCore.Infrastructure\Documents\DocumentManager.cs:line 154
         at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync) in D:\orchardcore\src\OrchardCore\OrchardCore.Data.YesSql\Documents\DocumentStore.cs:line 64
         at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl() in D:\yessql\src\YesSql.Core\Services\DefaultQuery.cs:line 1275
         at YesSql.Store.ProduceAsync[T,TState](WorkerQueryKey key, Func`2 work, TState state) in D:\yessql\src\YesSql.Core\Store.cs:line 343
         at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 475
---

One thread is saving a document in BeforeDisposeAsync -> CommitAsync, and the other thread is querying a document (the same?) with the same session (DefaultQuery -> FlushAsync because pending changes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants