Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Migration - The ConnectionString property has not been initialized #15290

Closed
BinderDyn opened this issue Feb 9, 2024 · 14 comments
Closed
Labels
Milestone

Comments

@BinderDyn
Copy link
Contributor

BinderDyn commented Feb 9, 2024

Describe the bug

I migrated from 1.7.x to 1.8.2 and fixed all the warnings concerning the obsolete content type and part migration steps. The system is hosted in a docker container on azure. After deploying, the system freezes and times out. Analyizing the stack trace I got this:

   at Microsoft.Data.SqlClient.SqlConnection.PermissionDemand (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlConnectionFactory.PermissionDemand (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlConnection.InternalOpenAsync (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Dapper.SqlMapper+<ExecuteImplAsync>d__39.MoveNext (Dapper.StrongName, Version=2.0.0.0, Culture=neutral, PublicKeyToken=e3e8412083d25dd3: /_/Dapper/SqlMapper.Async.cs:661)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at YesSql.Sql.SchemaBuilder+<ExecuteAsync>d__53.MoveNext (YesSql.Core, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at YesSql.Sql.SchemaBuilder+<AlterTableAsync>d__36.MoveNext (YesSql.Core, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at YesSql.Sql.SchemaBuilder+<AlterIndexTableAsync>d__37.MoveNext (YesSql.Core, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at REDACTED.Module.Main.Indices.SensorIndex+<WithSchemaBuilder>d__17.MoveNext (REDACTED.Module.Main, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /source/REDACTED.Module.Main/Indices/SensorIndex.cs:37)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at REDACTED.Module.Main.Migrations+<InitializeIndexes>d__4.MoveNext (REDACTED.Module.Main, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /source/REDACTED.Module.Main/Migrations.cs:39)

Besides that the error in my log says the following:

Message | The ConnectionString property has not been initialized. |  
Exception type | System.InvalidOperationException |  
Failed method | Dapper.SqlMapper+<ExecuteImplAsync>d__39.MoveNext

NOTE: The connection string IS SET.
Apparently something is going wrong when initalizing the indexes. Here is the example code for the failing sensor index:

public class SensorIndex : MapIndex
{
    public const string IndexTableName = "IDX_SensorIndex_DocumentId";
    public string SensorContentItemId { get; set; }
    public string SensorId { get; set; }
    public SensorType SensorType { get; set; }
    public string AddressId { get; set; }

    public static async Task WithSchemaBuilder(ISchemaBuilder schemaBuilder)
    {
        try
        {
            await schemaBuilder.CreateMapIndexTableAsync<SensorIndex>(table => table
                .Column<string>(nameof(SensorContentItemId), c => c.WithLength(26))
                .Column<string>(nameof(SensorId))
                .Column<SensorType>(nameof(SensorType))
                .Column<string>(nameof(AddressId))
            );
            await schemaBuilder.AlterIndexTableAsync<SensorIndex>(table => table
                .CreateIndex(IndexTableName,
                    nameof(SensorContentItemId),
                    nameof(SensorId),
                    nameof(SensorType),
                    nameof(AddressId))
            );
        }
        catch (Exception)
        {
            await schemaBuilder.AlterIndexTableAsync<SensorIndex>(table => table
                .CreateIndex(IndexTableName,
                    nameof(SensorContentItemId),
                    nameof(SensorId),
                    nameof(SensorType),
                    nameof(AddressId))
            );
        }
    }
}

As you can see, I already wrapped the initialization into a trycatch statement. In the past we got exceptions telling us that the index already exists. Since all tenants already exist and have migrated to the new indexes I summarized the updates for each index creation into one method (without harming the order of the migrations).

To Reproduce

Steps to reproduce the behavior:

  1. Create any index
  2. Add async migration code for any content part
  3. Apply the migration for indexes and content parts

Expected behavior

On one hand I would expect the index logic to check if the index already exists and only if that's not the case it should create the index tables. Besides that I want to know where the error is originating and what I can do to prevent it from happening.

Screenshots

@sebastienros
Copy link
Member

Can you log the exception in your Try/Catch?

Can you show the actual migration that invokes your custom WithSchemaBuilder? Just to see if the services are correctly defined and the methods return correct version updates.

@sebastienros sebastienros added this to the 1.x milestone Feb 15, 2024
@MikeAlhayek
Copy link
Member

I am also noticing this when upgrading 1.8.2 to main.

image

@sebastienros
Copy link
Member

@MikeAlhayek can you try to point to the previous SqlClient version locally (with the security advisory)

@MikeAlhayek
Copy link
Member

MikeAlhayek commented Feb 21, 2024

@sebastienros I added the following into my project. But not sure if it has any affect since none of my projects directly depend on the SqlClient.

Also, the problem is not consistent. I think it happens due to slow responding "like time out" but not really sure. I'll see if I can get the root cause locally.

    <PackageVersion Include="Microsoft.Data.SqlClient" Version="5.1.2" />

@BinderDyn
Copy link
Contributor Author

Can you log the exception in your Try/Catch?

Can you show the actual migration that invokes your custom WithSchemaBuilder? Just to see if the services are correctly defined and the methods return correct version updates.

Sorry for the delay. The code concerning the call of the migrations is the following:

private async Task InitializeIndexes()
    {
        try
        {
            await SensorIndex.WithSchemaBuilder(SchemaBuilder);
        }
        catch (Exception e)
        {
            logger.LogError(e, "Error creating SensorIndex");
        }
        
        try
        {
            await AverageSensorIndex.WithSchemaBuilder(SchemaBuilder);
        }
        catch (Exception e)
        {
            logger.LogError(e, "Error creating AverageSensorIndex");
        }
        
        try
        {
            await AddressIndex.WithSchemaBuilder(SchemaBuilder);
        }
        catch (Exception e)
        {
            logger.LogError(e, "Error creating AddressIndex");
        }
    }

This is subsequently called in one of the migration steps:

public async Task<int> UpdateFrom89Async()
    {
        await migrationFactory.CreateAsync<ChangeTextForTooltip>().MigrateAsync();
        
        await InitializeIndexes();
        
        return 90;
    }

Regarding the theorized timeout this could also be the case because the system is rather underpowered (it's a dev stage). This would also explain why this isn't happening on my local device

@sebastienros
Copy link
Member

Could it be a bug in the driver that when a connection is failing it's still returned to the pool in an incorrect state. I have seen issues related in the Microsoft.Data.SqlClient repository recently.

Something to try also is to add some reconnection properties to the connection string if that's supported.

@Piedone
Copy link
Member

Piedone commented May 3, 2024

Retargeting to 2.0 since this is a regression from 1.7.x.

We see the same exception (may or may not be the same issue, stack trace is naturally different) but only under /Admin/Workflows/Types. It's thrown every time, even after a fresh setup without any actual Workflow Types. With Azure SQL.

System.InvalidOperationException:
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlConnection.InternalOpenAsync (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at YesSql.Session+<CreateConnectionAsync>d__51.MoveNext (YesSql.Core, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at YesSql.Services.DefaultQuery+Query`1+<FirstOrDefaultImpl>d__4.MoveNext (YesSql.Core, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Lombiq.Hosting.Tenants.EmailQuotaManagement.Services.EmailQuotaService+<GetOrCreateCurrentQuotaAsync>d__11.MoveNext (Lombiq.Hosting.Tenants.EmailQuotaManagement, Version=1.367.1.0, Culture=neutral, PublicKeyToken=null: /home/runner/work/DotNest-Tenants-Core/DotNest-Tenants-Core/src/Modules/Lombiq.Hosting.Tenants/Lombiq.Hosting.Tenants.EmailQuotaManagement/Services/EmailQuotaService.cs:85)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Lombiq.Hosting.Tenants.EmailQuotaManagement.Services.EmailQuotaService+<IsQuotaOverTheLimitAsync>d__10.MoveNext (Lombiq.Hosting.Tenants.EmailQuotaManagement, Version=1.367.1.0, Culture=neutral, PublicKeyToken=null: /home/runner/work/DotNest-Tenants-Core/DotNest-Tenants-Core/src/Modules/Lombiq.Hosting.Tenants/Lombiq.Hosting.Tenants.EmailQuotaManagement/Services/EmailQuotaService.cs:75)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Lombiq.Hosting.Tenants.EmailQuotaManagement.Filters.DashboardQuotaFilter+<OnResultExecutionAsync>d__4.MoveNext (Lombiq.Hosting.Tenants.EmailQuotaManagement, Version=1.367.1.0, Culture=neutral, PublicKeyToken=null: /home/runner/work/DotNest-Tenants-Core/DotNest-Tenants-Core/src/Modules/Lombiq.Hosting.Tenants/Lombiq.Hosting.Tenants.EmailQuotaManagement/Filters/DashboardQuotaFilter.cs:37)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeNextResultFilterAsync>g__Awaited|30_0>d`2.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeNextResourceFilter>g__Awaited|25_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeFilterPipelineAsync>g__Awaited|20_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware+<Invoke>d__19.MoveNext (SixLabors.ImageSharp.Web, Version=3.0.0.0, Culture=neutral, PublicKeyToken=d998eea7b14cab13)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Media.Services.MediaFileStoreResolverMiddleware+<Invoke>d__7.MoveNext (OrchardCore.Media, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Liquid.ScriptsMiddleware+<Invoke>d__4.MoveNext (OrchardCore.Liquid, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__11.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DotNest.Hosting.Tenants.Middlewares.CustomDomainRedirectMiddleware+<InvokeAsync>d__3.MoveNext (DotNest.Hosting.Tenants, Version=1.367.1.0, Culture=neutral, PublicKeyToken=null: /home/runner/work/DotNest-Tenants-Core/DotNest-Tenants-Core/src/Modules/DotNest.Hosting.Tenants/Middlewares/CustomDomainRedirectMiddleware.cs:65)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware+<Invoke>d__5.MoveNext (Microsoft.AspNetCore.Localization, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Diagnostics.DiagnosticsStartupFilter+<>c__DisplayClass3_0+<<Configure>b__1>d.MoveNext (OrchardCore.Diagnostics, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware+<Invoke>d__3.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl+<<Invoke>g__Awaited|10_0>d.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl+<HandleException>d__11.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl+<<Invoke>g__Awaited|10_0>d.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.ContentPreview.PreviewStartupFilter+<>c+<<Configure>b__0_1>d.MoveNext (OrchardCore.ContentPreview, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Modules.ModularTenantRouterMiddleware+<Invoke>d__2.MoveNext (OrchardCore, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Modules.ModularTenantContainerMiddleware+<>c__DisplayClass4_0+<<Invoke>b__0>d.MoveNext (OrchardCore, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Scope.ShellScope+<UsingAsync>d__41.MoveNext (OrchardCore.Abstractions, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Scope.ShellScope+<UsingAsync>d__41.MoveNext (OrchardCore.Abstractions, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Scope.ShellScope+<UsingAsync>d__41.MoveNext (OrchardCore.Abstractions, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Scope.ShellScope+<UsingAsync>d__41.MoveNext (OrchardCore.Abstractions, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Scope.ShellScope+<UsingAsync>d__41.MoveNext (OrchardCore.Abstractions, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Modules.ModularTenantContainerMiddleware+<Invoke>d__4.MoveNext (OrchardCore, Version=1.8.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1+<ProcessRequestAsync>d__2.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

@Piedone
Copy link
Member

Piedone commented May 3, 2024

I have no local repro (neither with a local SQL Server nor with a remote Azure one) on v1.8.3 BTW, only prod (hosted in Azure, Azure SQL).

@MikeAlhayek
Copy link
Member

@Piedone i also cannot reproduce it locally. In fact, it’s not consistent in production so it hard to encounter it.

If you are able to, can you try @sebastienros theory #15290 (comment) on your production? So in your solution force using an older version of SqlClient package and see if that fixes it. Hopefully you can test it out since it seems to happens more in your environment

@Piedone
Copy link
Member

Piedone commented May 3, 2024

For us it's 100% consistent in prod, so yeah, that's a good starting point. I'll check.

@Piedone
Copy link
Member

Piedone commented May 3, 2024

I localized the issue to this line in a custom module of ours. As you can see, it's a simple ISession query. However, this runs on every admin request (from this filter), but it only fails on /Admin/Workflows/Types.

I added

<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.2" />

to the root web project, but this didn't fix the issue.

I've then taken v1.8.2 and changed only the removal of the using in the WorkflowTypeController, and used that as an overriding NuGet package. This fixes the issue.

I've also taken the other changes from your PR to make sure that it still works, and it does.

So, IMO no further fixes necessary and this can be closed.

@sebastienros
Copy link
Member

@Piedone FYI, not important but in the EmailQuotaService all the methods updating the quota instance also call SaveAsync but it's not necessary since it's either coming from the database or being created and then tracked. Which means that any change on this instance will be detected and saved automatically when SaveChagnesAsync() is invoked. But there are no side effects to call SaveAsync() since it will do a lookup, find the instance, and skip, it's just one extra async call for nothing.

@Piedone
Copy link
Member

Piedone commented May 3, 2024

Ah indeed, thank you!

@Piedone
Copy link
Member

Piedone commented May 7, 2024

So, as confirmed by @MikeAlhayek, the Workflows issue is indeed fixed. And this one here looks like an issue with the custom code.

So, closing it now; please reopen if still applicable.

@Piedone Piedone closed this as completed May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants