Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove secondary caching in ContentDefinitionManager, improve error message in DataMigrationManager #16212

Closed
wants to merge 30 commits into from

Conversation

hyzx86
Copy link
Contributor

@hyzx86 hyzx86 commented Jun 2, 2024

relate :#15989 (comment)

relate: #16214

@hyzx86 hyzx86 marked this pull request as ready for review June 2, 2024 17:21
if (_scopedTypeDefinitions.TryGetValue(name, out var typeDefinition))
{
return typeDefinition;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sebastienros should be the problem here, the Loadxx family of methods should no longer fetch content from the cache
/cc @MikeAlhayek

Copy link
Contributor Author

@hyzx86 hyzx86 Jun 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @MikeAlhayek , do I need to revert these changes? Or do I keep them?
Because there is already a cache in the ContentDefinitionStore and a second-level cache would make the logic more complex

Copy link
Contributor Author

@hyzx86 hyzx86 Jun 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we use both MemoryCache and Redis caching, it may have problems enabling distributed caching and multi-server deployments

@hyzx86 hyzx86 changed the title try fix ConcurrencyException Attempting to fix a ConcurrencyException raised by ContentDefinitionManager Jun 2, 2024
@hyzx86 hyzx86 marked this pull request as draft June 2, 2024 17:44
@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 2, 2024

Found some other issues, should I just change it here? Or are there other considerations?
The logic here seems a bit complicated 🤣

private ContentTypeDefinition LoadTypeDefinition(ContentDefinitionRecord document, string name) =>
!_scopedTypeDefinitions.TryGetValue(name, out var typeDefinition)
? _scopedTypeDefinitions[name] = Build(
document.ContentTypeDefinitionRecords.FirstOrDefault(type => type.Name.EqualsOrdinalIgnoreCase(name)),
document.ContentPartDefinitionRecords)
: typeDefinition;
private ContentTypeDefinition GetTypeDefinition(ContentDefinitionRecord document, string name) =>
_cachedTypeDefinitions.GetOrAdd(name, name => Build(
document.ContentTypeDefinitionRecords.FirstOrDefault(type => type.Name.EqualsOrdinalIgnoreCase(name)),
document.ContentPartDefinitionRecords));
private ContentPartDefinition LoadPartDefinition(ContentDefinitionRecord document, string name) =>
!_scopedPartDefinitions.TryGetValue(name, out var partDefinition)
? _scopedPartDefinitions[name] = Build(
document.ContentPartDefinitionRecords.FirstOrDefault(part => part.Name.EqualsOrdinalIgnoreCase(name)))
: partDefinition;

@hyzx86

This comment was marked as off-topic.

@hyzx86 hyzx86 marked this pull request as ready for review June 2, 2024 17:55
@hyzx86

This comment was marked as off-topic.

@hyzx86 hyzx86 marked this pull request as draft June 3, 2024 07:21
@hyzx86 hyzx86 marked this pull request as ready for review June 5, 2024 09:34
@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 5, 2024

Haven't solved the problem I'm having yet, but I think it's really a bug

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 5, 2024

This is because the DeploymentPlanIndex table is not created when performing the migration in OrchardCore.Contents.
I added OrchardCore.Deployment to the dependency list of the OrchardCore.Contents function and it solved the problem, but I don't think it's the best solution!

Error while running migration version 0 for 'OrchardCore.Contents'. 
OrchardCore.Recipes.Models.RecipeExecutionException: 
SQLite Error 1: 'no such table: DeploymentPlanIndex'.

Error while running migration version 0 for 'OrchardCore.Contents'. OrchardCore.Recipes.Models.RecipeExecutionException: SQLite Error 1: 'no such table: DeploymentPlanIndex'.
 ---> Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 1: 'no such table: DeploymentPlanIndex'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteCommand.PrepareAndEnumerateStatements()+MoveNext()
   at Microsoft.Data.Sqlite.SqliteCommand.GetStatements()+MoveNext()
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 434
   at YesSql.Store.<>c__DisplayClass44_0`2.<<ProduceAwaitedAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at YesSql.Data.WorkDispatcher`2.ScheduleAsync[TState](TKey key, TState state, Func`3 valueFactory)
   at YesSql.Store.ProduceAwaitedAsync[T,TState](WorkerQueryKey key, Func`3 work, TState state)
   at YesSql.Services.DefaultQuery.Query`1.ListImpl()
   at YesSql.Services.DefaultQuery.Query`1.ListImpl()
   at OrchardCore.Deployment.DeploymentPlanService.CreateOrUpdateDeploymentPlansAsync(IEnumerable`1 deploymentPlans) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore.Modules\OrchardCore.Deployment\DeploymentPlanService.cs:line 97
   at OrchardCore.Recipes.Services.RecipeExecutor.<>c__DisplayClass8_0.<<ExecuteStepAsync>b__0>d.MoveNext() in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 188
--- End of stack trace from previous location ---
   at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute, Boolean activateShell) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Abstractions\Shell\Scope\ShellScope.cs:line 275
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteStepAsync(RecipeExecutionContext recipeStep) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 171
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 103
   --- End of inner exception stack trace ---
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 128
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 147
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 157
   at EasyOC.Core.Recipes.EocRecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\EasyOC\src\Core\EasyOC.Core\Recipes\EocRecipeExecutor.cs:line 32
   at OrchardCore.Recipes.Services.RecipeMigrator.ExecuteAsync(String recipeFileName, IDataMigration migration) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeMigrator.cs:line 55
   at OrchardCore.Contents.Deployment.ExportContentToDeploymentTarget.ExportContentToDeploymentTargetMigrations.CreateAsync() in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore.Modules\OrchardCore.Contents\Deployment\ExportContentToDeploymentTarget\ExportContentToDeploymentTargetMigrations.cs:line 31
   at OrchardCore.Data.Migration.DataMigrationManager.InvokeMethodAsync(MethodInfo method, IDataMigration migration) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Data.YesSql\Migration\DataMigrationManager.cs:line 254
   at OrchardCore.Data.Migration.DataMigrationManager.UpdateAsync(String featureId) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Data.YesSql\Migration\DataMigrationManager.cs:line 216    at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 128
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 147
   at OrchardCore.Recipes.Services.RecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeExecutor.cs:line 157
   at EasyOC.Core.Recipes.EocRecipeExecutor.ExecuteAsync(String executionId, RecipeDescriptor recipeDescriptor, IDictionary`2 environment, CancellationToken cancellationToken) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\EasyOC\src\Core\EasyOC.Core\Recipes\EocRecipeExecutor.cs:line 32
   at OrchardCore.Recipes.Services.RecipeMigrator.ExecuteAsync(String recipeFileName, IDataMigration migration) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Recipes.Core\Services\RecipeMigrator.cs:line 55
   at OrchardCore.Contents.Deployment.ExportContentToDeploymentTarget.ExportContentToDeploymentTargetMigrations.CreateAsync() in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore.Modules\OrchardCore.Contents\Deployment\ExportContentToDeploymentTarget\ExportContentToDeploymentTargetMigrations.cs:line 31
   at OrchardCore.Data.Migration.DataMigrationManager.InvokeMethodAsync(MethodInfo method, IDataMigration migration) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Data.YesSql\Migration\DataMigrationManager.cs:line 254
   at OrchardCore.Data.Migration.DataMigrationManager.UpdateAsync(String featureId) in D:\SourceCodes\JZSoft\EasyOC.OrchardCore\OrchardCore\src\OrchardCore\OrchardCore.Data.YesSql\Migration\DataMigrationManager.cs:line 216

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 5, 2024

Debugging reveals that OrchardCore.Contents.Deployment.ExportContentToDeploymentTarget.ExportContentToDeploymentTargetStartup is run before OrchardCore.Deployment.Startup

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 5, 2024

Hi @gvkries , noticed that you've recently tweaked in #15793 , the feature loading logic, will that affect this?

@gvkries
Copy link
Contributor

gvkries commented Jun 5, 2024

I tried to keep startup ordering the same, but who knows 🙈

@hyzx86

This comment was marked as off-topic.

@gvkries
Copy link
Contributor

gvkries commented Jun 5, 2024

I don't think the startup order between extensions was ever guaranteed, if no dependencies were set.

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 5, 2024

I don't think the startup order between extensions was ever guaranteed, if no dependencies were set.

Yes, this dependency is really more of a headache

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 5, 2024

There seems to be a problem here, it has to do with #15793

/// </summary>
private IDataMigration[] GetDataMigrations(string featureId)
{
var migrations = _dataMigrations
.Where(dm => _typeFeatureProvider.GetFeaturesForDependency(dm.GetType()).Any(feature => feature.Id == featureId))
.ToArray();
return migrations;
}
/// <summary>

When OrchardCore.Contents is passed in, it recognises its submodule's migration type as its own, e.g. ExportContentToDeploymentTargetMigrations, which is actually the ExportContentToDeploymentTarget is part of the ExportContentToDeploymentTarget feature.

src/OrchardCore.Modules/OrchardCore.Contents/Deployment/ExportContentToDeploymentTarget/ExportContentToDeploymentTargetMigrations.cs

@hyzx86 hyzx86 marked this pull request as draft June 6, 2024 06:49
Comment on lines 54 to 58
"OrchardCore Deployment",
"OrchardCore.Deployment.Remote",
"OrchardCore.Contents.Deployment.AddToDeploymentPlan",
"OrchardCore.Contents.Deployment.Download",
"OrchardCore.Contents.Deployment.ExportContentToDeploymentTarget",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After changing the blog recipe to this, using the blog recipe when adding tenants will reproduce the problem
@gvkries

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks. I think I understand the problem. Give me a sec...

@gvkries
Copy link
Contributor

gvkries commented Jun 6, 2024

I think a quick fix for the problem is to use the last feature for migrations, instead of the first. But we may need a way to explicitly force a type like migrations to only have a single feature assigned.

Try the following changes to the DataMigrationManager

In line 92, use the last feature:

public async Task<IEnumerable<string>> GetFeaturesThatNeedUpdateAsync()
{
    var currentVersions = (await GetDataMigrationRecordAsync()).DataMigrations
        .ToDictionary(r => r.DataMigrationClass);

    var outOfDateMigrations = _dataMigrations.Where(dataMigration =>
    {
        Records.DataMigration record;
        if (currentVersions.TryGetValue(dataMigration.GetType().FullName, out record) && record.Version.HasValue)
        {
            return CreateUpgradeLookupTable(dataMigration).ContainsKey(record.Version.Value);
        }

        return GetMethod(dataMigration, "Create") != null;
    });

    return outOfDateMigrations.Select(m => _typeFeatureProvider.GetFeaturesForDependency(m.GetType()).Last().Id).ToArray();
}

And on line 279 as well:

private IDataMigration[] GetDataMigrations(string featureId)
{
    var migrations = _dataMigrations
            .Where(dm => _typeFeatureProvider.GetFeaturesForDependency(dm.GetType()).Last().Id == featureId)
            .ToArray();

    return migrations;
}

I will think of a better way to solve this, e.g. some marker interface to force types to have a single feature only.

@hyzx86 hyzx86 marked this pull request as ready for review June 6, 2024 08:50
@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 6, 2024

Try the following changes to the DataMigrationManager

It worked! 🎉🎉

Thank you !

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 6, 2024

Here's the recipe that went wrong, and I'm now going to restore the blog recipe change.

blog.recipe.json The full recipe is too long, only the `feature` step have been taken.
{
  "steps": [
    {
      "name": "feature",
      "enable": [
        // SaaS
        "OrchardCore.HomeRoute",
        "OrchardCore.Admin",
        "OrchardCore.Diagnostics",
        "OrchardCore.DynamicCache",
        "OrchardCore.Features",
        "OrchardCore.Navigation",
        "OrchardCore.Recipes",
        "OrchardCore.Resources",
        "OrchardCore.Roles",
        "OrchardCore.Security",
        "OrchardCore.Settings",
        "OrchardCore.Themes",
        "OrchardCore.Users",

        // Content Management
        "OrchardCore.Alias",
        "OrchardCore.AdminMenu",
        "OrchardCore.Autoroute",
        "OrchardCore.Html",
        "OrchardCore.ContentFields",
        "OrchardCore.ContentPreview",
        "OrchardCore.Contents",
        "OrchardCore Deployment",
        "OrchardCore.Deployment.Remote",
// start
        "OrchardCore.Contents.Deployment.AddToDeploymentPlan",
        "OrchardCore.Contents.Deployment.Download",
        "OrchardCore.Contents.Deployment.ExportContentToDeploymentTarget",
// end
        "OrchardCore.ContentTypes",
        "OrchardCore.CustomSettings",
        "OrchardCore.Feeds",
        "OrchardCore.Flows",
        "OrchardCore.Indexing",
        "OrchardCore.Layers",
        "OrchardCore.Lists",
        "OrchardCore.Markdown",
        "OrchardCore.Media",
        "OrchardCore.Menu",
        "OrchardCore.Placements",
        "OrchardCore.Queries",
        "OrchardCore.Queries.Sql",
        "OrchardCore.Shortcodes.Templates",
        "OrchardCore.Rules",
        "OrchardCore.Taxonomies",
        "OrchardCore.Title",
        "OrchardCore.Templates",
        "OrchardCore.Widgets",
        // Themes
        "TheBlogTheme",
        "TheAdmin",
        "SafeMode"
      ]
    }
  ]
}

@gvkries
Copy link
Contributor

gvkries commented Jun 6, 2024

I have created #16257 with a different approach for fixing this problem. Please check it out, if you don't mind.

@hyzx86 hyzx86 requested a review from MikeAlhayek June 6, 2024 14:50
@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 6, 2024

I reverted some logic changes to the DataMigrationManager file in this PR, which will be addressed in PR #16257

@hyzx86 hyzx86 changed the title Attempting to fix a ConcurrencyException raised by ContentDefinitionManager Remove secondary caching in ContentDefinitionManager, improve error alerts in DataMigrationManager Jun 6, 2024
@hyzx86 hyzx86 changed the title Remove secondary caching in ContentDefinitionManager, improve error alerts in DataMigrationManager Remove secondary caching in ContentDefinitionManager, improve error message in DataMigrationManager Jun 6, 2024
@sebastienros
Copy link
Member

Let's wait until #16257 is merged to see if you can still repro the issue. I'd rather not change this code unless there are repro steps.

@hyzx86
Copy link
Contributor Author

hyzx86 commented Jun 6, 2024

Let's wait until #16257 is merged to see if you can still repro the issue. I'd rather not change this code unless there are repro

I've tested it and it doesn't reproduce the problem.

@hyzx86 hyzx86 closed this Jun 6, 2024
@gvkries
Copy link
Contributor

gvkries commented Jun 7, 2024

Thanks @hyzx86 for all your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants