App locks during tenant setup when using SqlServerCache as IDistributedCache.

[willnationsdev]

I need to set up a SQL Server-based IDistributedCache implementation in OC CMS. Can't use Redis, but I basically copy/pasted the RedisCacheWrapper and swapped out the cache implementations / prefixed all the keys with the tenant name to achieve tenant isolation in a centralized table. I introduce it using an Org.OrchardCore.Distributed module that I added to my startup recipes.

The problem is that if I attempt to create a new tenant, then the "Setup" process locks and never completes only when I have the SqlServerCache wrapper registered. I'm hoping someone can offer some direction/guidance on how to go about resolving this.

Here's a snapshot of my Visual Studio inspection of the threads' stack traces:

It reaches SmtpSettingsConfiguration.Configure, executes the GetSiteSettingsAsync, seemingly manages to eventually resume the async operation, but then gets hung up when attempting to resolve the IContentDefinitionManager service within the Map callback of TaxonomyIndexProvider. No idea why the Configure operation's lock doesn't get released by that point.

Since this is OC's code, I can't tell if I'm doing something wrong. Here's the cache implementation (a singleton service):

public class TenantDistributedCache : IDistributedCache
{
    private readonly IDistributedCache _cache;
    private readonly string _prefix;

    public TenantDistributedCache(IServiceProvider services, ShellSettings shellSettings)
    {
        var cache = (IDistributedCache?)services.GetService<SqlServerCache>();
        cache ??= services.GetRequiredService<MemoryDistributedCache>();
        _cache = cache;

        _prefix = shellSettings.Name + ":";
    }

    public byte[]? Get(string key) => _cache.Get(_prefix + key);

    public Task<byte[]?> GetAsync(string key, CancellationToken token = default) => _cache.GetAsync(_prefix + key, token);

    public void Refresh(string key) => _cache.Refresh(_prefix + key);

    public Task RefreshAsync(string key, CancellationToken token = default) => _cache.RefreshAsync(_prefix + key, token);

    public void Remove(string key) => _cache.Remove(_prefix + key);

    public Task RemoveAsync(string key, CancellationToken token = default) => _cache.RemoveAsync(_prefix + key, token);

    public void Set(string key, byte[] value, DistributedCacheEntryOptions options) => _cache.Set(_prefix + key, value, options);

    public Task SetAsync(string key, byte[] value, DistributedCacheEntryOptions options, CancellationToken token = default)
        => _cache.SetAsync(_prefix + key, value, options, token);
}

[Piedone]

I suggest you add Debug.WriteLine calls into each of the TenantDistributedCache methods, one before and after the _cache calls. That way, at least you'll see what the last thing that happened was, what might offer some hints.

What you describe seems like a deadlock, perhaps related to the sync-over-async necessary in the *Configuration classes.

You can BTW debug into OC, since the NuGet packages are built with SourceLink. Furthermore, you can try to reproduce this with the full OC source too.

Also see #16149.

[willnationsdev]

Update:

I discovered that if I remove some of the sub-recipes that are executed towards the end of my setup recipe (ones that define certain ContentTypes and some default ContentItems for them), then the deadlock never triggers. When I stripped out the "content" step from those nested recipes, then performing the setup of a new tenant stopped having deadlocks (yay). So, the issue seems somehow related to my attempts to create content items at the end of my setup recipe. And this deadlock only occurs if I have distributed features turned on (OrchardCore.Tenants.Distributed in the Default tenant and my distributed cache module in all tenants via being an activated "feature" within my setup recipe). Without it, these same content-generating sub-recipes were working fine in the setup process.

In addition, I added logs before/after each call. Sure enough, the logs proceed up through to the "content" step of the sub-recipes and then stop.

This may or may not be actually related, but I noticed in the logs that while I have successful examples of it running SetAsync for various documents containing generated IDs or JSON objects...(apple is just a sample tenant name, so it's prefixed to all keys)

TenantDistributedCache|DEBUG|Cache `SetAsync`
key='apple:OrchardCore.Workflows.Http.Models.WorkflowTypeRouteDocument',
value='{"Entries":{},"Identifier":"4q9vnabaqgvy1xj77nfh0jndxs"}' 
---
TenantDistributedCache|DEBUG|Cache `SetAsync`
key='apple:OrchardCore.Workflows.Http.Models.WorkflowTypeRouteDocument',
value='{"Entries":{},"Identifier":"4q9vnabaqgvy1xj77nfh0jndxs"}' COMPLETE
---
TenantDistributedCache|DEBUG|Cache `SetAsync`
key='apple:ID_OrchardCore.Workflows.Http.Models.WorkflowTypeRouteDocument',
value='4q9vnabaqgvy1xj77nfh0jndxs'
---
TenantDistributedCache|DEBUG|Cache `SetAsync`
key='apple:ID_OrchardCore.Workflows.Http.Models.WorkflowTypeRouteDocument',
value='4q9vnabaqgvy1xj77nfh0jndxs' COMPLETE 

...and it can successfully do GetAsync and RemoveAsync calls for the ContentDefinitionRecord document, the first time it attempts to run SetAsync on the ContentDefinitionRecord document, all of the text logged for the value turns seems to have a messed up encoding or something.

TenantDistributedCache|DEBUG|Cache `SetAsync`
key='apple:OrchardCore.ContentManagement.Metadata.Records.ContentDefinitionRecord',
value='���
...(I can't even paste it into GitHub successfully. It gets truncated).

I can work around the issue for now by diverting the "content" steps to a migration recipe associated with the related feature. Will post another update if I experience any other issues, but I'm still not sure why this deadlock occurs.

[Piedone]

You can also try putting those features steps at the end of the recipe, after the content step, that contain the distributed service enables.

I have no idea from this what the issue might be, but figuring out deadlocks is always fun! (And by that I mean they aren't fun at all. :))

[willnationsdev]

I'm gonna mark this discussion post as done since the root cause and a sensible way forward has been found. Will open other discussion posts for other issues. Thanks again for your grace and advice.

[Piedone]

No problem. I'm sorry I couldn't be of more help.