[OC 1.8.2] Slow saving of content item

[MikeKry]

After update to OC 1.8.2 from OC 1.6 (also from 1.7). Saving of content item became slower than before.

For same content item, it takes ~5s to save on OC 1.6. Saving on OC 1.8 takes ~15s. It is 3-5 times slower even on small content items without widgets.

I tried to search for feature that could do this, but without luck. Maybe memory leaks fix can have something to do with this?

[MikeKry]

It seems it might be related to media - library is very slow, maybe it has something in common.

[MikeAlhayek]

@MikeKry can you please try to use the Mini profiler feature to see what is actually taking longer than usual to process?

Also, you may want to try a simple site using blog recipe and see if you get the same slow behavior just to ensure it is OC and not anything specific to your project?

[MikeKry]

@MikeAlhayek

I tried mini profiler, it seems there are multiple duplicate queries when saving items. I tried also to disable all project specific modules, but it has no effect.

Here is export from mini profiler after saving content item that is basically empty.
OrchardCore.Contents_Admin_Edit (6107.8 ms) - Profiling Results.pdf

[MikeKry]

@MikeAlhayek

I tried to compare it to older project, most queries look exactly the same. But all of them are slower.

Deletions from indexes 22 ms vs 4164ms. Updates 1-5 ms vs 30-50 ms.

So it seems that database is the problem. But both projects are running on same server, so it might be something in yessql? You did not notice this problem yet?

[Skrypt]

Do you have the AuditTrail feature enabled and do you use Workflows? You may need to prune records in these tables as they tend to grow a lot and cause perf issues.

[MikeKry]

@Skrypt

It is completely new and empty project. Only difference between fast and slow CMS is version of OrchardCore nugets. Also I am using MSSQL server.

So it does not happen to anyone else when using 1.8.2 version?

[Skrypt]

Not at all. I'm using SQL Server latest version in a Docker container locally and having no issue with perf so far. You may want to check the performance of executing SQL queries on the SQL Server with SQL Server Management or Azure Data Studio. Maybe you are experiencing network latency issues. Or, you need to configure SQL Server to use named pipes vs tcp/ip.

[MikeKry]

@Skrypt

Ok, I will try to do some more debugging. So far I have found out that it only happens when using MSSQL. I tried SQLite and it works ok.

[MikeKry]

Here are results for MSSQL:
MSSQL_OrchardCore.Contents_Admin_Edit (11368.77 ms) - Profiling Results.pdf

and for SQLITE:
SQLITE_OrchardCore.Contents_Admin_Edit (4426.16 ms) - Profiling Results.pdf

Same page, same setup - this time I run on main branch of OrchardCore repo. I have tested given queries directly in MSSQL and it seems that problem is not in database itself.

Note that page is pretty big - with nested widgets etc. There are lots of calls to content pickers and media pickers. Each of them takes some time

[Skrypt]

A full height page screenshot would give an idea of the size of that FlowPart editor. I'm assuming that it is what you are using. Then, you may have handlers work happening in there.

[MikeKry]

@Skrypt

Sure here it is.

screen

I would like to highlight that also empty pages of this content type are slower for me. Also tests on mssql vs sqlite are on same content item.

[Skrypt]

Ok, could you provide us with a repro solution with a database backup so that we can investigate where the perf issue is?

[MikeKry]

@Skrypt

Not sure I can share that. I will check.

But I have performed another tests and narrowed down suspicious parts. Tests have been performed on new content type which contains only flow part.

1/ Content types - slowing down 5 times
When I upload a bunch of content types, everytime I save something it gets slower. It seems that content type definitions get loaded every time from database and it is slow.

When I activate filesystem content definition, speed is OK. (same as it was on versions before)

So i think this is the root cause, maybe some incorrect transition to GetTypeDefinitionAsync?

2/ Workflows + HTTP Workflows Activities are slowing down page save by 300 ms

3/ some other modules like content indexing and audit trail, but they have only a slight impact.

[Skrypt]

Try updating the ContentDefinitions in your DB from the file. Maybe it is really slow trying to parse that JSON because there is an issue with it. But that would be a first, normally it would simply fail to parse it. Else, the issue seams to be also related with caching the ContentTypeDefinitions if it does a SQL Query every time (which should not be the case).

If you are using Redis try to clear that cache too.

[MikeKry]

@Skrypt

I have prepared setup for debugging here (it is just a deployment plan export)

In my experience, when importing to completely empty solution, it slows down drastically. Even when importing only content types. But activating features slows it even more.

It happens only for mssql.

Edit:
Test case that i tried is just saving the empty single content item that is included in export.

[Skrypt]

I'd probably need to have that content item that is slow else it may take time for me to reproduce by creating lots of content like you did there. So far, everything is still quick on my side.

[MikeKry]

@Skrypt

Okay, that is interesting - screen you sent is OK.

But this is results I get.. are you also on main branch of orchardcore?

and this is sqlite version

[MikeKry]

if it works for you, maybe it is just a coincidence and the problem will be in our database connection. I will try to check it again.. But still seems odd that loading gets slower after importing content types..

[MikeKry]

@Skrypt

OK. Main problem is definitely in connection to database. It has ~30 ms latency... So sorry for wasting your time and thanks for helping me figure this one out.

Anyway, I wonder why total time increases with each content type..?

[sebastienros]

Lots of queries (something we might want to look into based on your example) and 30ms latency between your local machine and the azure database so that explains. When you app is on azure in the same region it should be much faster. Check that it's the case.