Analyzing projects is extremely slow compared to MSBuild or Visual Studio

[DavidZidar]

I am working on a .NET Framework-project where a full build using all .NET 5 SDK analyzers takes about 40 seconds, Visual Studio can analyze the project in a similar amount of time.

OmniSharp on the other hand can analyze for hours on end and use many gigabytes of RAM. In fact, it never gets done because as I work things change so one CPU core is running hot all day long. I just had to kill the OmniSharp process because it was using 21 GB of RAM.

I have noticed this performance difference in other projects too, even in pure .NET Core projects. This has been the status quo for years.

So there seems to be a fundamental difference in how analysis is done in OmniSharp vs regular Roslyn since there is at least an order of magnitude or more difference in speed.

Would it be possible to involve Microsoft to get this properly fixed once and for all?

[kieranbenton]

Just to add my 2cents to this, I am experiencing exactly the same issue with a .NET 5 solution. Have been trying various things over literally the last year or so to improve the situation to no avail.

Have tried:

• Upgrading and running my own more recent version of mono locally (clears some omnisharp startup errors that I thought were implicated but performance is still really awful)
• Disabling CSX support (which was associated with instability at one point) as I have only a handful of scripts
• Restarting VSCode (full restart) a few times - this SOMETIMES will result in a mono instance that doesn't eat 100% of multiple cores and will settle down after 10s of minutes instead of spinning at 100% for forever

I did try running strace at one stage against the mono instance but I'm not familiar with linux internals to figure out whats going on.

This is all whilst running latest inside both WSL2 and a remote SSH (both Ubuntu 20.04) instances.

EDIT: Just to add - this is not a slight to omnisharp and the team, its an amazing piece of kit and VSCode is an absolute game changer. If this performance issue and a handful of other stability bugs could be fixed it would literally be perfect.

[seesharper]

One thing that affects this is Roslyn Analyzers. To disable them you could try this in omnisharp.json

{
  "RoslynExtensionsOptions": {
    "EnableAnalyzersSupport": false
}     

It is not ideal, but at least that makes the projects load much faster

[DavidZidar]

@seesharper Right, but Roslyn Analysers are active in VS too and I want them active for a reason. Disabling them is not a solution.

[DavidZidar]

From just observing the effects it seems like VS / MSBuild run all the analyzers at the same time in one pass but OmniSharp maybe runs them one at a time or maybe one file at a time instead of the whole project at once? This could perhaps explain the enormous performance difference.

[seesharper]

Agreed, it is really not a solution. When it comes to actually analyzing the projects I think Omnisharp just forwards this to the Roslyn libraries. Maybe it is the communication between OmniSharp and Vs Code that slows things down. I don't know. Just a thought

[kieranbenton]

I'm convinced that its some kind of interaction between omnisharp, mono and probably the solution/libraries I'm building. I've tried previously to build a smaller test solution (since I can't share my commercial source) to no avail.

I'm thinking that if I can catch it spinning at 100% CPU and capture logs/straces then that might be of help to someone more knowledgeable than myself?

[DavidZidar]

A year or two ago I tried to do some performance tracing with dotTrace but I didn't really know what to look for. I remember could see that it was the analyzers back then too. It must be possible to run them more efficiently somehow. Maybe Microsoft developers from the Roslyn project can help? They should be interested in making C# development experience great even in VS Code.

[kieranbenton]

Just to illustrate the problem - this is a screengrab from btop. I've just restarted VSCode about 10 minutes ago - and after a period of multicore usage (which would be fine) the usage settles down into mono alone continuously pegging at least one core to 100%.

Its this behaviour that then basically never goes away.

Running strace -p <monopid> results in obviously a huge amount of output - but from my untrained eye, it looks like the same sequence is just being repeated over and over with little variation:

futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x564fd31dd608, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGRT_3 {si_signo=SIGRT_3, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigprocmask(SIG_BLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigsuspend(~[RTMIN RT_1 RT_4], 8)    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGRT_4 {si_signo=SIGRT_4, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigreturn({mask=~[KILL STOP RTMIN RT_1]}) = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_UNBLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x564fd31dd608, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGRT_3 {si_signo=SIGRT_3, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigprocmask(SIG_BLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigsuspend(~[RTMIN RT_1 RT_4], 8)    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGRT_4 {si_signo=SIGRT_4, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigreturn({mask=~[KILL STOP RTMIN RT_1]}) = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_UNBLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x564fd31dd608, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGRT_3 {si_signo=SIGRT_3, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigprocmask(SIG_BLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigsuspend(~[RTMIN RT_1 RT_4], 8)    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGRT_4 {si_signo=SIGRT_4, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigreturn({mask=~[KILL STOP RTMIN RT_1]}) = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_UNBLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x564fd31dd608, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGRT_3 {si_signo=SIGRT_3, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigprocmask(SIG_BLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigsuspend(~[RTMIN RT_1 RT_4], 8)    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGRT_4 {si_signo=SIGRT_4, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigreturn({mask=~[KILL STOP RTMIN RT_1]}) = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_UNBLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x564fd31dd608, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGRT_3 {si_signo=SIGRT_3, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigprocmask(SIG_BLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigsuspend(~[RTMIN RT_1 RT_4], 8)    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGRT_4 {si_signo=SIGRT_4, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigreturn({mask=~[KILL STOP RTMIN RT_1]}) = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_UNBLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x564fd31dd608, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGRT_3 {si_signo=SIGRT_3, si_code=SI_TKILL, si_pid=11417, si_uid=1000} ---
rt_sigprocmask(SIG_BLOCK, [RT_4], NULL, 8) = 0
futex(0x564fd1b9aa00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigsuspend(~[RTMIN RT_1 RT_4], 8^Cstrace: Process 11417 detached

[seesharper]

@kieranbenton There is actually a version of OmniSharp that runs without mono. The code is in the https://github.com/OmniSharp/omnisharp-roslyn/tree/feature/omnisharp-vnext branch. You'll have to compile it yourself, but it might be an interesting experiment to see if similar problems occur when running on dotnet

[DavidZidar]

I'm on Windows but I tried the .NET 5 build on a smaller project at home:

• With the bundled version in VS Code project load took 12 seconds and code analysis took 30 seconds,
• With the .NET 5 version project project load took 10 seconds and code analysis still took 30 seconds.

I can try it on the bigger project at work tomorrow and maybe see if it ever finishes.

[DavidZidar]

For comparison, on the same project at home, running dotnet clean; dotnet build took just under 5 seconds.

[DavidZidar]

I wasn't able to use the .NET 5 OmniSharp at work because it's a .NET Framework project. I tried the .NET Framework v4.7.2 one in the feature/omnisharp-vnext branch but it does not seem any better.

[kieranbenton]

Ok, can I summarise and get some clarity here please? @DavidZidar are you saying your work project is a .NET Framework v4.7.2 project - and so you have to use the 'normal' mono Omnisharp build in order to interact with that?

And that the feature/omnisharp-vnext is a .NET 5 based (instead of mono) build of omnisharp @seesharper?

I don't understand @DavidZidar what you mean by "I tried the .NET Framework v4.7.2 one in the feature/omnisharp-vnext branch but it does not seem any better." since that seems counter to that branch being .NET 5 based?

If that is the case, then I will try and build vnext and run with that for a bit and report back.

[DavidZidar]

@kieranbenton feature/omnisharp-vnext contains several versions of OmniSharp that can be built, I built a net5.0 one using .\build.ps1 PublishNet5Builds which did work on my home project which is .NET 5 but it could not load my ASP.NET project at work so I tried .\build.ps1 PublishWindowsBuilds in that same branch which produces a build based on net472 but that one does not seem to show any improvements either.

[DavidZidar]

There is also a task to publish a Mono build in that branch which is confusing. Can you explain further @seesharper ?

[seesharper]

@DavidZidar I don't work on this repo, but my guess is that the vnext branch is simply an effort to make it work on net5.0 and that there is still some leftovers from the original branch. That being said, I have also tried this a couple of weeks back and could not really see a big difference in performance.

[kieranbenton]

Any idea how you get the vscode omnisharp extension to load Omnisharp using .NET 5 instead of mono? After overriding my "omnisharp.path" I'm getting...

Starting OmniSharp server at 10/7/2021, 12:16:30 PM
    Target: ...

OmniSharp server started with Mono 6.12.0.
    Path: /home/kieranbenton/source/microsoft/omnisharp-roslyn/artifacts/publish/OmniSharp.Stdio.Driver/linux-x64/net5.0/OmniSharp
    PID: 27884

Cannot open assembly '/home/kieranbenton/source/microsoft/omnisharp-roslyn/artifacts/publish/OmniSharp.Stdio.Driver/linux-x64/net5.0/OmniSharp': File does not contain a valid CIL image.

[DavidZidar]

@kieranbenton Maybe try adding .dll or .exe.

[kieranbenton]

Ok, I've got that working now - thanks to some help on the slack channel. FYI, you need to configure it as follows:

"omnisharp.useGlobalMono": "never",
	"omnisharp.path": "/home/<user>/source/microsoft/omnisharp-roslyn/artifacts/publish/OmniSharp.Stdio.Driver/linux-x64/net5.0/OmniSharp",

First report is that I believe that is significantly better.

I've restarted a few times this morning and after an initial surge of the analyzers obviously kicking in every time its reached steady state of 0% CPU after about 5 minutes.

I'll keep running like this for this week and report back.

[DavidZidar]

I don't know this code base but I attached dotTrace using timeline view and recorded 20 seconds of churn, it doesn't look great.

If I interpret what I'm seeing there seems to be a shared text writer that is blocking everything almost all the time so the analyzers are effectively running one at a time instead of in parallel.

[DavidZidar]

I tried letting the analyzers just go until completion at work and it took just over 20 minutes to do what MSBuild does in 40 seconds.

[DavidZidar]

I just had a little time to actually look in the code and the problem was not at all what I thought but rather more simple.

The problem is foreach-loops, like this one in particular (but I found 6 such loops):

omnisharp-roslyn/src/OmniSharp.Roslyn.CSharp/Workers/Diagnostics/CSharpDiagnosticWorkerWithAnalyzers.cs

Lines 245 to 250 in 1521fca

	foreach (var documentId in documentsGroupedByProject)
	{
	var document = project.GetDocument(documentId);
	var diagnostics = await AnalyzeDocument(project, allAnalyzers, compilation, workspaceAnalyzerOptions, document);
	UpdateCurrentDiagnostics(project, document, diagnostics);
	}

Which means it really is only analyzing one document at a time.

I tried replacing the above with this quick hack and now the analysis is actually using more of my cores and complete much faster:

await Task.WhenAll(documentsGroupedByProject.Select(async documentId =>
{
    var document = project.GetDocument(documentId);
    var diagnostics = await AnalyzeDocument(project, allAnalyzers, compilation, workspaceAnalyzerOptions, document);
    UpdateCurrentDiagnostics(project, document, diagnostics);
}));

That's probably too parallel though, it would be better to limit the amount being done to n number of threads where n might be the number of cores on the system. The new Parallel.ForEachAsync in .NET 6 might be perfect.

[kieranbenton]

Great work @DavidZidar! Definitely the new Parallel.ForEachAsync would be a perfect fit for that.

Out of interest, are you actually interested in analyzers running for the whole solution? I would much prefer if they would just run 'on demand' for the currently opened files. Otherwise even with making those code paths parallel, for a decently sized solution thats going to absolutely clobber multiple cores -> 100% until the whole solution is complete. In some ways only clobbering a single core is actually preferable to me at least... (can't believe I'm saying that).

[DavidZidar]

@kieranbenton Ideally it should be a toggle or a setting. Personally I prefer the whole solution to be analyzed, it's all to easy to miss warnings in files that aren't currently open when refactoring code. Also, I've got 16 cores, I want to put them to work. :)

[alexrp]

Personally I prefer the whole solution to be analyzed, it's all to easy to miss warnings in files that aren't currently open when refactoring code.

Strong agree fwiw. I've always found it very impractical that when I do manual refactoring in VS Code, I have to either force a full reanalysis or do a build to find out what broke. Would love to have a toggle for full solution analysis.

[TomasEkeli]

i've made an implementation that seems to work nicely for my machine
TomasEkeli@37b2fb0

it defaults to using half of the available cores, but allows configuration with a new key in omnisharp.json: RoslynExtensionsOptions.threadsToUseForAnalyzers

in my testing on a pretty large c# solution (32 projects) it is a lot quicker than it used to be and spins up the number of cores i configure it to.

[TomasEkeli]

Great work @DavidZidar! Definitely the new Parallel.ForEachAsync would be a perfect fit for that.

Out of interest, are you actually interested in analyzers running for the whole solution? I would much prefer if they would just run 'on demand' for the currently opened files. Otherwise even with making those code paths parallel, for a decently sized solution thats going to absolutely clobber multiple cores -> 100% until the whole solution is complete. In some ways only clobbering a single core is actually preferable to me at least... (can't believe I'm saying that).

isn't this analyze-only-open-files what the foreground worker does? #1507

if that is correct - then what you are asking for could essentially be an option to turn off the background worker, and just do the foreground analysis?

[kieranbenton]

@TomasEkeli yes, that sounds bang on - am I correct in assuming there is no way to turn off the background worker right now?

My workflow is effectively run a full project analyze once after I add a new analyzer - do any wide ranging cleanup for results I deem important enough to tackle up front, then crank the rest down to informational and fix as I work on individual files as part of other work.

Avoiding the 5 mins of 100% CPU on VSCode startup (even now after the vnext branch has landed - which seems much more stable) would be great.

[TomasEkeli]

@TomasEkeli yes, that sounds bang on - am I correct in assuming there is no way to turn off the background worker right now?

My workflow is effectively run a full project analyze once after I add a new analyzer - do any wide ranging cleanup for results I deem important enough to tackle up front, then crank the rest down to informational and fix as I work on individual files as part of other work.

Avoiding the 5 mins of 100% CPU on VSCode startup (even now after the vnext branch has landed - which seems much more stable) would be great.

can i suggest registering a new issue for that? i like the suggestion

[DaRosenberg]

I took a stab at this also in #2312, building upon the excellent work by @TomasEkeli in #2285 but with some further improvements.

[DavidZidar]

Thanks everyone that worked on this, it will help a lot! But seeing how this issue was closed entirely I have to say it doesn't really solve the underlying problem, OmniSharp is still using several times more processing power to do the same thing that MSBuild and Visual Studio do with much less. Using more cores is a good step on the way though!

[JohnGalt1717]

Ominisharp is still brutally slow for me and really flaky. The solution in question has 25 projects in it. Even when I tell it to only analyze open files, it still takes forever to analyze and even when done, it takes forever to autocomplete. VS.net is immediate on both accounts.

What's the solution for this?

[DavidZidar]

I'm not sure there is a solution currently, it's just a constant pain.

If only Microsoft could take VS Code seriously and rewrite OmniSharp and use the Roslyn API efficiently in the same way that Visual Studio does, that would be something.

[TomasEkeli]

that would be nice, but i doubt they will - the recent hot-reload kerfuffle showed that microsoft are still aiming to make visual-studio the better experience.