Windows Defender causing slowdown on contents of .cargo\bin folder

[krolli]

Hi,
a recently I noticed that running rustfmt is much slower than it used to be. Digging into it, I found out the slowdown is caused by Windows Defender (specifically MsMpEng.exe, Antimalware Service Executable), but is not limited to rustfmt. In fact, rustfmt in toolchain runs fast on its own, it's rustfmt binary in .cargo\bin directory that is causing large CPU usage spikes in scanning process. Setting Windows Defender to ignore rust-related directories or changing PATH environment variable to search directly in toolchain directory instead of taking the roundabout way through .cargo/bin dispatch binaries fixes the problem.

Since this affects all binaries, it also drags build performance down. I did some measurements on my test project (relatively small) with/without exception in Windows Defender and with direct path to toolchain and indirect path via .cargo:

full rebuild:
no exception, indirect path : 143s
exception, indirect path : 92s
no exception, direct path: 84s
exception, direct path: 83s

incremental build (single changed file):
no exception, indirect path: 4.6s (11.6s from button press)
exception, direct path: 2.8s (3.8s from button press)

Incremental build times are combination of cargo output value (shorter) and ETW trace measurement from pressing build button until link.exe is finished. Incremental build also includes running rustfmt (since the file gets saved).

edit: I tried signing rust binaries in both directories (toolchain and .cargo) with trusted certificate, but it didn't seem to help. However, it was my first time doing this, so I might have made a mistake somewhere.

[ehuss]

I have noticed this, too. The rustup wrapper itself adds a significant amount of time to execution.

Exe	With Defender	Without Defender
rustc -V	0.7s	0.09s
~/.rustup/toolchains/stable-x86_64-pc-windows-msvc/bin/rustc.exe -V	0.1s	0.08s
cl.exe	0.08s	0.056s
MinGW gcc --version	0.2s	0.11s

This may not be a cargo-specific issue. It would be nice if the rustup wrapper didn't add a nearly 8x slowdown.

I typically just turn it off while doing development. I don't know if there is a realistic workaround.

[krolli]

I thought this was a cargo thing, given ".cargo" folder name.

[alexcrichton]

Oh dear this seems bad! Is there a known way to fix this though? I just realized that I ended up disabling Windows Defender for the .cargo/.rustup directories presumably long ago...

[krolli]

I just heard from some people this happens even for nvidia gl dll, which is definitely signed.

I don't know of any way to convince Windows Defender to trust those binaries (short of contacting Microsoft). Workaround I am using is I just added path to toolchain directly into PATH variable, ahead of .cargo\bin, but I'm not sure why those dispatching binaries are there, so maybe this is not acceptable in general. Switching toolchains in rustup would then be just changing PATH variable. Cleaner way to do that might be to introduce another variable (eg. RUST_TOOLCHAIN) and just reference that from PATH using.

But, like I said, I'm not sure what is being solved by those dispatch binaries, so this might not be the right way.

edit: Also, I noticed that all of the .cargo\bin binaries are exactly the same. Maybe pulling most of their contents into dll that would be shared by all running processes would prevent it from being scanned multiple times. It would still be painful on single run though.

[steffengy]

@krolli
Is this related to the referenced rustup issue?
Try replacing your rustc.exe, rustup.exe or whatever slow rust binary you're using with https://win.rustup.rs/x86_64.

[krolli]

@steffengy
It is indeed faster. While adding exception to Defender and directly invoking rust tools still shaves off some time, it is not nearly as pronounced as before. I did a quick trace and it appears that scanning still occurs. It seems to take up about quarter of the original time, which is somewhat better improvement than I'd expect if the slowdown was linear in size of scanned binaries. Still, I think speedup is largely due to their smaller size.

[steffengy]

@krolli

Still, I think speedup is largely due to their smaller size.

It actually isn't.
i686-msvc is still slow, even though it's only 5MB.
x86-64-gnu (13 MB) is fast, even though the file size is even bigger.
So it's likely related to some architectural differences (and special treatment from windows defender).

Anyways those fixes landed in rustup.

[krolli]

@steffengy
I haven't tried i686-msvc, but I though I had x86-64-gnu installed before when it was really slow. I'll play around with it later to find out more.

[krolli]

@steffengy
You are right. Both gnu and msvc 32-bit binaries are taking way longer to scan. Looking at the results from profiler, it looks quite odd. Scanning itself is spending a bunch of extra time in things that are not there in fast case (eg. Lua VM and regex matching). But, there are things taking up majority of time in slow case with what seems to be erased callstack information...

[stale]

As there hasn't been any activity here in over 6 months I've marked this as stale and if no further activity happens for 7 days I will close it.

I'm a bot so this may be in error! If this issue should remain open, could someone (the author, a team member, or any interested party) please comment to that effect?

The team would be especially grateful if such a comment included details such as:

• Is this still relevant?
• If so, what is blocking it?
• Is it known what could be done to help move this forward?

Thank you for contributing!

(The cargo team is currently evaluating the use of Stale bot, and using #6035 as the tracking issue to gather feedback.)

If you're reading this comment from the distant future, fear not if this was closed automatically. If you believe it's still an issue please leave a comment and a team member can reopen this issue. Opening a new issue is also acceptable!

[retep998]

Is this still relevant?

Yes.

If so, what is blocking it?

The lack of a developer willing to put in the time and effort to get this fixed.

Is it known what could be done to help move this forward?

Someone could use a tool like procmon to compare how slow various syscalls are with and without Defender. A side by side comparison should make it clear which syscalls are impacted most by Defender and then the code can be adjusted to avoid those syscalls.

[weihanglo]

FWIW, #11917 might speed up this a bit with rust-lang/rustup#3178 enabled.

@rustbot label +S-needs-design