Regression in R2R compilation time

[sebastienros]

Linux and Windows, PlaintextPlatform benchmark in this example, Single file, Trimmed and Self-contained

Binlogs before/after show

msbuild.binlog.zip

[EgorBo]

@sebastienros is there a commit range? (I don't see an ability to view it on the e.g. Single File tab)

[EgorBo]

Likely, ffb2578...e6c1a49

[sebastienros]

That's what I have yes, we could probably go shorter by trying builds from within the range if you don't find anything

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario plaintext --profile intel-lin-app --profile intel-load-load --application.buildArguments "/p:PublishReadyToRun=true /p:PublishSingleFile=true" --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.4.24205.2 --application.runtimeVersion 9.0.0-preview.4.24204.3 --application.sdkVersion 9.0.100-preview.4.24208.11

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario plaintext --profile intel-lin-app --profile intel-load-load --application.buildArguments "/p:PublishReadyToRun=true /p:PublishSingleFile=true" --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.4.24208.6 --application.runtimeVersion 9.0.0-preview.4.24209.8 --application.sdkVersion 9.0.100-preview.4.24211.4

[EgorBo]

TP regressed on NativeAOT too:

[sebastienros]

Haven't had time to file this one, are you saying it's related? I wouldn't think so

[EgorBo]

Haven't had time to file this one, are you saying it's related? I wouldn't think so

Dates match

[EgorBo]

The thing is - when something goes wrong for AOT'd code - JIT can just re-jit it, while NAOT cannot, so it explains why TP only got hurt for NAOT.

[sebastienros]

Here are the commands for the AOT one:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.DotNet.ILCompiler=9.0.0-preview.4.24204.3 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.4.24205.2 --application.runtimeVersion 9.0.0-preview.4.24204.3 --application.sdkVersion 9.0.100-preview.4.24208.11

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.DotNet.ILCompiler=9.0.0-preview.4.24209.5 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.4.24208.6 --application.runtimeVersion 9.0.0-preview.4.24209.5 --application.sdkVersion 9.0.100-preview.4.24209.32

[EgorBo]

The main suspect is #99982

[sebastienros]

Minimal changeset identified for the nativeaot benchmark:

a8c2a5e...404b286

[EgorBo]

Oh, so #100728 then?

[sebastienros]

nativeaot may be special, not sure if moving the runtime version without the corresponding sdk might change the outcome. But at least I set the framework reference and Microsoft.DotNet.ILCompiler to the same runtime. On 9.0.0-preview.4.24208.9+a8c2a5e5bbc2 we get 398K RPS and on 9.0.0-preview.4.24208.10+404b286b2309 only 178K RPS.

[stephentoub]

Oh, so #100728 then?

Really??

[MichalPetryka]

Oh, so #100728 then?

Really??

@SingleAccretion suspects this to be the reason since CastCache is a mutable struct.

Feels like there should be a builtin analyzer warning on readonly fields and in parameters with non-readonly structs for this.

[stephentoub]

Ugh. I looked for each such type I wasn't sure about, and I swear I saw it was a class, but I probably incorrectly mistook

runtime/src/coreclr/vm/castcache.h

Line 48 in 7b18be5

class CastCache

for it as I was searching.

[stephentoub]

Feels like there should be a builtin analyzer warning on readonly fields and in parameters with non-readonly structs for this.

#33773

dotnet/roslyn-analyzers#2811

dotnet/roslyn-analyzers#2831

#50389

[jkotas]

Fixed by #100996

[EgorBo]

It doesn't look like it's solved, e.g. NAOT run from today is still slow and build time is still slow for R2R

[EgorBo]

@sebastienros can you take a look if it's solved

[sebastienros]

I confirm the NAOT one is fixed, I had to set --application.packageReferences Microsoft.DotNet.ILCompiler=9.0.0-preview.4.24215.2 as I assume it hasn't been ingested by the installer yet. But the build times are still slow. We should check again when we know this is in installer builds.

[sebastienros]

All fixed, thank you!

R2R

NAOT