Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in MicroBenchmarks.Serializers.Xml_FromStream #70030

Closed
performanceautofiler bot opened this issue May 26, 2022 · 5 comments
Closed

Regressions in MicroBenchmarks.Serializers.Xml_FromStream #70030

performanceautofiler bot opened this issue May 26, 2022 · 5 comments
Labels
arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline 334989f6cc0600f9c12e9fa989dc1814c8300d1d
Compare 8626da3cc62d940513141b7c58c3495a87bb2bba
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 8.83 μs 9.63 μs 1.09 0.38 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.63240612988184 > 9.252880439142293.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/20/2022 7:26:52 AM, 5/26/2022 4:44:18 AM falls between 5/17/2022 2:28:28 PM and 5/26/2022 4:44:18 AM.
IsRegressionStdDev: Marked as regression because -14.15975450263905 (T) = (0 -9791.560434343146) / Math.Sqrt((2380.9791169829336 / (19)) + (108102.47519884912 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.11167310099712986 = (8807.949410272207 - 9791.560434343146) / 8807.949410272207 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 18.04
Baseline 334989f6cc0600f9c12e9fa989dc1814c8300d1d
Compare 8626da3cc62d940513141b7c58c3495a87bb2bba
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 16.79 μs 18.64 μs 1.11 0.24 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.636491856455297 > 17.5581706128025.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/20/2022 7:26:52 AM, 5/26/2022 4:44:18 AM falls between 5/17/2022 2:28:28 PM and 5/26/2022 4:44:18 AM.
IsRegressionStdDev: Marked as regression because -13.413048981135882 (T) = (0 -19548.663232683353) / Math.Sqrt((249503.4124652317 / (19)) + (582236.2114925911 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.15544941396592 = (16918.66644822232 - 19548.663232683353) / 16918.66644822232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels May 26, 2022
@kunalspathak kunalspathak changed the title [Perf] Changes at 5/20/2022 2:20:06 PM Regressions in MicroBenchmarks.Serializers.Xml_FromStream May 31, 2022
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues May 31, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label May 31, 2022
@ghost
Copy link

ghost commented May 31, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline 334989f6cc0600f9c12e9fa989dc1814c8300d1d
Compare 8626da3cc62d940513141b7c58c3495a87bb2bba
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 8.83 μs 9.63 μs 1.09 0.38 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.63240612988184 > 9.252880439142293.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/20/2022 7:26:52 AM, 5/26/2022 4:44:18 AM falls between 5/17/2022 2:28:28 PM and 5/26/2022 4:44:18 AM.
IsRegressionStdDev: Marked as regression because -14.15975450263905 (T) = (0 -9791.560434343146) / Math.Sqrt((2380.9791169829336 / (19)) + (108102.47519884912 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.11167310099712986 = (8807.949410272207 - 9791.560434343146) / 8807.949410272207 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 18.04
Baseline 334989f6cc0600f9c12e9fa989dc1814c8300d1d
Compare 8626da3cc62d940513141b7c58c3495a87bb2bba
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 16.79 μs 18.64 μs 1.11 0.24 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.636491856455297 > 17.5581706128025.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/20/2022 7:26:52 AM, 5/26/2022 4:44:18 AM falls between 5/17/2022 2:28:28 PM and 5/26/2022 4:44:18 AM.
IsRegressionStdDev: Marked as regression because -13.413048981135882 (T) = (0 -19548.663232683353) / Math.Sqrt((249503.4124652317 / (19)) + (582236.2114925911 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.15544941396592 = (16918.66644822232 - 19548.663232683353) / 16918.66644822232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: kunalspathak
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, ubuntu 18.04, RunKind=micro, Regression, CoreClr, arm64
Milestone: -

@kunalspathak
Copy link
Member

introduced by #69272 @eerhardt

@kunalspathak kunalspathak removed their assignment May 31, 2022
@kunalspathak kunalspathak added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels May 31, 2022
@kunalspathak
Copy link
Member

Another in dotnet/perf-autofiling-issues#5568

@kunalspathak kunalspathak mentioned this issue May 31, 2022
Closed
@eerhardt
Copy link
Member

Isn't this a dupe of #69730?

@kunalspathak
Copy link
Member

Looks like it. Our global test history is lagging behind, so didn't realize it.

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label May 31, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jun 30, 2022
@jeffhandley jeffhandley added arch-arm64 runtime-coreclr specific to the CoreCLR runtime and removed arm64 labels Dec 28, 2022
@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) and removed ubuntu 18.04 labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jeffhandley @eerhardt @kunalspathak