Change deserves_attention algorithm

[rylev]

This changes the deserves_attention algorithm in two ways:

• Before one "medium" sized result in secondary benchmarks would be considered deserving of attention. Now, the threshold is "large" sized results.
• Before, primary benchmark changes were considered twice as important as secondary. Now they are three times as important.

This was motivated by @pnkfelix's struggle with a lot of changes in the 2022-04-19 triage.

Under this change there would be

• 4 less regressions
• 4 less improvements
• 7 less mixed
• 8 less rollups

Unfortunately, this still would have likely been a nightmare for @pnkfelix as there still would have been 17 mixed results to go through...

Note: the regressions and mixed results (11 in total) would also not receive a perf-regression label either. Spot checking, it seems like most of these were found to be ignorable after investigation, so this change likely would only save time and not lead to actual regressions sneaking in.

For details on which PRs would no longer be included, see below:

Regressions

Rollup of 7 pull requests #95966 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	2	2	0	0	2
mean	0.2%	0.3%	N/A	N/A	0.2%
max	0.2%	0.4%	N/A	N/A	0.2%

Rollup of 4 pull requests #95999 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	3	0	0	0
mean	N/A	1.0%	N/A	N/A	N/A
max	N/A	1.2%	N/A	N/A	N/A

Rollup of 7 pull requests #96123 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	6	0	0	0
mean	N/A	0.4%	N/A	N/A	N/A
max	N/A	0.5%	N/A	N/A	N/A

Add slice::remainder #92287 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	2	0	0	0
mean	N/A	1.2%	N/A	N/A	N/A
max	N/A	1.2%	N/A	N/A	N/A

Improvements

Rollup of 5 pull requests #95974 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	0	0	7	0
mean	N/A	N/A	N/A	-0.3%	N/A
max	N/A	N/A	N/A	-0.6%	N/A

Allow self-profiler to only record potentially costly arguments when argument recording is turned on #95689 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	0	0	3	0
mean	N/A	N/A	N/A	-1.1%	N/A
max	N/A	N/A	N/A	-1.1%	N/A

Fix x test --doc --stage 0 library/std #95993 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	0	0	5	0
mean	N/A	N/A	N/A	-1.1%	N/A
max	N/A	N/A	N/A	-1.2%	N/A

Rollup of 6 pull requests #96134 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	0	0	3	0
mean	N/A	N/A	N/A	-1.1%	N/A
max	N/A	N/A	N/A	-1.1%	N/A

Mixed

Skip Lazy for some metadata tables #95867 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	5	0	1	0
mean	N/A	0.4%	N/A	-0.7%	N/A
max	N/A	0.5%	N/A	-0.7%	N/A

Rollup of 6 pull requests #96015 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	6	0	2	0
mean	N/A	0.7%	N/A	-0.3%	N/A
max	N/A	1.1%	N/A	-0.4%	N/A

Update cargo #96031 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	3	0	1	0
mean	N/A	1.1%	N/A	-0.4%	N/A
max	N/A	1.1%	N/A	-0.4%	N/A

Only check the compiler and standard library before documenting them (take 2) #95450 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	1	1	0	2	1
mean	0.9%	0.4%	N/A	-1.2%	0.9%
max	0.9%	0.4%	N/A	-1.2%	0.9%

Rollup of 9 pull requests #96108 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	3	0	2	0
mean	N/A	1.1%	N/A	-0.5%	N/A
max	N/A	1.1%	N/A	-0.7%	N/A

Rollup of 7 pull requests #96117 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	3	0	1	0
mean	N/A	1.1%	N/A	-0.4%	N/A
max	N/A	1.1%	N/A	-0.4%	N/A

Fix rustdoc duplicated blanket impls #96091 (Comparison Link)

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count	0	3	0	3	0
mean	N/A	1.1%	N/A	-1.1%	N/A
max	N/A	1.2%	N/A	-1.1%	N/A

[Kobzol]

From the linked results it looks like ctfe-stress-5 has been super noisy lately. These changes looks reasonable, the current bar seemed to be too low.

I think that an additional heuristic could be something like "amongst all regressions/improvements, were there at least two separate benchmarks"? Which could also help a bit with noisy benchmarks. But that's for another PR.

[Mark-Simulacrum]

I think we should consider hiding results from benchmarks we don't yet have a noise floor computed for - I think today we assume a constant noise floor (.2%?) but that's not necessarily true.

I'm not sure it would've helped in this case though.

[rylev]

I think we should consider hiding results from benchmarks we don't yet have a noise floor computed for - I think today we assume a constant noise floor (.2%?) but that's not necessarily true.

I'm not sure how many benchmarks are impacted by this, but we could consider it. I think 0.2% is actually somewhat high of a significance threshold and thus benchmarks are less likely to be considered significant if we don't have historical data.

[nnethercote]

I think that an additional heuristic could be something like "amongst all regressions/improvements, were there at least two separate benchmarks"?

There are several secondary benchmarks that are there to detect a very specific thing (e.g. issue-*). This change would make regressions involving them harder to detect.