Use probe-stack=inline-asm in LLVM 11+

[erikdesjardins]

Fixes (?) #74405, related to #43241

r? @cuviper

[cuviper]

Thanks! This may have perf implications, so I'm excluding it from rollups.

@bors r+ rollup=never

[bors]

📌 Commit 95269c2d2ee2d59305d0599f0b389f4583cdb5ea has been approved by cuviper

[bors]

⌛ Testing commit 95269c2d2ee2d59305d0599f0b389f4583cdb5ea with merge 1272a1aeb3d394ebcf93f59298e04cd63d45e050...

[bors]

💔 Test failed - checks-actions

[cuviper]

The specific failures start here. I'm not sure why this change would cause new stack overflows, but it needs investigation.

failures:

---- [ui] ui/unsized-locals/by-value-trait-object-safety-rpass.rs stdout ----

error: test run failed!
status: signal: 6
command: "/checkout/obj/build/i686-unknown-linux-gnu/test/ui/unsized-locals/by-value-trait-object-safety-rpass/a"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------

thread 'main' has overflowed its stack
fatal runtime error: stack overflow

------------------------------------------


---- [ui] ui/unsized-locals/by-value-trait-object-safety-withdefault.rs stdout ----

error: test run failed!
status: signal: 6
command: "/checkout/obj/build/i686-unknown-linux-gnu/test/ui/unsized-locals/by-value-trait-object-safety-withdefault/a"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------

thread 'main' has overflowed its stack
fatal runtime error: stack overflow

------------------------------------------


---- [ui] ui/unsized-locals/autoderef.rs stdout ----

error: test run failed!
status: signal: 6
command: "/checkout/obj/build/i686-unknown-linux-gnu/test/ui/unsized-locals/autoderef/a"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------

thread 'main' has overflowed its stack
fatal runtime error: stack overflow

------------------------------------------



failures:
    [ui] ui/unsized-locals/autoderef.rs
    [ui] ui/unsized-locals/by-value-trait-object-safety-rpass.rs
    [ui] ui/unsized-locals/by-value-trait-object-safety-withdefault.rs

test result: FAILED. 10827 passed; 3 failed; 109 ignored; 0 measured; 0 filtered out

[erikdesjardins]

Looks like a comparison is inverted. This is a code snippet from by-value-trait-object-safety-rpass. Immediately before this, the size of the alloca is loaded into ebx, which in our case is 0 because the unsized trait object has size 0.

	add	ebx, 15
	and	ebx, -16
	mov	eax, esp
	sub	eax, ebx
	mov	dword ptr [ebp - 368], edx
	mov	dword ptr [ebp - 372], esi
	mov	dword ptr [ebp - 376], edi
	mov	dword ptr [ebp - 380], eax
.LBB87_43:
	mov	eax, dword ptr [ebp - 380]
	cmp	eax, esp
	jl	.LBB87_45
	mov	dword ptr [esp], 0
	sub	esp, 4096
	jmp	.LBB87_43
.LBB87_45:
	mov	eax, dword ptr [ebp - 380]
	mov	esp, eax

Due to:

cmp	eax, esp
jl	.LBB87_45

...if ebx is nonzero, the new stack pointer in eax will be below esp, and it will exit the loop without probing.
...if ebx is zero, the new stack pointer in eax will be equal to, and then greater than esp, and it will probe infinitely.
(and that's what happens running it locally under gdb)

The same thing shows up in Clang 11 with -fstack-clash-protection (https://godbolt.org/z/vxbaxd) with just:

int size;

void foo(void*);

int main() {
    foo(alloca(size));
}

I suppose nobody noticed this in C because it's rare to call alloca(0), and alloca(<nonzero>) will still run, it just won't actually probe anything.

Assuming I haven't misinterpreted something, can you report this to LLVM? I don't have an account yet.

[pietroalbini]

@bors r-

The PR failed but bors forgot about that during synchronize.

[cuviper]

Hmm, I thought we fixed that cmp direction in D82867, but it's possible we missed a case...
(edit: actually, I think that change may have mis-corrected the cmp for dynamic alloca... still investigating...)

[erikdesjardins]

Yeah, the changes to stack-clash-dynamic-alloca.ll in that revision look wrong; it was correct before (although maybe replace jl .LBB0_3 with jle to avoid probing for zero size allocas).
stack-clash-large.ll looks right if it's guaranteed that we always probe an exact multiple of the page size for static allocas.

[serge-sans-paille]

@erikdesjardins / @cuviper https://reviews.llvm.org/D90216 should do the trick. Thanks for spotting this!

[cuviper]

Ah, so there are two parts to this...

I suppose nobody noticed this in C because it's rare to call alloca(0),

This actually was already reported in bug 47657 and fixed in D88548.

and alloca(<nonzero>) will still run, it just won't actually probe anything.

This problem remains, but I just confirmed locally that it is fixed by D90216. I stepped through the assembly in gdb just to be sure. 🙂

I think both fixes would be good to have in LLVM 11.0.1. For Rust's part, we can backport the fixes to our bundled LLVM fork, but we should be careful about how this is enabled for external LLVM. I don't think we usually check the patch version for functionality, but maybe this one justifies it -- assuming 11.0.1 does get fixed.

[crlf0710]

Triage: So it seems this is blocked on a llvm bug.

[pnkfelix]

by the way, doesn't adding this mean that we are effectively no longer unit-testing our old stack-probe strategy, since we will just skip this test if the LLVM version is not sufficiently new?

Maybe we should have two files, one for LLVM < 11.0.1 and one for LLVM >= 11.0.1?

[nagisa]

We don't generally test very meticulously the setups that use LLVM versions outside of what is in the Rust repository. This is reflected in e.g. tests for all our (oft backported) soundness fixes only being applicable to the newest LLVM versions and our fork.

So far it has been the unwritten policy that it is up to the users who build with alternative LLVM versions to ensure that their builds do what they expect. (This is also another of those things that I should eventually look into codifying in some document outlining our LLVM support strategy)