non-temporal stores: use inline assembly #1541

RalfJung · 2024-02-25T16:30:16Z

LLVM treats !nontemporal as just a hint on store operations, which is unsound -- they have a totally different semantics, similar to atomic memory orderings. So I'd like to avoid any risk of that causing any issues by entirely avoiding their !nontemporal attribute. Is it acceptable to use inline assembly to implement these intrinsics?

Note that this is my first time ever writing inline assembly, so the code may or may not make any sense.^^

rustbot · 2024-02-25T16:30:20Z

r? @Amanieu

rustbot has assigned @Amanieu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Amanieu · 2024-02-25T16:48:47Z

My understanding is that LLVM can turn a nontemporal store into a normal one, but not the other way around. This seems to be fine as far as I understand.

The CI failure happens because the target_feature attribute only enables sse2 and rustc isn't smart enough to figure out that this implies sse (only LLVM knowns that). You fix it by enabling the sse feature as well.

RalfJung · 2024-02-25T16:53:59Z

My understanding is that LLVM can turn a nontemporal store into a normal one, but not the other way around. This seems to be fine as far as I understand.

It's completely unclear. LangRef talks about it as a hint:

The optional !nontemporal metadata must reference a single metadata name <nontemp_node> corresponding to a metadata node with one i32 entry of value 1. The existence of the !nontemporal metadata on the instruction tells the optimizer and code generator that this load is not expected to be reused in the cache. The code generator may select special instructions to save cache bandwidth, such as the MOVNT instruction on x86.

That would mean the flag can be added or removed arbitrarily ("this load is not expected to be reused in the cache" -- but no semantic constraints or anything). But that's clearly wrong. LLVM doesn't acknowledge in the slightest the extra UB that can be caused by non-temporal stores (llvm/llvm-project#64521). Therefore I have zero confidence that anyone thought about how !nontemporal interacts with all the LLVM passes that work on load (almost all of which probably just ignore the attribute entirely). I'm not even aware of any cross-platform memory model with support for nontemporal stores that they could be using here -- and they clearly need a cross-platform memory model since they are doing optimizations in the context of a C++11-style model.

RalfJung · 2024-02-25T17:03:41Z

SDE ERROR:  TID: 1064 executed instruction with an unaligned memory reference to address 0x7f27229035e0 INSTR: 0x562d8a5e21f3: IFORM: VMOVNTPS_MEMf32_ZMMf32_AVX512 :: vmovntps zmmword ptr [rax], zmm0
	IMAGE:    /checkout/target/x86_64-unknown-linux-gnu/release/deps/core_arch-59198cd2fc79a24a
	FUNCTION: _ZN9core_arch9core_arch3x867avx512f5tests20test_mm512_stream_ps20test_mm512_stream_ps17hb7f0b28acc824410E.llvm.13799798511543115899
	FUNCTION ADDR: 0x562d8a5e21c0

Hm, yes, this requires alignment, but that shouldn't be new...?

RalfJung · 2024-02-28T17:39:41Z

crates/core_arch/src/x86/avx512f.rs

        struct Memory {
-            pub data: [f32; 16],
+            pub data: [f32; 16], // 64 bytes
        }


This test should have failed many times already. The only explanation I have for why that did not happen is that maybe LLVM optimizes away the entire test...

Maybe the whole stack frame gets 64-byte aligned, since there are __m512 values involved.

If that happened it would also happen with this PR.

The stack frame is probably different because you are missing the nostack option on the inline assembly. In fact these assembly blocks should include both nostack and preserves_flags.

I'm pretty sure the test was just buggy before this PR. It doesn't actually ensure that the data is properly aligned.

In fact these assembly blocks should include both nostack and preserves_flags.

I have added the options in the last commit.

RalfJung · 2024-06-21T12:17:07Z

I think I found where LLVM defines the x86 intrinsics: https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/IR/IntrinsicsX86.td.

I found nothing with "stream" in the name, and the only "movnt" is int_x86_mmx_movnt_dq, probably accessible via llvm.x86.mmx.movnt.dq, which I assume is not the right thing.

There seem to be already quite a few asm! in stdarch so I guess using that here is acceptable? IMO it's better than just using normal loads since presumably people actually want the streaming semantics when using this operation.

RalfJung · 2024-06-21T15:18:01Z

Awesome, thanks. :)

After the next stdarch bump we can then remove the intrinsic from rustc.

Amanieu · 2024-06-21T16:02:18Z

The intrinsic is only broken on x86, it still has value on other targets.

RalfJung · 2024-06-21T21:18:06Z

Hm, fair. Maybe we should then document the intrinsic as "it is semantically equivalent to a regular load, just a hint", and on x86 actually compile it to just a load since that architecture doesn't have a "just a hint" version of this. For all other architectures we'd have to check whether what LLVM does there is sensible or not.

RalfJung · 2024-07-13T16:59:48Z

@Amanieu any chance we could get a stdarch bump in the rustc repo that includes this change? :)

Amanieu · 2024-07-13T17:41:38Z

We're waiting on a bootstrap bump that should happen next week.

…ouxu,Amanieu,Jubilee nontemporal_store: make sure that the intrinsic is truly just a hint The `!nontemporal` flag for stores in LLVM *sounds* like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is [highly dubious](llvm/llvm-project#64521). Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores. ~~Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the `_mm_stream` intrinsics are unaffected by this change.~~ Fixes rust-lang#114582 Cc `@Amanieu` `@workingjubilee`

Rollup merge of rust-lang#128149 - RalfJung:nontemporal_store, r=jieyouxu,Amanieu,Jubilee nontemporal_store: make sure that the intrinsic is truly just a hint The `!nontemporal` flag for stores in LLVM *sounds* like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is [highly dubious](llvm/llvm-project#64521). Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores. ~~Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the `_mm_stream` intrinsics are unaffected by this change.~~ Fixes rust-lang#114582 Cc `@Amanieu` `@workingjubilee`

…ieu,Jubilee nontemporal_store: make sure that the intrinsic is truly just a hint The `!nontemporal` flag for stores in LLVM *sounds* like it is just a hint, but actually, it is not -- at least on x86, non-temporal stores need very special treatment by the programmer or else the Rust memory model breaks down. LLVM still treats these stores as-if they were normal stores for optimizations, which is [highly dubious](llvm/llvm-project#64521). Let's avoid all that dubiousness by making our own non-temporal stores be truly just a hint, which is possible on some targets (e.g. ARM). On all other targets, non-temporal stores become regular stores. ~~Blocked on rust-lang/stdarch#1541 propagating to the rustc repo, to make sure the `_mm_stream` intrinsics are unaffected by this change.~~ Fixes rust-lang/rust#114582 Cc `@Amanieu` `@workingjubilee`

rustbot assigned Amanieu Feb 25, 2024

RalfJung force-pushed the movnt branch from f4acd83 to 343b318 Compare February 25, 2024 16:31

RalfJung force-pushed the movnt branch from 343b318 to e5bfd0d Compare February 25, 2024 16:54

RalfJung mentioned this pull request Feb 25, 2024

Non-temporal stores (and _mm_stream operations in stdarch) break our memory model rust-lang/rust#114582

Closed

RalfJung commented Feb 28, 2024

View reviewed changes

RalfJung added 2 commits June 21, 2024 16:29

non-temporal stores: use inline assembly

26bddf5

fix test_mm512_stream_ps test

bf0d024

RalfJung force-pushed the movnt branch from cdd1d53 to bf0d024 Compare June 21, 2024 14:29

set asm attributes

3e10d09

Amanieu enabled auto-merge (rebase) June 21, 2024 14:38

Amanieu merged commit fd5fc64 into rust-lang:master Jun 21, 2024
29 checks passed

RalfJung deleted the movnt branch June 21, 2024 15:17

RalfJung mentioned this pull request Jul 24, 2024

nontemporal_store: make sure that the intrinsic is truly just a hint rust-lang/rust#128149

Merged

sayantn mentioned this pull request Aug 3, 2024

Fix _mm_stream_si64 #1623

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

non-temporal stores: use inline assembly #1541

non-temporal stores: use inline assembly #1541

RalfJung commented Feb 25, 2024 •

edited

Loading

rustbot commented Feb 25, 2024

Amanieu commented Feb 25, 2024

RalfJung commented Feb 25, 2024 •

edited

Loading

RalfJung commented Feb 25, 2024

RalfJung Feb 28, 2024

eduardosm Feb 28, 2024

RalfJung Feb 28, 2024

Amanieu Jun 21, 2024

RalfJung Jun 21, 2024

RalfJung Jun 21, 2024

RalfJung commented Jun 21, 2024

RalfJung commented Jun 21, 2024

Amanieu commented Jun 21, 2024

RalfJung commented Jun 21, 2024

RalfJung commented Jul 13, 2024

Amanieu commented Jul 13, 2024

non-temporal stores: use inline assembly #1541

non-temporal stores: use inline assembly #1541

Conversation

RalfJung commented Feb 25, 2024 • edited Loading

rustbot commented Feb 25, 2024

Amanieu commented Feb 25, 2024

RalfJung commented Feb 25, 2024 • edited Loading

RalfJung commented Feb 25, 2024

RalfJung Feb 28, 2024

Choose a reason for hiding this comment

eduardosm Feb 28, 2024

Choose a reason for hiding this comment

RalfJung Feb 28, 2024

Choose a reason for hiding this comment

Amanieu Jun 21, 2024

Choose a reason for hiding this comment

RalfJung Jun 21, 2024

Choose a reason for hiding this comment

RalfJung Jun 21, 2024

Choose a reason for hiding this comment

RalfJung commented Jun 21, 2024

RalfJung commented Jun 21, 2024

Amanieu commented Jun 21, 2024

RalfJung commented Jun 21, 2024

RalfJung commented Jul 13, 2024

Amanieu commented Jul 13, 2024

RalfJung commented Feb 25, 2024 •

edited

Loading

RalfJung commented Feb 25, 2024 •

edited

Loading