apfloat bug: incorrect handling of infinities in FMA

[RalfJung]

The Rust port of LLVM's apfloat has a bug in its FMA (fused-multiply-add) implementation. See rust-lang/miri#2468 (comment) for details, but basically it fails this assertion:

    let neg_inf: f32 = f32::NEG_INFINITY;
    assert_eq!((-3.2f32).mul_add(2.4, neg_inf), neg_inf);

AFAIK we don't use apfloat FMA anywhere (Miri stopped using them to work around this bug), but this should still be tracked, and will need to be fixed before we can allow FMA in const.

[eddyb]

Copying the relevant parts here (it's a clumsy mistake I made when porting):

• LLVM's APFloat has an unconditional sign multiplication first
• rustc_apfloat instead only does it outside of handling infinities etc.

Seems like this bug was there from the start (of the port), I don't really understand how it happened (but obviously I screwed up the otherwise-simple early-return control-flow refactor) - what worries me a bit more is that I heavily relied on tests and there's barely any for FMA (they're all finite AFAICT!).
And it's not like I lost some tests during porting, they only had that many.

[eddyb]

... (it's a clumsy mistake I made when porting):

• LLVM's APFloat has an unconditional sign multiplication first
• rustc_apfloat instead only does it outside of handling infinities etc.

Turns out it's not that simple. I'm not actually doing the same thing as the C++ code, but instead I'm bypassing their multiplySpecials distinction entirely and using the normal multiplication facilities.

So it's more likely a case of missing some subtler special case because of the odd refactor.

_{(I'm not sure entirely why I did it that way, but FMA is at least special in one way: it needs more precision, and [Limb; 2] instead of [Limb; 1], which multiplySignificand handles in the C++ version by whether addend is NULL, i.e. it has FMA and non-FMA modes, but I inlined multiplySignificand to remove the dynamic choice which was handled in the C++ code with dynamic heap allocations)}

---

EDIT: yupp, the bug is that status == Status::OK should be status != Status::INVALID_OP, because the Rust version can set more flags than the C++ one could - OTOH, now I have a bunch of FMA testcases that all trip the same assertion in sig::add_or_sub (assert(!carry); in the original C++ code).

EDIT2: it was a bit tricky to enable C++ assertions in my custom build, but it does repro:

rustc_apfloat-fuzz: /home/eddy/Projects/rustc_apfloat/target/debug/build/rustc_apfloat-fuzz-a919140aae98d3e2/out/llvm-project-f3598e8fca83ccfb11f58ec7957c229e349765e3/
  llvm/lib/Support/APFloat.cpp:1482: llvm::lostFraction llvm::detail::IEEEFloat::addOrSubtractSignificand(const llvm::detail::IEEEFloat &, bool):
    Assertion `!carry' failed.

My best guess for that one is that it likely got fixed in llvm/llvm-project@e62555c.

EDIT3: ignore the status == Status::OK comment, the real problem/solution is subtler: FMA of the form finite * finite + infinite must ignore the finite * finite multiplication entirely, instead of letting it overflow (and that seems to deal with all the FMA mismatches found by fuzzing).

EDIT4: the assertion I mentioned earlier might be the same bug as #93224.

[eddyb]

I posted a longer update to #55993 (comment) regarding the progress with the whole rustc_apfloat situation, but of note for this issue is that the WIP repo already has this bug fixed.