[Clang] --target=mips-linux-muslsf Generates Broken Code by Default

[Thraetaona]

These have been tested under Debian 12's default (i.e., Clang v14.0.6), but I do think that it continues to affect the latest version, too.
(It is worth mentioning that these bugs affect all variations of mips[eb]-[vendor]-linux-musl[sf] just the same.)

Before proceeding with the below steps as to replicate this bug, make sure that you install the following packages (commands are for Debian 12):

sudo apt install llvm clang
sudo apt install lld
sudo apt install libgcc1-mips-cross
sudo apt install gcc-mips-linux-gnu

I have went through all of these steps on an actual MIPS32r2 device (Qualcomm Atheros QCA9533), with an up-to-date musl libc ( 1.2.4-4) and Linux kernel (5.15.150)

---

Without a multitude of manual hacks to command-line switches, Clang will (by default) generate broken code for soft-float MIPS targets using musl libc:

clang -xc - --target=mips-linux-muslsf  -fuse-ld=lld -o test <<< '#include <stdio.h>
int main() { printf("Float number: %f \n", (float)5/2); return 0; }'

FIRST BUG: After doing chmod +x ./test on the target machine, you will be encountered with a ./test: not found error; file ./test indicates that the compiled program has been configured to use /lib/ld-musl-mips.so.1 as its interpreter, even though it had been compiled with mips-linux-muslsf (i.e., musl*sf*) and should have instead been using /lib/ld-musl-mips-sf.so.1.

As a workaround, we could hardcode this path into the final binary:

clang -xc - --target=mips-linux-muslsf -fuse-ld=lld  -o test \
-Wl,--dynamic-linker=/lib/ld-musl-mips-sf.so.1 <<< '#include <stdio.h>
int main() { printf("Float number: %f \n", (float)5/2); return 0; }'

SECOND BUG: However, a MIPS machine would still be unable to run our program; this time, the shell will throw a new error: Illegal instruction.

Despite the presence of sf in muslsf, Clang still seems to generate hard-float instructions for CPUs that otherwise lack an FPU, and this happens regardless of whether or not we specify an -march (e.g., -march=mips32r2 results in the same broken code). Attempting to specify either one of -msoft-float, -mfloat-abi=soft, or -mattr=+soft-float results in compilation failure:

clang -xc - --target=mips-linux-muslsf -fuse-ld=lld  -o test \
-Wl,--dynamic-linker=/lib/ld-musl-mips-sf.so.1 \
-msoft-float <<< '#include <stdio.h>
int main() { printf("Float number: %f \n", (float)5/2); return 0; }'

=================
ld.lld: error: /tmp/--471681.o: floating point ABI '-msoft-float' is incompatible with target floating point ABI '-mfpxx'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

THIRD BUG: For some reason, lld is given the -mfpxx option by default, which conflicts with -msoft-float. Fortunately, by manually supplying -flto, we can bypass this third issue:

clang -xc - --target=mips-linux-muslsf -fuse-ld=lld  -o test \
-Wl,--dynamic-linker=/lib/ld-musl-mips-sf.so.1 \
-msoft-float -flto <<< '#include <stdio.h>
int main() { printf("Float number: %f \n", (float)5/2); return 0; }'

FOURTH BUG: Due to the presence of #include <stdio.h>, one final bug occurs at this point: /usr/include/gnu/stubs.h:8:11: fatal error: '/usr/mips-linux-gnu/include/gnu/stubs-o32_soft.h' file not found. This file is neither existent nor needed and, in fact, mips-linux-gnu-gcc itself does not even include it in the first place. Indeed, a simple sudo touch /usr/mips-linux-gnu/include/gnu/stubs-o32_soft.h satisfies this "requirement," making the compilation (finally) successful:

sudo touch /usr/mips-linux-gnu/include/gnu/stubs-o32_soft.h && \
clang -xc - --target=mips-linux-muslsf -fuse-ld=lld  -o test \
-Wl,--dynamic-linker=/lib/ld-musl-mips-sf.so.1 \
-msoft-float -flto <<< '#include <stdio.h>
int main() { printf("Float number: %f \n", (float)5/2); return 0; }'

Finally, we get Clang to generate proper code that outputs the following: Float number: 2.500000

Summary:

1. Interpreter Path Bug: Clang incorrectly configures the interpreter path to /lib/ld-musl-mips.so.1 instead of /lib/ld-musl-mips-sf.so.1 for soft-float MIPS targets.;
2. Illegal FPU Instruction Bug: In the spite of -msoft-float flag, Clang generates hard-float instructions, resulting in an "illegal instruction" error on CPUs without an FPU;
3. Conflict with -mfpxx Option: Without -flto, lld automatically applies the -mfpxx option, which conflicts with the soft-float ABI and causes a compilation failure; and
4. Missing Header File Bug: The /usr/mips-linux-gnu/include/gnu/stubs-o32_soft.h file does not exist and is not required, yet Clang ceases compilation as long as a dummy (empty) file has not been provided.

[Thraetaona]

UPDATE: In addition to Clang-14 in the original post, I also got to test this under Clang-18 (Ubuntu 22.04); Clang-18 is actually more broken in this regard.

For the third bug, you are able to use -flto to bypass the -msoft-float vs. -mfpxx option clash in Clang-14; however, you cannnot do the same in Clang-18, because it always errors out with error: test.lto.o: floating point ABI '-msoft-float' is incompatible with target floating point ABI '-mfpxx'.

Other than this extra bug in Clang-18, the four other bugs replicate the same.

[brad0]

@yingopq @wzssyqa

[alexrp]

FWIW, note that prior to #107664, muslsf wasn't even recognized by LLVM/Clang. And it's an environment name that's mainly intended for LoongArch.