windows-gnu target compatibility with toolchains providing LLVM tools only

[mati865]

Rust currently assumes all GNU libraries and tools are present. This is not the case when using LLVM only toolchains like this one: https://github.com/mstorsjo/llvm-mingw

Remaining issues:

• LLD doesn't work with version-scripts (MinGW: enable dllexport/dllimport #72049)
• Rust explicitly links libgcc, libgcc_eh, libgcc_s

Opening the issue so interested people can track it easily.

[mati865]

cc @petrochenkov for the second issue, we just need possibility of providing custom late_link_args_dynamic and late_link_args_static.

[petrochenkov]

@mati865
Do you mean ability to pass -C link-args-dynamic / -C link-args-static on the command line in addition to the target spec, or ability to drop the target spec args and start from scratch with regular -C link-args?

[mati865]

@petrochenkov in this specific case we need to replace libgcc with compiler-rt and libgcc_eh or libgcc_s with libunwind. late_link_args hopefully could be left intact but I have to verify that.
Adding new target would do the trick but it might be an overkill if things can work with existing target.

[petrochenkov]

So, I think this is a job for #[link(cfg(...))] (if we want to avoid a new target).

One cfg feature is provided by the compiler and allows to discern between the link_args_dynamic and link_args_static at link time.

Another cfg feature is a regular cargo feature of the src/libunwind crate which allows to switch between "LLVM toolchain" (compiler-rt + libunwind) and "GCC toolchain" (libgcc + libgcc_eh).

Then 8e3467c is reverted and all the linking is done through #[link] attributes in src/libunwind.

#[link(name = "gcc_s", cfg(all(rustc_detect_dynamic, gcc_toolchain)))]
#[link(name = "gcc_eh", cfg(all(not(rustc_detect_dynamic), gcc_toolchain)))]
#[link(name = "unwind", cfg(not(gcc_toolchain)))]
extern {}

or something.

[petrochenkov]

This way we'll eliminate late_link_args_(dynamic,static) stuff from target specs which felt like a hack to me since its introduction.

Before 8e3467c #![no_std] programs (or rather programs not depending on src/libunwind) didn't link to the exception support libraries, which seems like the right thing to me.

With these options moved to the target spec they apply to all programs, even no-std ones.
(In this sense any libraries linked from target specs seem at least suspicious.)

[mati865]

I've tried similar approach in #71001
That makes i686 executables depend on having libgcc_s-dw2-1.dll (or libunwind.dll) either in PATH or next to them.

I wanted to avoid new target if possible because we will definitely need new target to allow linking with UCRT instead of MSVCRT since one cannot mix libraries build with UCRT and MSVCRT.
C/C++ toolchains using UCRT already exists in the wild and I'm already being asked when it'll be possible to use Rust built libraries with them.

EDIT: For x86_64 we could even build libunwind as part of std and link it statically but this somehow results in errors because proc_macro crates are no longer built (just like previously on musl with +crt-static).

[petrochenkov]

#71001 didn't use cfg(rustc_detect_dynamic) proposed in #72241 (comment) though.
With it executables won't depend on gcc_s.

[mati865]

Wait, so we can use features here not only when building std but also when building user code? 😮

[petrochenkov]

@mati865
Yes, cfgs in #[link] are serialized into crate metadata "as is" and evaluated only when the final binary is build.
That's how cfg(target_feature = "crt-static") works, for example.

[petrochenkov]

cc #72510

[petrochenkov]

@nu8
I've seen the __ms_vsnprintf error a few days ago when building rustc using gcc toolchain (https://packages.msys2.org/package/mingw-w64-x86_64-gcc) (so I had to switch to msvc).
So it seems like it isn't specific to LLVM toolchain, some recent changes just broke mingw in general.

[mati865]

No it's not specific to LLVM at all.
This error happens when you mix UCRT and MSVCRT and the only solution for it is to never mix different CRTs.
Rust doesn't provide UCRT based toolchain (I mentioned it in #72241 (comment)). For that to happen we need more recent mingw-w64 and maybe even built to target UCRT by default.

I've seen the __ms_vsnprintf error a few days ago when building rustc using gcc toolchain (https://packages.msys2.org/package/mingw-w64-x86_64-gcc) (so I had to switch to msvc).

@petrochenkov few days ago MSYS2 upgraded mingw-w64 and it broke winpthreads library. Fix was committed on the next day I think.

Issue described by @nu8 is caused by using toolchain from llvm-mingw project.
That toolchain is using LLVM only tools and targets UCRT. either of those can cause build to fail because neither is currently supported by Rust.

[mati865]

@petrochenkov I reproduced your issue by upgrading mingw-w64 among the other packages...
That's because MinGW doesn't seem to have stable order of it's linkage.

Currently Rust does it this way:

rust/src/librustc_target/spec/windows_gnu_base.rs

Lines 20 to 38 in 64ad709

	late_link_args.insert(
	LinkerFlavor::Gcc,
	vec![
	"-lmingwex".to_string(),
	"-lmingw32".to_string(),
	"-lmsvcrt".to_string(),
	// mingw's msvcrt is a weird hybrid import library and static library.
	// And it seems that the linker fails to use import symbols from msvcrt
	// that are required from functions in msvcrt in certain cases. For example
	// `_fmode` that is used by an implementation of `__p__fmode` in x86_64.
	// Listing the library twice seems to fix that, and seems to also be done
	// by mingw's gcc (Though not sure if it's done on purpose, or by mistake).
	//
	// See https://github.com/rust-lang/rust/pull/47483
	"-lmsvcrt".to_string(),
	"-luser32".to_string(),
	"-lkernel32".to_string(),
	],
	);

Now compare it to GCC which adjusts their list as need, last adjustment happened 2 weeks ago in gcc-mirror/gcc@850533a.
Interesting parts of gcc hello.c -###:
-lmingw32 -lgcc -lgcc_eh -lmoldname -lmingwex -lmsvcrt -lkernel32 -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc -lgcc_eh -lmoldname -lmingwex -lmsvcrt -lkernel32

Clang solves it by using --start-group and --end-group around mingw-w64 libs.
Interesting parts of clang hello.c -###:
-lmingw32 -lgcc -lgcc_eh -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc -lgcc_eh -lmoldname -lmingwex -lmsvcrt

At this point I don't know whether Rust should:

• follow GCC by adjusting that list everytime something breaks,
• follow Clang and use --start-group and --end-group (there is performance hit here)
• say "I'm out of this mess" by setting no_default_libraries to false and letting the linker handle it

I'm currently looking for workaround to make stage0 std build...

[petrochenkov]

follow GCC by adjusting that list everytime something breaks

Hmm, shouldn't it be possible to make a "union" list that would work both pre- and post-gcc-mirror/gcc@850533a by passing some libraries multiple times?
Which would be similar to what linker does with --(start,end)-group, except that we would be doing it manually.

--(start,end)-group would make it harder to move these libraries to #[link] attributes in libc or libstd crates from the target spec, which would be nice to do because it's more in line with what other targets do.

[mati865]

That seems to do the trick for now:

index f556bf03f02..744f26239ca 100644
--- a/src/librustc_target/spec/windows_gnu_base.rs
+++ b/src/librustc_target/spec/windows_gnu_base.rs
@@ -20,9 +20,9 @@ pub fn opts() -> TargetOptions {
     late_link_args.insert(
         LinkerFlavor::Gcc,
         vec![
+            "-lmsvcrt".to_string(),
             "-lmingwex".to_string(),
             "-lmingw32".to_string(),
-            "-lmsvcrt".to_string(),
             // mingw's msvcrt is a weird hybrid import library and static library.
             // And it seems that the linker fails to use import symbols from msvcrt
             // that are required from functions in msvcrt in certain cases. For example

For stage0 to work you can build Rust with RUSTFLAGS_BOOTSTRAP='-C link-arg=-lmsvcrt' ./x.py build.

Generally GCC passes unnecessary libs like totally empty -lmoldname.
It would be nice if there was a tool to create dependency graph for import/static libraries so we could confidently choose the order instead of current "let's see if that order works".

[mati865]

@nu8

However with Clang, at least with this version:

mstorsjo/llvm-mingw

I havent had to do that at all.

Maybe it uses --start-group and --end-group implicitly for user libraries?

I think Clang uses this:

http://theory.stanford.edu/~nikolaj/programmingz3.html

to resolve the Libraries.

MSYS2 ships LLVM with z3, llvm-mingw builds it without that.

Maybe that brings some overhead, but Clang seems about
30% faster to me as well.

It depends on the use case but Clang is often just faster than GCC.

[mati865]

Opened #73184 to fix __ms_vsnprintf issue building Rust.

[mati865]

I wanted to avoid new target if possible because we will definitely need new target to allow linking with UCRT instead of MSVCRT since one cannot mix libraries build with UCRT and MSVCRT.

Okay, seems like this will need separate target (GCC's emutls vs Clang's native TLS is a good example).
Instead we could avoid creating separate MSVCRT vs UCRT targets for each compiler until somebody needs them. Windows-gnu would keep MSVCRT as the default and new MinGW clang toolchain would use UCRT.

Should I close this issue and open another one or MCP for the new target?

[mstorsjo]

However with Clang, at least with this version:
mstorsjo/llvm-mingw
I havent had to do that at all.

Maybe it uses --start-group and --end-group implicitly for user libraries?

As LLD's COFF backend is modelled after MS link.exe, it handles static libs in the same way as that tool, which means that it looks in all referenced static libs for undefined references - pretty much as if --start-group/--end-group was wrapped around the full list of arguments. (And Clang passes those arguments around the libs to the linker, to solve that issue when using ld.bfd as well.)

I think Clang uses this:
http://theory.stanford.edu/~nikolaj/programmingz3.html
to resolve the Libraries.

MSYS2 ships LLVM with z3, llvm-mingw builds it without that.

Z3 is entirely unrelated to linking. A quick grep through the source tree would point to it being used by the clang static analyzer, possibly...

Maybe that brings some overhead, but Clang seems about
30% faster to me as well.

It depends on the use case but Clang is often just faster than GCC.

In my experience, Clang isn't that fast in itself any longer, it's often similar to GCC or slower, for compilation. But for linking, LLD is often an order of magnitude faster than ld.bfd. This is especially noticable for large C++ projects - see e.g. https://bugs.llvm.org/show_bug.cgi?id=42217#c17, where a user said "I can finally link my project with lld. This reduces link time from ~26s to ~1s. Amazing!"

[kleisauke]

@mati865 Thanks for adding these *-pc-windows-gnullvm targets! I was wondering if there is also interest in the i686-pc-windows-gnullvm and armv7-pc-windows-gnullvm targets? If so, I could try to upstream these patches:
https://github.com/libvips/build-win64-mxe/blob/master/build/plugins/llvm-mingw/patches/rust-1-fixes.patch
https://github.com/libvips/build-win64-mxe/blob/9ffefdfb91117cb679cd1ed08a0f0f20f6366ed3/build/patches/librsvg-llvm-mingw.patch#L52-L191

(I'm able to test the (unusual) armv7-pc-windows-gnullvm target on my Raspberry Pi 3B with Windows 10 IoT, if necessary)

[mati865]

@kleisauke sorry, thought I had already anwsered here.
32-bit LLVM targets are blocked right now on having working unwinding. It doesn't manifest right away when cross-compilling but prevents bootstrapping Rust natively.

[tschuett]

LLVM (https://github.com/llvm/llvm-project/blob/main/llvm-libgcc/docs/LLVMLibgcc.rst) hosts theses strange:
libgcc, libgcc_eh, libgcc_s
files.

[mati865]

I doubt it helps in any way. Implementation details for Windows Dwarf 2 between libgcc and libunwind differ a lot and that library doesn't seem to take it into account.

[mstorsjo]

LLVM (https://github.com/llvm/llvm-project/blob/main/llvm-libgcc/docs/LLVMLibgcc.rst) hosts theses strange:
libgcc, libgcc_eh, libgcc_s
files.

They're not really of any value for this setup. Those files are used for letting libunwind (and compiler-rt) fill the gap where preexisting binaries on Linux have a dependency on e.g. libgcc_s.so, to make libunwind and compiler-rt, which provide the same symbols, provide a library interface for ABI compatibility.