separate codegen/LLVM from julia runtime

[JeffBezanson]

This is a follow-up to #39129. The current state of the PR is that by default, everything works as normal, except during the build we generate both libjulia-internal (which does not link against LLVM) and libjulia-codegen (which does). The loader attempts to load libjulia-codegen after libjulia-internal. If that fails, it populates the codegen entry points with fallbacks in libjulia-internal that do nothing. That means you can simply delete libjulia-codegen, start with --compile=no, and enjoy a compiler-free julia runtime (assuming you don't actually need the compiler of course 😄 ).

This probably still needs a bunch of cleanup, and it also tends to segfault with --compile=no, which I assume can be fixed.

All in all, the loader framework we have is quite a nice way to separate the system into multiple optional components, and I foresee doing more of that. The parser and front-end would be another logical component to separate (and then ultimately replace with a new implementation written in julia and separately compiled!). It would be nice if we could streamline adding plugins a bit --- naturally, we thought the loader would only ever load one thing, so a few too many places need to be modified to add another.

[jpsamaroo]

@JeffBezanson are you intending to expose something more flexible, like a CLI flag, to disable loading codegen?

[JeffBezanson]

That would be fine, it's just a bit tricky since when the loader runs we haven't looked at the command line arguments yet. We'd have to figure out how to reorganize things a bit.

[jpsamaroo]

Will dynamic dispatch still work when we just have the runtime available?

[JeffBezanson]

Yes, that is implemented in the runtime. The only parts of the language that actually need the compiler are ccall, cfunction and llvmcall.

[AriMKatz]

What's the size of the runtime? If Signatures are known, can those features be precompiled?

[JeffBezanson]

On my system:

-rwxrwxr-x 1 jeff jeff 11192464 Aug 30 17:09 libjulia-internal.so.1.8
-rwxrwxr-x 1 jeff jeff   266416 Aug 30 17:20 libjulia.so.1.8

but if you strip the files:

-rwxrwxr-x 1 jeff jeff  2757176 Aug 30 22:33 libjulia-internal.so.1.8
-rwxrwxr-x 1 jeff jeff   151096 Aug 30 22:34 libjulia.so.1.8

Yes, those features can be precompiled in a system image.

[imciner2]

The only parts of the language that actually need the compiler are ccall, cfunction and llvmcall.

What does this mean for the functionality that will be available if someone uses the runtime-only part? Will it mean you can't use packages that wrap C libraries? Or would those libraries work if they are pre-compiled still?

[JeffBezanson]

Anything precompiled will work.

[vchuravy]

The only parts of the language that actually need the compiler are ccall, cfunction and llvmcall.

For call and maybe cfunction libffi might provide an LLVM less solution

[JeffBezanson]

We are; I guess command line options are handled in libLLVMSupport?

[vtjnash]

CI is also now failing on macOS due to a conflict between this PR and the update to LLVM: https://build.julialang.org/#/builders/63/builds/4301

[tkf]

tell ASAN to "skip"

As Valentin said, we are loading (initializing?) LLVM twice somehow (and only ASAN complains this somehow?). I think I can dig into it this weekend (though I'm not the most efficient person for debugging this).

[vtjnash]

one possible solution is to detect if it is loaded twice:

diff --git a/cli/loader_lib.c b/cli/loader_lib.c
index abc144c41c..81bb25062d 100644
--- a/cli/loader_lib.c
+++ b/cli/loader_lib.c
@@ -197,6 +197,11 @@ __attribute__((constructor)) void jl_load_libjulia_internal(void) {
 
     // Unpack our special library names.  This is why ordering of library names matters.
     libjulia_internal = load_library(special_library_names[0], lib_dir, 1);
+
+    void (*pZN4llvm2cl22ResetCommandLineParserEv)(void) = (void (*)(void))lookup_symbol(libjulia_internal, "_ZN4llvm2cl22ResetCommandLineParserEv");
+    if (pZN4llvm2cl22ResetCommandLineParserEv)
+       pZN4llvm2cl22ResetCommandLineParserEv();
+
     void *libjulia_codegen = load_library(special_library_names[1], lib_dir, 0);
     const char * const * codegen_func_names;
     if (libjulia_codegen == NULL) {
@@ -219,7 +224,7 @@ __attribute__((constructor)) void jl_load_libjulia_internal(void) {
     }
     // jl_options must be initialized very early, in case an embedder sets some
     // values there before calling jl_init
-    ((void (*)())jl_init_options_addr)();
+    ((void (*)(void))jl_init_options_addr)();
 
     for (unsigned int symbol_idx=0; codegen_func_names[symbol_idx] != NULL; ++symbol_idx) {
         void *addr = lookup_symbol(libjulia_codegen, codegen_func_names[symbol_idx]);

[DilumAluthge]

I've opened an issue to track progress on fixing the asan failure: #42540

[chriselrod]

Git bisect blames this commit:

git bisect bad                                                                                 (base)
628209c1f2f746e3fc21ccd7cb34e67289403d44 is the first bad commit
commit 628209c1f2f746e3fc21ccd7cb34e67289403d44
Author: Jeff Bezanson <[email protected]>
Date:   Tue Oct 5 16:14:00 2021 -0400

    separate codegen/LLVM from julia runtime (#41936)

    separate libjulia-internal and libjulia-codegen

    makes codegen optional via a plugin interface

    Add special `@` character to `LOADER_BUILD_DEP_LIBS` and friends

    This allows us to mark a library as `"special"` so that the loader
    doesn't attempt to open it blindly, but rather interprets it as a list
    of positional arguments, that it knows how to interpret.  We strictly
    require exactly the number of special libraries, to help avoid errors in
    the future as we add to this list.

    Strip windows runtime symbols from the windows import library

    Without stripping these symbols out of the import library, we end up with duplicate symbol definition errors.

    Co-authored-by: Julian Samaroo <[email protected]>

    Co-authored-by: Elliot Saba <[email protected]>

for this ICE while building src/runtime_intrinsics.so:

> make                                                                                               (base)
removed './bin/7z'
    CC src/runtime_intrinsics.o
during RTL pass: expand
/home/chriselrod/Documents/languages/julia/src/runtime_intrinsics.c: In function 'jl_copysign_float':
/home/chriselrod/Documents/languages/julia/src/runtime_intrinsics.c:1065:1: internal compiler error: Segmentation fault
 1065 | }
      | ^
/home/chriselrod/Documents/languages/julia/src/runtime_intrinsics.c:1322:1: note: in expansion of macro 'bi_fintrinsic'
 1322 | bi_fintrinsic(copysign_float,copysign_float)
      | ^~~~~~~~~~~~~
0x8d91c9 crash_signal
        ../../gcc-11.2.0/gcc/toplev.c:327
0x7f14b05c5e4f ???
        ../sysdeps/unix/sysv/linux/libc_sigaction.c:10
0x1170fda mark_jump_label_1
        ../../gcc-11.2.0/gcc/jump.c:1097
0x117104f mark_jump_label_1
        ../../gcc-11.2.0/gcc/jump.c:1221
0x117104f mark_jump_label_1
        ../../gcc-11.2.0/gcc/jump.c:1221
0x1170cb3 mark_jump_label(rtx_def*, rtx_insn*, int)
        ../../gcc-11.2.0/gcc/jump.c:1084
0x1170cb3 mark_all_labels
        ../../gcc-11.2.0/gcc/jump.c:332
0x1170cb3 rebuild_jump_labels_1
        ../../gcc-11.2.0/gcc/jump.c:74
0x1087d5f execute
        ../../gcc-11.2.0/gcc/cfgexpand.c:6811
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
make[1]: *** [Makefile:215: runtime_intrinsics.o] Error 1
make: *** [Makefile:76: julia-src-release] Error 2

Maybe this is a bug in my gcc, or a function of compiler flags somehow?
I didn't get this on the other computers I've tried.

[DilumAluthge]

@chriselrod Can you open a new issue?

[chriselrod]

@chriselrod Can you open a new issue?

Done: #42588.

[pemryan]

re-implement ccall via libffi maybe a nice move.

I tried Julia 1.8.0 (2021-12-06) Commit 30fe8cc*, and found there exists such a dependency chain:

LinearAlgebra --> Linearlibblastrampoline_jll --> OpenBLAS_jll --> ccall --> compiler

which makes LLVM a must-have library at runtime.