RFC: take libclang as a dependency

[graydon]

Integrating libclang into our llvm build is a "very likely candidate" for solving all of the following issues:

• Building and compiling C code with rustc/cargo #1850 Building and compiling C code with rustc/cargo (and sub-ticket rustc should support compiling C code #1862)
• Add swig binding and/or C-compiler-fe-based bindings generator #104 Add swig binding and/or C-compiler-fe-based bindings generator (see also: Adopt rust-bindgen #2045 Adopt rust-bindgen)
• OS X Universal Binary Support #2024 OS X Universal Binary Support (clang knows how to operate lipo on OSX)
• support LLVM subtargets #1879 support LLVM subtargets
• Use msvc on windows, not mingw #1768 Use msvc on windows, not mingw

I have a submodule'd version of the requisite pieces in my workspace here and it seems to work ok. It makes initial llvm build time longer and will involve expanding the librustllvm we ship (or shipping an additional lib). Not sure exactly how much, size-wise, but it seems to be a bit "pay as you go", so for example I think we could start using its driver module (that knows how to run various toolchains on different platforms) before we did anything concerning bindgen or compiling C code through our own libclang (vs. handing to the 'native' compiler).

Ideally, going down this road could eventually free us from using external toolchains at all; that is, once LLVM has its own linker, I expect the clang driver module will default to invoke it too, and at that point our download would be "self contained" on all platforms. Despite possibly carrying a C compiler in its belly.

Comments welcome. I realize this is a contentious proposal. But it seems like it'd be very useful, to me, and the "integration" into our build seems trivial (I just did it; took ~3 hours, mostly due to waiting on LLVM devs to fix a bug I hit).

[brson]

I'm tentatively in favor since there are so many potential benefits, though it's a huge, probably irrevocable step. Implementing #1850 and #104 in the main rust repo or in rustc itself will mean that Rust is effectively a superset of C, imposing a big (but not insurmountable) burden on any future implementations of Rust.

[graydon]

I'm not sure if it's quite fair to say it's a "superset" of C, since the languages are not subset/supersets. But it would oblige any compatible rust implementation to learn how to compile C code on its own, yes. This could be as simple as shelling out to a local C compiler (which is how we'll solve those bugs in absence of clang).

[nikomatsakis]

I am tentatively in favor as well. When I worked with scala, I was always very happy that you could pass it both *.scala and *.java and it would parse the Java files and make those classes available for use from within Scala (you had to run javac yourself, though).

I do know that @pcwalton was opposed to this idea the last time it came up, though. @brson's concerns are valid too.

[graydon]

https://github.com/graydon/rust/tree/clang-submodule (that is, branch clang-submodule in my repo) has the submodule and configury code wired up, if you want to play with it. Or ought to be. Let me know if it doesn't work.

[brson]

It's a shame that it adds submodules to the llvm repo, since I assume that won't be upstreamable.

[graydon]

Re submodules: might be, might not be. @erickt suggested they might take it since svn would just look straight through it.

[grahame]

In the HPC world (and at the other extreme end of performance, the embedded world) custom, proprietary compilers are often used – eg. the Intel and Portland C/C++ compilers. I can see Rust getting some use as a nice language to glue together existing code bases in C/C++, but users will still want the C/C++ compiler that produces the fastest code.

One other point - Rust's excellent compatibility with the C ABI means it can link easily with other languages that also make that effort. Fortran 2003's BIND(C) facility makes it trivial to provide a nice C interface to Fortran code, which would then also be callable/bindable by rust. A lot of the popular open source maths libraries – and a lot of science/physics code – are written in Fortran. As an example, numpy/scipy depend upon underlying Fortran libraries for matrix and vector operations.

... so I'm not sure how far the idea of embedding compilers for "other" languages will take things. I suspect a lot of the time, people linking C code into a rust crate will produce a static archive and then link that with the rust code. Is it possible the C ABI (which a number of languages can emit code to be called across) is being conflated with C code itself?

Conscious that I might be making things overly complex here; if you think my examples here are too niche, fair enough.

[graydon]

The main uses proposed are:

• running compiler drivers
• compiling small glue/shims in c
• reading c headers to generate bindings

In other words, I don't expect, and am not proposing, rustc positions itself as a go-to compiler for c projects. I don't want, for example, to feel obliged to offer an identical our even terribly similar command line interface to a c compiler.

These are all effectively "interfacing" tasks. I'm assuming most extern libs will still be built with proper c compilers.

[graydon]

This is officially done as of 079c3b0 though it has not "directly" fixed the dependent bugs; they're now "less blocked" than they were before.