Support native FFI calls via libffi

[DemiMarie]

It would be nice to use libffi to allow for external calls.

• Passing non-pointer scalars across the FFI barrier: Adding FFI support to Miri, by emarteca #2365
• Letting C read from Miri-allocated memory: Enable Miri to pass pointers through FFI rust#129684
• Letting C write to Miri-allocated memory
  • C code can initialize previously initialized memory
  • C code can store pointers in Miri-allocated memory
• Letting Miri access memory that was allocated in C
• Passing non-scalar types across the FFI barrier

[eddyb]

Previously suggested on IRC based on @cmr's mentions in a different context.
The closure example presents what I think is one of the most valuable features of this library: the ability to JIT a C function which hardcodes extra owned values, a whole closure masquerading in a function pointer.

[solson]

@drbo We've definitely been considering that.

For a bit of context, there are two versions of Miri we'll have in the future. If the Rust teams decide to use a MIR-based constant evaluator, some version of Miri will get incorporated into rustc, and that version will never use libffi because you want to ban arbitrary FFI calls in constant evaluation.

But it would be a lot of fun to add libffi to this external version of Miri. I would probably wait until after any Miri → rustc merging happens, though.

[BatmanAoD]

Now that Miri is indeed in rustc (wooooo!), any further thoughts on this?

If I understand correctly, FFI would be necessary for us to use Miri as the basis for a Jupyter kernel.

[eddyb]

@BatmanAoD I'd advise against using miri for such a purpose - have you looked at webassembly?

[BatmanAoD]

@eddyb I don't actually know what the requirements are for a Jupyter kernel; I was basing my understanding on an old discussion about using rusti instead of Miri for the basis of the kernel.

I can't actually find documentation on how the kernel API works, though (I probably just haven't spent enough time looking). It may be that communication between JupyterLab and the kernel is entirely over HTTP.

In any case: I'm not sure how WebAssembly would solve the problem. The Jupyter kernel is a REPL for a given language; the goal of a Rust kernel would be to permit people to create Jupyter notebooks that would use the Rust language instead of IPython (or any of the other languages that are already supported via kernels). So the need is for a Rust REPL that supports whatever interface JupyterLab uses.

[eddyb]

@BatmanAoD Oops, I misunderstood "kernel" as "OS kernel", not an IPython-like thing.
Then miri is likely the way to go, and I agree libffi would be needed.

[BatmanAoD]

@eddyb Yeah, I think the terminology is a bit confusing!

[oli-obk]

This project is almost ready (as in compiling and testing on Travis) again, I have a PR against rustc ready that should fix all the tests. After that we can accept new PRs. So if anyone wants to hack up a prototype using libffi to execute some c functions that we're currently having horrible fake support for (like syscall), that would be totally awesome. Things like mallocare a little harder though, but with a new allocation kind totally doable without touching rustc.

[eddyb]

@oli-obk I think pointers in general are hard. Various syscalls take pointers (e.g. read and write) while others return them (e.g. mmap) - so malloc is not unique in this difficulty.
It's easier to detect miri pointers being passed to FFI, and do something about it, than it is to handle the other way - I'm thinking maybe miri could be self-aware (e.g. via /proc/self/maps) and allow access via integer pointers to memory regions that would surely not segfault (that would, amusingly, allow corrupting miri itself - which C code could do anyway, so it's not introducing a new issue).

[BatmanAoD]

@eddyb If it were possible to guarantee that improper writes would segfault, wouldn't that be preferable to corrupting the Miri process without terminating it?

[eddyb]

@BatmanAoD My point was that integer pointers could be used to corrupt miri itself - regardless of how you handle segfaults, the values for which corruption could happen are generally hard to detect, and pointless anyway, because you're also calling into C, which can do anything anyway.

[oli-obk]

Yea.... I don't think we should care too much about users doing horrible things. If you do care, don't enable the ffi feature ^^

[elichai]

Any news on this?
Would love to use miri on some projects that have a lot of FFI calls.

[Ericson2314]

libffi is due for a new release. (GHC and probably other projects have had to vendor it which is a pain.) If you do end up using it, please lobby upstream so we can all get a working release!

[emarteca]

Hey! I've started working on this issue -- here is a design doc of our proposed plan so far: link here. We'd love your feedback/comments on the proposed approach!
@oli-obk @RalfJung

Thanks! :)

[RalfJung]

@emarteca that's a very welcome surprise. :-)
Let's make a new issue to specifically discuss your project: #2365