Language interop

[expectocode]

We've been discussing this a lot!

A few things we've been thinking about revolve around how the UI and the async network "backend" should speak to each other.
One way would be to have a lot of FFI methods that the (kotlin) UI can call into the Rust (still on the UI side, maybe in a short-lived thread), which would then push a message onto a queue for the backend to process.

Another way would be to have the UI side serialize the message in Kotlin, send this serialised message to one single Rust method (still on the UI side) that de-serializes the message and pushes a message to the backend as before.

Hand-drawn diagram, legibility not guaranteed.

[expectocode]

@9ary @Lonami please feel free to weigh in, too

[expectocode]

We've also been discussing FFI for the message passing, or using sockets. I experimented with both and found a lot of annoyances with unix sockets:

• Kotlin side of things likes to use the abstract namespace, and this isn't supported in the Rust stdlib, so we'd need to use an entirely different socket library like nix (unless there's a way to use nix's UnixAddr with stdlib sockets)
• Filesystem sockets need file system permissions, I think. Unless something else is going on.

[expectocode]

Also, sockets are really byte streams and that's not a super pleasant interface for just trying to pass messages in.

I enjoyed using channels a lot more, once I actually got them working (with lots of input from @Lonami).
Here's a guided tour of my initial socket message passing proof of concept (available at expectocode/android-rust-experiments@a2ff41e):

Android startup code (yes i know we should use a Service not an activity)

This calls the Rust listen() function in a dedicated thread.
listen() looks like this:

We can see evidence of this running in the logs:

The next relevant thing that happens is the button press handler:

This is the key part - UI trying to send a message to the async backend.

So what does sendMsg do?

Yep, it ignores the argument because I haven't yet understood JNI enough to get a byte array from java into rust.

But we can see that we do send a message - "heyyoooo".
And this message is received by the listen() loop!

You can actually tell by the pattern of numbers that it's the "heyyoooo" string, too. Success.

[expectocode]

Using OnceCell for this is also a bit questionable. lonami suggests maybe we actually want mutex<option>.

[expectocode]

@expectocode While typing this i realised i should really be looking at the primitive types, array of byte

[expectocode]

@expectocode https://docs.rs/jni/0.19.0/jni/struct.JNIEnv.html#method.get_array_elements

[Lonami]

WRT ignoring arguments, isn't the input: JObject the string that you passed? Surely there should be a way to interpret it as a JString? I'm not sure if you need the environment for that.

[expectocode]

@Lonami The way to take input: JString as a rust String would be let input: String = env .get_string(input) .expect("Couldn't get Java string!") .into();

But in any case, I think we should actually pass a byte array from the Kotlin side since we'll want to serialise arbitrary data into these messages

[expectocode]

Alright, I've now figured out how to actually use the Kotlin message contents in sendMsg. Resources that were helpful here were the jni-rs chat room (gitter/matrix), where I found a reference to a project using serialisation a lot: https://github.com/exonum/exonum-java-binding/blob/a6627aa00d10bc8175d68392e9156fbf99f48bfe/exonum-java-binding/core/rust/src/testkit/mod.rs#L79.

This led me to jbyteArray and env.convert_byte_array, which made the rest of the work pretty simple.
It's up at expectocode/android-rust-experiments@e272e29.

[expectocode]

Next up is communication going in the other direction. This could work using long polling for receiving - UI continually calls blocking recv() call in a short-lived thread. Sending should be pretty trivial as Rust just needs to put something into a channel.

[expectocode]

credit due, none of this is my design. i'm just implementing it

[expectocode]

We also still need to think about higher-level problems, like how UI makes backend calls that will return data needed for UI updates. example:

User taps on a chat participant to see their profile.
This needs to change the UI to display the profile, and needs to make some backend calls to get profile information.

our solution here is going to have to be heavily informed by the way Jetpack Compose works.

[9ary]

@expectocode single activity navigation in jetpack, compose-specific information about the above

for showing a user profile you'd navigate() to the profile route, which pulls the profile composable into view, and attaches a viewmodel to it

[9ary]

see also https://developer.android.com/jetpack/compose/state on how compose and viewmodels tie together

[9ary]

I've been thinking about this long and hard, and after studying a few systems this is what I've come up with.
We need an IDL to describe the boundary between both languages. Syntax should be fairly straightforward, can take inspiration from other IDLs and also programming languages in general).

Type system, mostly stolen from Thrift

? means inclusion is TBD/not an initial goal

• Primitive types:
  • bool
  • i8, i16, i32, i64
  • unsigned integers?
    • in the interest of keeping this a bit generic, many languages don't support unsigned integers
    • they're a beta feature in Kotlin
    • it's easy to include support but emit an error when using them to generate for a language that can't deal with them
  • f64
    • f32?
• Containers:
  • string
  • array/list/vector
  • set?
  • k/v map?
• Composite types:
  • structs
    • optional fields (Option<T>, nullable references)
    • default values?
      • while not terribly useful for structured data, they'd be great for method calls
    • exceptions? (special structs for error handling)
  • enums
  • tagged unions? (can be mapped to both Rust and Kotlin constructs, but should probably be avoided)
• Others:
  • constants
  • traits and trait objects (see below)

Traits and trait objects

This is the primary mechanism for languages to call into each other. Call it a "Foreign Object Interface" if you will. This is what really sets this idea apart from other RPC systems, where you define singleton services with "static" methods, and references to "remote" objects are not first class.
Traits are basically Rust traits, aka interfaces in Kotlin/Java, though a bit simplified. A trait can define a number of methods, and a language-native struct/class can implement an IDL trait for the other side to own instances of it.
Trait object life-cycle:

• methods can return trait objects
• the runtime assigns the object an ID, and stashes it in a hashmap
• the ID is passed to the other side of the channel
• a wrapper object created
• now the other side can call methods on the wrapper and the runtime will transparently forward those calls to the real object
• the caller language is responsible for requesting object destruction when it's done

Other details:

• Chicken and egg problem, how to obtain an object to call methods on at startup?
  • static methods could allow constructors to be called without any pre-existing instance
    • how are we even supposed to have static methods on a trait and not an implementor?
• Method calls should be represented just like structs internally, but that should be hidden as an implementation detail
• Method calls should be async, implemented with coroutines on both sides of the channel
  • assign a serial number to the call
  • put it in the pipe, await result
  • other side picks it up, calls async implementation of the method
  • return value is put into the pipe with the serial number
  • the promise/future is fulfilled, caller gets the return value
• Everything is pass by value, use move semantics where possible to avoid unnecessary copies
• Needless to say we're aiming for memory and thread safety, hopefully this design makes that easy to implement

Memory layout of encoded types

These rules apply for data that's passed over the pipe. Languages that can make direct use of these representations should go ahead and do that, otherwise conversion is required, not a big deal. Hopefully this should at least avoid serializing/marshaling data only to deserialize it immediately, only one conversion should be necessary.
This is designed with single process interop or shared memory IPC in mind, rather than sending data over the network. Also we drop deterministic data layout to take advantage of the C ABI instead, because the intended use is for co-developed programs running on the same machine to interface together. Maintaining ABI stability is still feasible but out of scope for now, and we don't have to worry about differences between platforms.

• strings should be UTF-8
• strings and containers:
  • in the pipe, pointers to/IDs of runtime objects
  • the implementations should see native objects
• structs should use the C layout
  • struct fields are inline
• enums should be encoded as a 32 bit integer, explicit values are allowed, automatic assignment when none is set

Why roll our own solution?

• No existing solution for FFI bindings handles Rust/Java very well, Rust/Kotlin pretty much unheard of
• Kotlin/JVM and Kotlin/Native have very different FFI systems (JNI vs bespoke traditional FFI design)
• Existing RPC systems are not designed for language interop in a single process (at least not specifically), they include features that are completely superfluous for us (like schema/encoding stability) and neglect some of our needs (zero-copy data movement, first class references to objects on the other side of the interface)

Anything else?

I've probably forgotten something important, feel free to comment and criticize.

[bb010g]

[My responses are in quotes, to deal with GFM not supporting lifting quote levels while maintaining list levels.]

WIP response: not yet finished. Wanted to get this posted in its current state to start.

I've been thinking about this long and hard, and after studying a few systems this is what I've come up with. We need an IDL to describe the boundary between both languages. Syntax should be fairly straightforward, can take inspiration from other IDLs and also programming languages in general).

I don't think this needs to be our own IDL (serializing to bytestrings), and will be comparing a few existing IDLs that serialize to bytestrings: Protocol Buffers (Protobufs) (proto3), Apache Thrift, External Data Representation (XDR) (RFC 4506), and Cap'n Proto.

Type system, mostly stolen from Thrift

? means inclusion is TBD/not an initial goal

Notation:

• x y = xy. (juxatposition)
• {x, y} z = x z, y z; x {y, z} = x y, x z. (juxaposition)
• ¤(¤ ¤) = (λx. x x); ¤(¤2 ¤1) = (λx. λy. y x).

• Primitive types:

  Rust	proto3	Thrift	XDR	Cap'n Proto
  (bool)	(bool)	(bool)	(bool)	(Bool)
  (i{8,16,32,64})	(Void, Void, ¤(int¤ + sint¤ + sfixed¤){32,64})	(i{8,16,32,64})	(Void, Void, integer, hyper integer)	(Int{8,16,32,64})
  (u{8,16,32,64})	(Void, Void, ¤(uint¤ + fixed¤){32,64})	(byte, Void, Void, Void)	(Void, Void, unsigned{ ,hyper}integer	(UInt{8,16,32,64})
  (f{32,64,128})	(float, double, Void)	(Void, double, Void)	(float, double, quadruple)	(Float{32,64}, Void)

• Containers:

  Rust	proto3	Thrift	XDR	Cap'n Proto
  (String)	(string)	(string)	(string identifier<m>) (*) (**)	(Text)
  (Vec<u8>, Vec<T>)	(bytes, repeated T)	(binary + list<byte>, list<T>)	(opaque identifier<m>, type-name identifier<m>) (*)	(Data + List(UInt8), List(T))
  ([u8; N], [T; N])	(Void, Void)	(Void, Void)	(opaque identifier[N], T identifier[N])	(Void, Void)
  (?) (HashMap<K, V>)	(map<K, V>) (***)	(map<K, V>)	(Void)	(Void, Void) (****)
  (?) (HashSet<T>)	(Void)	(set<T>)	(Void)

  Note that Set(k) is isomorphic to Map(k, Bool). Note that Set(k) is isomorphic to List(k) where each k is unique and list item order is forgotten. Note that Map(k, v) is isomorphic to Set((k, v)) where each tuple is unique by k.

  (*): m is an optional maximum byte count, defaulting to (2**32) - 1.

  (**): Officially, an ASCII string, but string identifier<m> is encoded exactly the same as opaque identifier<m>, permitting non-compliance via UTF-8.

  (***): map<K, V> my_map = N; is equivalent to the following, where each entry is unique by key and entry repetition order is unstable:

  message MyMapEntry {
    K key = 1;
    V value = 2;
  }
  repeated MyMapEntry my_map = N;

  (****): No Map(K, V) is provided by default, but the documentation recommends the generic declaration:

  struct Map(Key, Value) {
    entries @0 :List(Entry);
    struct Entry {
      key @0 :Key;
      value @1 :Value;
    }
  }

• Composite types:

  Rust	proto3	Thrift	XDR	Cap'n Proto
  struct Foo { bar: bool, baz: i32, }	struct Foo { 1: bool bar, 2: i32 baz, }	message Foo { bool bar = 1; int32 baz = 2; }	struct { bool bar; integer baz; } foo;	struct Foo { bar @0 :Bool; baz @1 :Int32; }

  • structs

    • optional fields (Option, nullable references)

    • default values?

      • while not terribly useful for structured data, they'd be great for method calls

    • exceptions? (special structs for error handling)

  • enums

  • tagged unions? (can be mapped to both Rust and Kotlin constructs, but should probably be avoided)

• Others:

  • constants

  • traits and trait objects (see below)

Traits and trait objects

This is the primary mechanism for languages to call into each other. Call it a "Foreign Object Interface" if you will. This is what really sets this idea apart from other RPC systems, where you define singleton services with "static" methods, and references to "remote" objects are not first class. Traits are basically Rust traits, aka interfaces in Kotlin/Java, though a bit simplified. A trait can define a number of methods, and a language-native struct/class can implement an IDL trait for the other side to own instances of it. Trait object life-cycle:

• methods can return trait objects

• the runtime assigns the object an ID, and stashes it in a hashmap

• the ID is passed to the other side of the channel

• a wrapper object created

• now the other side can call methods on the wrapper and the runtime will transparently forward those calls to the real object

• the caller language is responsible for requesting object destruction when it's done

Other details:

• Chicken and egg problem, how to obtain an object to call methods on at startup?

  • static methods could allow constructors to be called without any pre-existing instance

    • how are we even supposed to have static methods on a trait and not an implementor?

• Method calls should be represented just like structs internally, but that should be hidden as an implementation detail

• Method calls should be async, implemented with coroutines on both sides of the channel

  • assign a serial number to the call

  • put it in the pipe, await result

  • other side picks it up, calls async implementation of the method

  • return value is put into the pipe with the serial number

  • the promise/future is fulfilled, caller gets the return value

• Everything is pass by value, use move semantics where possible to avoid unnecessary copies

• Needless to say we're aiming for memory and thread safety, hopefully this design makes that easy to implement

Memory layout of encoded types

These rules apply for data that's passed over the pipe. Languages that can make direct use of these representations should go ahead and do that, otherwise conversion is required, not a big deal. Hopefully this should at least avoid serializing/marshaling data only to deserialize it immediately, only one conversion should be necessary. This is designed with single process interop or shared memory IPC in mind, rather than sending data over the network. Also we drop deterministic data layout to take advantage of the C ABI instead, because the intended use is for co-developed programs running on the same machine to interface together. Maintaining ABI stability is still feasible but out of scope for now, and we don't have to worry about differences between platforms.

• strings should be UTF-8

• strings and containers:

  • in the pipe, pointers to/IDs of runtime objects

  • the implementations should see native objects

• structs should use the C layout

  • struct fields are inline

• enums should be encoded as a 32 bit integer, explicit values are allowed, automatic assignment when none is set

Why roll our own solution?

• No existing solution for FFI bindings handles Rust/Java very well, Rust/Kotlin pretty much unheard of

• Kotlin/JVM and Kotlin/Native have very different FFI systems (JNI vs bespoke traditional FFI design)

• Existing RPC systems are not designed for language interop in a single process (at least not specifically), they include features that are completely superfluous for us (like schema/encoding stability) and neglect some of our needs (zero-copy data movement, first class references to objects on the other side of the interface)

Feature	proto3	Thrift	XDR	Cap'n Proto
Zero-copy	No	Currently, no, but protocol-dependent	Yes	Yes
First-class object references (RPC)	N/A	No	N/A	Yes
First-class peer heap object references (RPC)	N/A	No	N/A	Currently, no, but possible

Cap'n Proto RPC viably supporting inproc, shared memory vat networks was pretty unexpected. rpc.capnp defines the official RPC system.

Cap'n Proto RPC takes place between "vats". A vat hosts some set of objects and talks to other
vats through direct bilateral connections. Typically, there is a 1:1 correspondence between vats
and processes (in the unix sense of the word), although this is not strictly always true (one
process could run multiple vats, or a distributed virtual vat might live across many processes).

An example VatNetwork interface sketch is given at the end of the file, and nothing in that relies upon traditional connection streams actually existing either. All that's necessary basically is a way to identify vats, ways to connect to & accept connections from vats, and then ways to send & recieve messages on connections. Magically fast, zero-copy, shared memory message-passing is perfectly legal. Plus, I think we could get away with a decently simple implementation.

[9ary]

@bb010g has motivated me to look further into it, and it looks like capnproto pretty much does what we need! We may need to mess around with the Rust implementation a bit, and the Java implementation should work to start with (though it's missing any RPC support), and a Kotlin/Multiplatform port shouldn't be too difficult to implement.

[9ary]

Things I've been looking at:

• uniffi seems like a great baseline, suits our use case much better than capnp
  • cross-language futures/promises could be built on top of that, it's pretty much the only thing missing from capnp
    • channels are a possible alternative that doesn't rely on async/await
• eventfd is probably a better synchronization primitive than spawning a thread that blocks on a channel