Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WASM running of hydroflow #329

Closed
MingweiSamuel opened this issue Jan 4, 2023 · 10 comments
Closed

WASM running of hydroflow #329

MingweiSamuel opened this issue Jan 4, 2023 · 10 comments
Assignees
@tylerhou
Labels
enhancement New feature or request hydroflow syntax Hydroflow's custom surface syntax

Comments

@MingweiSamuel
Copy link
Member

Should be possible to do, but do we want to? Would be fun but maybe not high priority. Mentioned in slack briefly @shadaj
@MingweiSamuel MingweiSamuel added the enhancement New feature or request label Jan 4, 2023
@shadaj
Copy link
Member

shadaj commented Jan 4, 2023

I've been slowly working through this in down time, current goal is to get a demo on the website

@MingweiSamuel
Copy link
Member Author

The surface syntax parsing and processing should be good, but I anticipate issues with Tokio on WASM

@shadaj
Copy link
Member

shadaj commented Jan 4, 2023

Oh yes true, I guess there are two issues: get the Hydroflow Language / Dedalus compilers to work on WASM, and get the Hydroflow runtime to work. Found https://github.com/shosti/wraft as a potentially interesting benchmark for the runtime; in principle we should be able to wire up WebRTC as the networking layer with Tokio on WASM.

@MingweiSamuel
Copy link
Member Author

MingweiSamuel commented Jan 6, 2023 •
edited
Loading

hydroflow_lang #331

@MingweiSamuel MingweiSamuel added the P1 High priority label Jan 17, 2023
@shadaj
Copy link
Member

shadaj commented Feb 1, 2023

@tylerhou will look into this soon

@tylerhou
Copy link
Contributor

tylerhou commented Feb 7, 2023 •
edited
Loading

I took a brief look at this today. The latest version of Tokio has limited support for WASM including runtime: https://github.com/tokio-rs/tokio/blob/a7945b469d634cf205094d8a1661720358622cc0/tokio/src/lib.rs#L387. Our current version also supports all those features above but it is not documented (tested by compiling for WASM with those features enabled).

For WASM we would need to not specify tokio { ... features = "full" } in hydroflow/Cargo.toml and instead selectively enable the features based if the target is WASM. hydroflow compiles if we do this and also disable diagnostics for hydroflow_lang. Very rough patch that compiles for WASM: https://gist.github.com/tylerhou/bd3ae3e2bb58c3f0b05fc100a63298b7

There are still two outstanding issues:

  1. Documentation for tokio says that while time compiles for non-WASI WASM targets, it panics at runtime. We use timers in only one file -- hydroflow/tests/surface_async.rs -- and its a test, so maybe this is not an issue.
  2. cargo test -p hydroflow --target wasm32-unknown-unknown does not work because the getrandom library (which is included tvia rand and transitively through textnonce -> rand in dev dependencies) requires us to build with the js feature flag or target wasm32-unknown-emscripten so it knows which random implementation to call. https://docs.rs/getrandom/latest/getrandom/#webassembly-support

wasm.patch

@shadaj
Copy link
Member

shadaj commented Feb 7, 2023

This is very exciting progress! For the second issue, I think we can add a js feature flag to hydroflow itself that then enables the js flag on getrandom. That way we can support both wasm32-unknown-unknown and wasm32-wasi (I'm not particularly concerned about -emscripten).

@tylerhou tylerhou mentioned this issue Feb 8, 2023
Closed
tylerhou added a commit to tylerhou/hydroflow that referenced this issue Feb 27, 2023
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
a6c7d76 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only with X enabled
   (assuming same host and target arch). With the v2 resolver, Cargo
   will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
tylerhou added a commit to tylerhou/hydroflow that referenced this issue Feb 27, 2023
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
a7efdcc 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
tylerhou added a commit to tylerhou/hydroflow that referenced this issue Feb 27, 2023
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
a422222 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
tylerhou added a commit to tylerhou/hydroflow that referenced this issue Feb 27, 2023
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
796eeda 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
tylerhou added a commit to tylerhou/hydroflow that referenced this issue Mar 1, 2023
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
de27289 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
tylerhou added a commit that referenced this issue Mar 1, 2023
@tylerhou
Get hydroflow to compile to WASM (#329)
24354d2 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
@MingweiSamuel MingweiSamuel added the hydroflow syntax Hydroflow's custom surface syntax label Mar 13, 2023
@MingweiSamuel MingweiSamuel changed the title WASM compilation of hydroflow WASM compilation/running of hydroflow Nov 20, 2023
nickjiang2378 pushed a commit to nickjiang2378/hydroflow that referenced this issue Jan 24, 2024
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
465bcab 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
nickjiang2378 pushed a commit to nickjiang2378/hydroflow that referenced this issue Jan 25, 2024
@tylerhou
Get hydroflow to compile to WASM (hydro-project#329)
16a8864 
Tests do not work yet; I have a followup PR.

1. Turn off diagnostics for non-proc-macro crates. Diagnostics don't
   work when building for WASM. The proc-macro crates that are compiled
   at compile time still enable diagonstics because they target the
   (presumably non-WASM) host architecture where diagonstics work.

2. Use version 2 for the Cargo feature resolver. The version 1 resolver
   does not treat proc-macro edges differently from regular dependency
   edges. That is, suppose we depend on package P transitively via both
   a proc-macro edge and a regular edge, and the proc-macro edge enables
   feature X but the regular edge does not.

   With the version 1 resolver, Cargo will compile P only once with X
   enabled (assuming same host and target arch). With the v2 resolver,
   Cargo will compile P (at least) twice; once with X enabled for the
   proc-macro crate, once with P disabled.

   In our project, P = {hydroflow_datalog_core, hydroflow_lang}, X =
   diagnostics. The version 1 resolver works for non-WASM builds
   (because diagnostics compiles fine), but for WASM builds the v1
   resolver attempts to build hydroflow_datalog_core for WASM with
   diagnostics enabled (because of feature resolution).

   Note that enabling the v2 feature resolver should not change compile
   times: on non-WASM, diagnostics is always enabled so we only compile
   P once. On WASM builds, we would have compiled twice anyway (once for
   the host architecture for the proc-macro package, another for WASM).

3. Turn on only a subset of Tokio features when compiling for WASM. In
   particular, the WASM environment does not natively support threads
   and fails to compile with threading features enabled.

4. Manually enable the "js" feature for hydroflow/getrandom. See the
   comment in hydroflow/Cargo.toml.
@MingweiSamuel MingweiSamuel removed the P1 High priority label Mar 26, 2024
@MingweiSamuel MingweiSamuel changed the title WASM compilation/running of hydroflow WASM running of hydroflow Mar 26, 2024
@MingweiSamuel
Copy link
Member Author

(Compilation in WASM, i.e. running hydroflow_lang/macro, has been working for a long time)

@MingweiSamuel
Copy link
Member Author

Works in general, networking (TCP analogous to websockets, other options as well), #1440 will be the main issue

@MingweiSamuel MingweiSamuel closed this as not planned Won't fix, can't repro, duplicate, stale Nov 21, 2024
@MingweiSamuel
Copy link
Member Author

MingweiSamuel commented Nov 21, 2024 •
edited
Loading

#1447 wasm hackathon branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tylerhou tylerhou

Labels
enhancement New feature or request hydroflow syntax Hydroflow's custom surface syntax
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@shadaj @tylerhou @MingweiSamuel