Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(hydroflow_lang): initial flo implementation #1585

Merged
merged 8 commits into from
Dec 3, 2024

Conversation

MingweiSamuel
Copy link
Member

@MingweiSamuel MingweiSamuel commented Nov 21, 2024

Basic first pass implementation, without changing the current scheduler, no breaking behavior. Adds the flo syntax (loop { ... }), basic graph structure checks, nested loop blocks, and two basic windowing operators (batch() and all_once()).

Next steps:

  • Needs scheduler changes:
    • Implement un-windowing operators
    • Implement windowing operator repeat_n() -
  • Pipeline flags: Needs checking of bounded vs unbounded for batch() vs all_once()
  • Needs checking that all inputs into a loop agree
  • State type negotiations (for batch() and handoffs (?) - don't always use Vec)
  • Dag performance optimizations (lots of things)

@MingweiSamuel MingweiSamuel changed the title wip: flo impl feat: flo impl (wip) Nov 21, 2024
@MingweiSamuel MingweiSamuel force-pushed the flo-impl-1 branch 6 times, most recently from f5b6683 to dbf0879 Compare November 27, 2024 21:23
@MingweiSamuel MingweiSamuel changed the title feat: flo impl (wip) feat: initial flo implementation Nov 27, 2024
…ect#1585)

Just the parsing. Error checking not yet implemented, semantics/scheduling not yet implemented.

```rust
users = source_stream(0..);
messages = source_stream(0..);
loop {
    users -> batch() -> [0]cp;
    messages -> batch() -> [1]cp;
    cp = cross_join() -> for_each(|(user, message)| println!("notify {} of {}", user, message));
}
```
(`batch()` does not yet exist)
…#1585)

Check for non-root sources and cycles in loop blocks

Depends on hydro-project#1507
Depends on hydro-project#1512
@MingweiSamuel MingweiSamuel marked this pull request as ready for review November 27, 2024 21:59
@MingweiSamuel MingweiSamuel changed the title feat: initial flo implementation feat(hydroflow_lang): initial flo implementation Nov 27, 2024
MingweiSamuel added a commit to MingweiSamuel/hydroflow that referenced this pull request Nov 27, 2024
Copy link
Member

@shadaj shadaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Cool that we can do this in a somewhat backwards-compatible way.

hydroflow_lang/src/graph/ops/batch.rs Outdated Show resolved Hide resolved
hydroflow_lang/src/graph/ops/batch.rs Show resolved Hide resolved
@MingweiSamuel MingweiSamuel merged commit b8acd84 into hydro-project:main Dec 3, 2024
3 checks passed
@MingweiSamuel MingweiSamuel deleted the flo-impl-1 branch December 3, 2024 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants