Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate New Annotation Library #623

Merged
merged 29 commits into from
Sep 9, 2022
Merged

Integrate New Annotation Library #623

merged 29 commits into from
Sep 9, 2022

Conversation

festutz
Copy link
Collaborator

@festutz festutz commented Aug 31, 2022 •
edited
Loading

Things to do before we can merge:

  • fix tr-test.sh in testing script: when run outside of the script, pash produces the same results but the testing script reports different outputs
  • make sure that CI tests work (cannot be checked for all at the moment since some parts of CI need to be fixed)
  • fix commit in annotations-repository for the dependency / deploy the annotations library as a python-library

festutz and others added 22 commits June 27, 2022 13:38
@festutz
First PR for new annotations and new model for DFG nodes (#583)
613d090 
* WIP: first changes for new annotations

* Make parallelization use mapper info from new annotations

* Make parallelization use mapper info from new annotations

* Delete com_mapper field in dfg_node and use the info from new annotations

* Use input info from new annotations

* Use output info from new annotations

* Make dfg_options use the information from new annotations and completely remove use of old aggregator com_aggregator

* Make to_ast work for wf.sh with some hacks, e.g. to_ast for eager to handle special case of intermediate file as last operand and fixed parsing issue for newline

* Remove com_mapper and com_aggregator from DFGNode

* WIP: incorporating remodelled command invocations

* Parsing with new dataflow node model works

* WIP: parallelization

* Rudimentary parallelization with new annotations works

* Add way to specify where to find repository for annotations repository

* Do not require flag `r_split` since we do consecutive chunks for now

* 1st part of changes due to comments for PR

* minor fix

* 2nd part of changes due to comments for PR

Co-authored-by: Felix Stutz <[email protected]>
@festutz
Add support for eager-nodes (#589)
8407ca5 
* Add support for eager-nodes

Signed-off-by: Felix Stutz <[email protected]>

* Remove eager from TODO-list

Signed-off-by: Felix Stutz <[email protected]>

* Addressed comments from PR

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Add reduce/aggregation trees (#590)
d92cbba 
* Add reduce trees

Signed-off-by: Felix Stutz <[email protected]>

* Remove task in TODO

Signed-off-by: Felix Stutz <[email protected]>

* Remove log statements

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Add support for round-robin parallelization, including unwrap for com…
2b9935d 
…mutative commands (#591)

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Fix unwrap and commutative interplay (#593)
77da707 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Add support for dgsh_tee nodes (#594)
a1c87a5 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Update TODOs before merging to future
034f41d 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Modify dgsh-wrapper to not require operands before options but pass o…
f6a3df1 
…ptions for input and output directly (#595)

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Remove dgsh-tee wrapper and call dgsh-tee directly (#596)
558dc45 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Separate parallelization into choose and apply phases (#597)
a77d4ee 
* Refactor parallelizing transformations in separate choose and apply phases

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Fuse cat and subsequent split (#599)
f1221ff 
* Fuse cat and subsequent split

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Fuse r_merge and subsequent r_split (#600)
334091f 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Fuse r_merge and subsequent commutative command (#601)
58fbd29 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Add TODO
4a0e5a4 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Support round-robin parallelization for commutative commands (#602)
cd00918 
Signed-off-by: Felix Stutz <[email protected]>
@angelhof
Install annotations lib using pip (#603)
dfa03a7 
* Add a proper installation of the annotation lib

Signed-off-by: Konstantinos Kallas <[email protected]>

* Remove unnecessary sys.path

Signed-off-by: Konstantinos Kallas <[email protected]>

* Fix bug in setup

Signed-off-by: Konstantinos Kallas <[email protected]>

* fix setup script

Signed-off-by: Konstantinos Kallas <[email protected]>
@angelhof
Add a whitespace to trigger CI
624d817 
Signed-off-by: Konstantinos Kallas <[email protected]>
@festutz
Refactored to remove __future__ from annotations library (#609)
3bd0cf6 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Cover all but one test case from compiler/test_evaluation_scripts.sh (#…
44017dc 
…612)

* Fix bug in parser to switch from flag to operand mode when reading hyphen

Signed-off-by: Felix Stutz <[email protected]>

* Adapt shortest_scripts.sh to work with parser

Signed-off-by: Felix Stutz <[email protected]>

* Cover more test cases from script_microbenchmarks

Signed-off-by: Felix Stutz <[email protected]>

* Parallelize spell-grep as done in `future`, i.e., not RR but CC for `set_diff`

Signed-off-by: Felix Stutz <[email protected]>

* Clean up and clarifying comment in parser

Signed-off-by: Felix Stutz <[email protected]>

* Simplify control flow in parallelization

Signed-off-by: Felix Stutz <[email protected]>
@festutz
Add support for bigrams (#614)
a75378c 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Refactor to have defaults for AnnotationInfo (#615)
3f79626 
Signed-off-by: Felix Stutz <[email protected]>
@festutz
Minor changes due to typing in annotations repository (#622)
b5bd70b 
Signed-off-by: Felix Stutz <[email protected]>
@github-actions

This comment was marked as outdated.

Sign in to view
@festutz festutz changed the title Future annotations Integrate New Annotation Library Aug 31, 2022
@festutz festutz mentioned this pull request Sep 2, 2022
Closed
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@angelhof
revert setup merge issue
1ee23bc 
Signed-off-by: Konstantinos Kallas <[email protected]>
@github-actions
Copy link

github-actions bot commented Sep 5, 2022

OS:ubuntu-20.04
Mon Sep 5 16:35:22 UTC 2022
intro: 2/2 tests passed.
interface: 39/39 tests passed.
compiler: 54/54 tests passed.
agg: 109/109 tests passed.

@github-actions
Copy link

github-actions bot commented Sep 5, 2022

OS = Debian 10
CPU = Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Ram = 15752
Hash = 1ee23bc
Kernel= Linux 4.15.0-167-generic x86_64

benchmark tests passed failed untested unresolved unsupported not_in_use other_status
posix 0 0 0 0 0 0 0 0
intro 2 0 2 0 0 0 0 0
interface 39 6 33 0 0 0 0 0
compiler 54 4 50 0 0 0 0 0
aggregator 109 11 98 0 0 0 0 0

@angelhof
Copy link
Member

angelhof commented Sep 6, 2022

@festutz we should completely remove the annotations directory from this branch and move it under the new annotations repo as old_annotations for reference. In any case, it shouldn't be in the PaSh directory anymore.

@github-actions
Copy link

github-actions bot commented Sep 8, 2022

OS:ubuntu-20.04
Thu Sep 8 12:04:41 UTC 2022
intro: 2/2 tests passed.
interface: 39/39 tests passed.
compiler: 54/54 tests passed.
agg: 109/109 tests passed.

@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@angelhof
whitespace for CI
3a7e7f4 
Signed-off-by: Konstantinos Kallas <[email protected]>
@github-actions
Copy link

github-actions bot commented Sep 9, 2022

OS:ubuntu-20.04
Fri Sep 9 18:36:19 UTC 2022
intro: 2/2 tests passed.
interface: 39/39 tests passed.
compiler: 54/54 tests passed.
agg: 109/109 tests passed.

@github-actions
Copy link

github-actions bot commented Sep 9, 2022

OS = Debian 10
CPU = Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Ram = 15752
Hash = 3a7e7f4
Kernel= Linux 4.15.0-167-generic x86_64

benchmark tests passed failed untested unresolved unsupported not_in_use other_status
posix 494 375 41 31 6 40 1 0
intro 2 2 0 0 0 0 0 0
interface 39 39 0 0 0 0 0 0
compiler 54 54 0 0 0 0 0 0
aggregator 109 109 0 0 0 0 0 0

@angelhof
Copy link
Member

angelhof commented Sep 9, 2022

All the tests pass here! I think we should merge first (since this PR has been waiting here for too long) and after that we can make sure to properly install the annotation library using pypi (instead of cloning it and locally building it using pip). See issue #633

@angelhof angelhof merged commit eea912a into future Sep 9, 2022
@angelhof angelhof deleted the future_annotations branch September 9, 2022 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@festutz @angelhof