Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backporting changes to release-v1.2 #1517

Merged
merged 67 commits into from
Feb 23, 2023
Merged

Conversation

eapolinario and others added 30 commits February 22, 2023 11:52
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
…1240)

* first commit

Signed-off-by: Ryan Nazareth <[email protected]>

* add tensorflow example tf record transformer

Signed-off-by: Ryan Nazareth <[email protected]>

* refactor

Signed-off-by: Ryan Nazareth <[email protected]>

* correct tfexample description

Signed-off-by: Ryan Nazareth <[email protected]>

* fix test_native.py

Signed-off-by: Ryan Nazareth <[email protected]>

* add tensorflow docs and reqs

Signed-off-by: Ryan Nazareth <[email protected]>

* add tensorflow docs and reqs1

Signed-off-by: Ryan Nazareth <[email protected]>

* tensorflow import in init

Signed-off-by: Ryan Nazareth <[email protected]>

* fix failing tests

Signed-off-by: Ryan Nazareth <[email protected]>

* add tensorflow pinned version to reqs

Signed-off-by: Ryan Nazareth <[email protected]>

* pin grpcio-status to remove protobuf error

Signed-off-by: Ryan Nazareth <[email protected]>

* add suggested changes

Signed-off-by: Ryan Nazareth <[email protected]>

* redesign transformer

Signed-off-by: Ryan Nazareth <[email protected]>

* remove old script

Signed-off-by: Ryan Nazareth <[email protected]>

* fix type reference for TFREcordDataset

Signed-off-by: Ryan Nazareth <[email protected]>

* refactor

Signed-off-by: Ryan Nazareth <[email protected]>

* refactor

Signed-off-by: Ryan Nazareth <[email protected]>

* spacing and uppercase

Signed-off-by: Ryan Nazareth <[email protected]>

* redesign with tfdir and tfrecordfile subclass

Signed-off-by: Ryan Nazareth <[email protected]>

* fix conflicts and typos

Signed-off-by: Ryan Nazareth <[email protected]>

* address majority of comments

Signed-off-by: Ryan Nazareth <[email protected]>

* refactor

Signed-off-by: Ryan Nazareth <[email protected]>

* fix test with flytefile and metadata annotated

Signed-off-by: Ryan Nazareth <[email protected]>

* fix check for example records in directory

Signed-off-by: Ryan Nazareth <[email protected]>

* refactor and correct typing

Signed-off-by: Ryan Nazareth <[email protected]>

* lint

Signed-off-by: Ryan Nazareth <[email protected]>

* import annotated from typing_extensions

Signed-off-by: Ryan Nazareth <[email protected]>

* tweak to tests to test case when Config not passed in as type

Signed-off-by: Ryan Nazareth <[email protected]>

* add suggested changes

Signed-off-by: Ryan Nazareth <[email protected]>

* add task for tfrecord dir with no config in test

Signed-off-by: Ryan Nazareth <[email protected]>

* get filenames from local dir instead of remote

Signed-off-by: Ryan Nazareth <[email protected]>

Signed-off-by: Ryan Nazareth <[email protected]>
Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
* Set default format of structured dataset to empty

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* last error (#1364)

Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
* Adds pyflyte CLI reference guide

Signed-off-by: Samhita Alla <[email protected]>

* bump python version

Signed-off-by: Samhita Alla <[email protected]>

* bump python version

Signed-off-by: Samhita Alla <[email protected]>

* resolve docs error

Signed-off-by: Samhita Alla <[email protected]>

* set nested to none

Signed-off-by: Samhita Alla <[email protected]>

* remove flyteidl version constraint

Signed-off-by: Samhita Alla <[email protected]>

* update requirements

Signed-off-by: Samhita Alla <[email protected]>

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
#1375)

Signed-off-by: H. Furkan Vural <[email protected]>

Implemented cache overwrite feature is added on flytekit as well for the completeness. In order to support the cache eviction RFC, an overwrite parameter was added, indicating the data store should replace an existing artifact instead of creating a new one on local calls.
* Register databricks plugin

Signed-off-by: Kevin Su <[email protected]>

* Update databricks plugin

Signed-off-by: Kevin Su <[email protected]>

* register databricks

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
For Pod tasks, if the primary_container_name is not specified, it should default.

Signed-off-by: Ketan Umare <[email protected]>
* Add support for Optional to dataclass transformer

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add one more test

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add one more test

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix serialization of optional flyte types

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
* Added more pod tests and an example pod task

Signed-off-by: Ketan Umare <[email protected]>

* fixing test and name

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
* MLFlow plugin in progress

Signed-off-by: Ketan Umare <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* update test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update readme

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* dwip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* change experiment name

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Add mlflow to index.rst

Signed-off-by: Kevin Su <[email protected]>

* use experiment name that user provided

Signed-off-by: Kevin Su <[email protected]>

* update doc-requirements.txt

Signed-off-by: Kevin Su <[email protected]>

* Add backend plugin deployment

Signed-off-by: Kevin Su <[email protected]>

* generate doc for method

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* update docstring

Signed-off-by: Niels Bantilan <[email protected]>

* update docstring

Signed-off-by: Niels Bantilan <[email protected]>

* Update tracking.py

Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>
* Read polars dataframe in a folder

Signed-off-by: Kevin Su <[email protected]>

* Read polars dataframe in a folder

Signed-off-by: Kevin Su <[email protected]>

* Load huggingface and spark plugin implicitly

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* remove _pyspark alias

Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
* Add dummy task type to test backend plugin

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add docs page

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add dask models

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add function to convert resources

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add tests to `dask` task

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Remove namespace

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update setup.py

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add dask to `plugin/README.md`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add README.md for `dask`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Top level export of `JopPodSpec` and `DaskCluster`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update docs for images

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update README.md

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update models after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update task after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Raise error when less than 1 worker

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update flyteidl to >= 1.3.2

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update doc requirements

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update doc-requirements.txt

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Re-lock dependencies on linux

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update dask API docs

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix documentation links

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Default optional model constructor arguments to `None`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Refactor `convert_resources_to_resource_model` to `core.resources`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Use `convert_resources_to_resource_model` in `core.node`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Incorporate review feedback

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Bernhard Stadlbauer <[email protected]>
Signed-off-by: Bernhard Stadlbauer <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
bethebunny and others added 9 commits February 22, 2023 14:21
* support checkpointing in local mode from cached tasks

* clear cache before tests

---------

Co-authored-by: Stef Nelson-Lindall <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
* Deprecate FlyteSchema

Signed-off-by: Kevin Su <[email protected]>

* Remove version

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
* Use scarf images

Signed-off-by: Eduardo Apolinario <[email protected]>

* Use scarf names in tests.

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
* add reference_launch_plan to flytekit api ref

Signed-off-by: Niels Bantilan <[email protected]>

* import in init, add docstrings

Signed-off-by: Niels Bantilan <[email protected]>

* add more to references

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

* update

Signed-off-by: Niels Bantilan <[email protected]>

* fix up docstrings

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Samhita Alla <[email protected]>
* wip

Signed-off-by: Kevin Su <[email protected]>

* Fix mypy errors

Signed-off-by: Kevin Su <[email protected]>

* Fix mypy errors

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Update type

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update dev-requirements.txt

Signed-off-by: Kevin Su <[email protected]>

* Address comment

Signed-off-by: Kevin Su <[email protected]>

* upgrade torch

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* support dynamic task

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* lazy compile

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update test

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
* Get the origin type when serializing dataclass

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update test

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
wild-endeavor
wild-endeavor previously approved these changes Feb 22, 2023
eapolinario and others added 3 commits February 22, 2023 14:39
Signed-off-by: Eduardo Apolinario <[email protected]>
* Delay initialization of SynchronousFlyteClient in FlyteRemote

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix spark plugin flyteremote test.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
This reverts commit 3798450.

Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
This reverts commit 41a9c7a.

Signed-off-by: Eduardo Apolinario <[email protected]>
@codecov
Copy link

codecov bot commented Feb 23, 2023

Codecov Report

❗ No coverage uploaded for pull request base (release-v1.2@10bf232). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             release-v1.2    #1517   +/-   ##
===============================================
  Coverage                ?   69.31%           
===============================================
  Files                   ?      304           
  Lines                   ?    28601           
  Branches                ?     2709           
===============================================
  Hits                    ?    19826           
  Misses                  ?     8252           
  Partials                ?      523           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
@eapolinario eapolinario merged commit d75c800 into release-v1.2 Feb 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.