-
Notifications
You must be signed in to change notification settings - Fork 301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backporting changes to release-v1.2 #1517
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
…1240) * first commit Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow example tf record transformer Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * correct tfexample description Signed-off-by: Ryan Nazareth <[email protected]> * fix test_native.py Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow docs and reqs Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow docs and reqs1 Signed-off-by: Ryan Nazareth <[email protected]> * tensorflow import in init Signed-off-by: Ryan Nazareth <[email protected]> * fix failing tests Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow pinned version to reqs Signed-off-by: Ryan Nazareth <[email protected]> * pin grpcio-status to remove protobuf error Signed-off-by: Ryan Nazareth <[email protected]> * add suggested changes Signed-off-by: Ryan Nazareth <[email protected]> * redesign transformer Signed-off-by: Ryan Nazareth <[email protected]> * remove old script Signed-off-by: Ryan Nazareth <[email protected]> * fix type reference for TFREcordDataset Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * spacing and uppercase Signed-off-by: Ryan Nazareth <[email protected]> * redesign with tfdir and tfrecordfile subclass Signed-off-by: Ryan Nazareth <[email protected]> * fix conflicts and typos Signed-off-by: Ryan Nazareth <[email protected]> * address majority of comments Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * fix test with flytefile and metadata annotated Signed-off-by: Ryan Nazareth <[email protected]> * fix check for example records in directory Signed-off-by: Ryan Nazareth <[email protected]> * refactor and correct typing Signed-off-by: Ryan Nazareth <[email protected]> * lint Signed-off-by: Ryan Nazareth <[email protected]> * import annotated from typing_extensions Signed-off-by: Ryan Nazareth <[email protected]> * tweak to tests to test case when Config not passed in as type Signed-off-by: Ryan Nazareth <[email protected]> * add suggested changes Signed-off-by: Ryan Nazareth <[email protected]> * add task for tfrecord dir with no config in test Signed-off-by: Ryan Nazareth <[email protected]> * get filenames from local dir instead of remote Signed-off-by: Ryan Nazareth <[email protected]> Signed-off-by: Ryan Nazareth <[email protected]>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
* Set default format of structured dataset to empty Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * last error (#1364) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
* Adds pyflyte CLI reference guide Signed-off-by: Samhita Alla <[email protected]> * bump python version Signed-off-by: Samhita Alla <[email protected]> * bump python version Signed-off-by: Samhita Alla <[email protected]> * resolve docs error Signed-off-by: Samhita Alla <[email protected]> * set nested to none Signed-off-by: Samhita Alla <[email protected]> * remove flyteidl version constraint Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
#1375) Signed-off-by: H. Furkan Vural <[email protected]> Implemented cache overwrite feature is added on flytekit as well for the completeness. In order to support the cache eviction RFC, an overwrite parameter was added, indicating the data store should replace an existing artifact instead of creating a new one on local calls.
…serialized context (#1378) Signed-off-by: Yee Hing Tong <[email protected]>
… when loading module (#1348) Signed-off-by: Ketan Umare <[email protected]>
* Register databricks plugin Signed-off-by: Kevin Su <[email protected]> * Update databricks plugin Signed-off-by: Kevin Su <[email protected]> * register databricks Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
For Pod tasks, if the primary_container_name is not specified, it should default. Signed-off-by: Ketan Umare <[email protected]>
…for --domain (#1381) Co-authored-by: Eduardo Apolinario <[email protected]>
* Add support for Optional to dataclass transformer Signed-off-by: Eduardo Apolinario <[email protected]> * Add one more test Signed-off-by: Eduardo Apolinario <[email protected]> * Add one more test Signed-off-by: Eduardo Apolinario <[email protected]> * Fix serialization of optional flyte types Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Co-authored-by: Kevin Su <[email protected]>
* Added more pod tests and an example pod task Signed-off-by: Ketan Umare <[email protected]> * fixing test and name Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Patrick Brogan <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
* MLFlow plugin in progress Signed-off-by: Ketan Umare <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * update test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update readme Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * dwip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * change experiment name Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Add mlflow to index.rst Signed-off-by: Kevin Su <[email protected]> * use experiment name that user provided Signed-off-by: Kevin Su <[email protected]> * update doc-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Add backend plugin deployment Signed-off-by: Kevin Su <[email protected]> * generate doc for method Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * update docstring Signed-off-by: Niels Bantilan <[email protected]> * update docstring Signed-off-by: Niels Bantilan <[email protected]> * Update tracking.py Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Niels Bantilan <[email protected]>
* Read polars dataframe in a folder Signed-off-by: Kevin Su <[email protected]> * Read polars dataframe in a folder Signed-off-by: Kevin Su <[email protected]> * Load huggingface and spark plugin implicitly Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * remove _pyspark alias Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
* Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add docs page Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add dask models Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <[email protected]> * Remove namespace Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update setup.py Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update docs for images Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update README.md Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <[email protected]> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <[email protected]> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <[email protected]> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <[email protected]> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
* support checkpointing in local mode from cached tasks * clear cache before tests --------- Co-authored-by: Stef Nelson-Lindall <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* Deprecate FlyteSchema Signed-off-by: Kevin Su <[email protected]> * Remove version Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* Use scarf images Signed-off-by: Eduardo Apolinario <[email protected]> * Use scarf names in tests. Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* add reference_launch_plan to flytekit api ref Signed-off-by: Niels Bantilan <[email protected]> * import in init, add docstrings Signed-off-by: Niels Bantilan <[email protected]> * add more to references Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> * fix up docstrings Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Samhita Alla <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
* wip Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Update type Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update dev-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Address comment Signed-off-by: Kevin Su <[email protected]> * upgrade torch Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
* wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * support dynamic task Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lazy compile Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>
* Get the origin type when serializing dataclass Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]>
eapolinario
requested review from
wild-endeavor,
kumare3 and
pingsutw
as code owners
February 22, 2023 22:29
wild-endeavor
previously approved these changes
Feb 22, 2023
Signed-off-by: Eduardo Apolinario <[email protected]>
* Delay initialization of SynchronousFlyteClient in FlyteRemote Signed-off-by: Eduardo Apolinario <[email protected]> * Fix spark plugin flyteremote test. Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
This reverts commit 3798450. Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
This reverts commit 41a9c7a. Signed-off-by: Eduardo Apolinario <[email protected]>
Codecov Report
@@ Coverage Diff @@
## release-v1.2 #1517 +/- ##
===============================================
Coverage ? 69.31%
===============================================
Files ? 304
Lines ? 28601
Branches ? 2709
===============================================
Hits ? 19826
Misses ? 8252
Partials ? 523 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
wild-endeavor
approved these changes
Feb 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Backport recent changes to the release-v1.2 branch
List of backported changes: