last error #1364
Merged
last error #1364
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Strip newline from client secret * Add logging and rework the secret file comparison to work on windows Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* Adds docstrings for pod task Adds docstrings for pod task * update task.py fix lint errors Signed-off-by: SmritiSatyanV [email protected] * update task.py Based on review Signed-off-by: SmritiSatyanV [email protected] * update task.py Changes based on review Signed-off-by: SmritiSatyanV <[email protected]>
Add directives for pyflyte run and register Signed-off-by: SmritiSatyanV <[email protected]>
* Overwrite SQLite3Task image Signed-off-by: Kevin Su <[email protected]> * fix lint error Signed-off-by: Kevin Su <[email protected]> * remove comment Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
* flytekit-dbt plugin * Supports dbt run and dbt test tasks * The plugin includes integration test that need local PostgreSQL database Signed-off-by: ariefrahmansyah <[email protected]> * Revert README.md about unit tests Signed-off-by: ariefrahmansyah <[email protected]> * Merge conflicts Signed-off-by: Eduardo Apolinario <[email protected]> * Update requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Move to dbt-sqlite Signed-off-by: Eduardo Apolinario <[email protected]> * Linting Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Delete setup_db.sh Signed-off-by: Eduardo Apolinario <[email protected]> * Fix test_task.py tests Signed-off-by: Eduardo Apolinario <[email protected]> * Move testdata to a separate directory Signed-off-by: Eduardo Apolinario <[email protected]> * Delete unused test Signed-off-by: Eduardo Apolinario <[email protected]> * Use flytekit logger Signed-off-by: Eduardo Apolinario <[email protected]> * Use my fork in the plugins tests Signed-off-by: Eduardo Apolinario <[email protected]> * Get string path for call to touch.touch Signed-off-by: Eduardo Apolinario <[email protected]> * Use pathlib.Path.touch Signed-off-by: Eduardo Apolinario <[email protected]> * Remove the touch package Signed-off-by: Eduardo Apolinario <[email protected]> * Simplify CI Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Simplify CI" This reverts commit 134fa1c. Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Use my fork in the plugins tests" This reverts commit 02ef380. Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused file Signed-off-by: Eduardo Apolinario <[email protected]> * Add dbt to README.md Signed-off-by: Eduardo Apolinario <[email protected]> * Remove "Set up postgres" from CI job Signed-off-by: Eduardo Apolinario <[email protected]> * Leave a note in requirements.in Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused jaffle_shop dir Signed-off-by: Eduardo Apolinario <[email protected]> * Set upperbound flytekit version to 2.0.0 Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: ariefrahmansyah <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: ariefrahmansyah <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* refreshing examples and to_html() method details: - examples on our README were not using the latest whylogs.core.constraints.factories the same way as on flytesnacks. - WhylogsDatasetProfileTransformer.to_html() now returns a ProfileSummary HTML string Signed-off-by: murilommen <[email protected]> * fixing lint on test file Signed-off-by: murilommen <[email protected]> Signed-off-by: murilommen <[email protected]>
* Open HashMethod to all types * Fix test in test_type_hints.py Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Andrew Dye <[email protected]>
…ies (#1192) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-PROTOBUF-3031740
* pyflyte run non-fast register Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]>
* Add support chain multiple tasks Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
* Fix handling of Optional in DataclassTransformer Signed-off-by: Eduardo Apolinario <[email protected]> * Add and excluse clause to the dbt test Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* Bump protobuf versions in requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Move onnx-scikitlearn plugin protobuf version to 3.19.5 Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
* Handle special characters in snowflake query Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
…mote.sync (#1212) * Open HashMethod to all types (#1171) * Open HashMethod to all types * Fix test in test_type_hints.py Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: LN <[email protected]> * Return None for SyncCheckpoint.read() when src is empty (#1189) Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: LN <[email protected]> * fix: plugins/flytekit-k8s-pod/requirements.txt to reduce vulnerabilities (#1192) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-PROTOBUF-3031740 Signed-off-by: LN <[email protected]> * pyflyte non-fast register (#1205) * pyflyte run non-fast register Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: LN <[email protected]> * Update control_plane.rst Added about large response while using Remote.Sync() and how to handle it in the doc Signed-off-by: LN <[email protected]> * Update docs/source/design/control_plane.rst updating FlyteRemote sync message Co-authored-by: Samhita Alla <[email protected]> * Update control_plane.rst Updated the fix for the error and also removed the repeated error message line. Signed-off-by: LN <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: LN <[email protected]> Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: LN <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Andrew Dye <[email protected]> Co-authored-by: Snyk bot <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Samhita Alla <[email protected]>
* ONNX plugin docs Signed-off-by: Samhita Alla <[email protected]> * update flytekit API reference Signed-off-by: Samhita Alla <[email protected]> * make requirements Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Samhita Alla <[email protected]>
…1213) This loads the python_val from the test to match the device logic used in to_python_value. A check is done for the case of PyTorchCheckpoint which does not have a to method.
…type (#1209) * Adds support for modifying allow_pickle and mmap_mode in numpy array type Signed-off-by: Samhita Alla <[email protected]> * ordered dict to dict Signed-off-by: Samhita Alla <[email protected]> * add test Signed-off-by: Samhita Alla <[email protected]> * change metadata default Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Samhita Alla <[email protected]>
* Use console endpoint config if available Signed-off-by: Andrew Dye <[email protected]> * Init platform.console_endpoint Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Andrew Dye <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: arpitbhardwaj <[email protected]> Signed-off-by: arpitbhardwaj <[email protected]>
* Set reasonable defaults for pandas dataframe default renderer * Remove erroneous use of Optional Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
#1239) Signed-off-by: Rahul Mehta <[email protected]> Signed-off-by: Rahul Mehta <[email protected]>
* 3006 - Added a version constraint for grpcio and grpcio-status in order to prevent an incompatible verison of protobuf. Signed-off-by: Dennis O'Brien <[email protected]> * Generated requirements-spark2.txt by running make requirements-spark2.txt Signed-off-by: Dennis O'Brien <[email protected]> Signed-off-by: Dennis O'Brien <[email protected]>
* Single dockerfile Signed-off-by: Eduardo Apolinario <[email protected]> * Install sklearn in the image Signed-off-by: Eduardo Apolinario <[email protected]> * Install deck plugin Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Hongxin Liang <[email protected]>
* feat: add type transformer for sklearn BaseEstimator Signed-off-by: Tushar Mittal <[email protected]> * chore: update requirements Signed-off-by: Tushar Mittal <[email protected]> Signed-off-by: Tushar Mittal <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
* Update fork from upstream * Recommit changes with proper signoff as rebase caused issues Signed-off-by: Kilian <[email protected]> * Add sources output for dbt freshness input Signed-off-by: Kilian <[email protected]> * Fix documentation of dataclass Signed-off-by: Kilian <[email protected]> Signed-off-by: Kilian <[email protected]> Co-authored-by: Kilian <[email protected]>
# TL;DR The bump up recently to setuptools version from 65.5.1 to 65.6.0 caused one of our click unit tests to fail - some interaction between the directory copy utility provided by setuptools.distutils and the python logging library. Getting around this by using shutil instead. Also realized that * `FlyteSchema` transformers doesn't currently handle well you ```python @task def t3(df: FlyteSchema) -> FlyteSchema: return df ``` Because it's never downloaded, there's nothing to upload back. Added a special case to prevent this. * [Nested dataclasses](https://github.com/flyteorg/flytekit/blob/430795d9ee4aa48957554a6eb1446fa80993681a/tests/flytekit/unit/core/test_type_engine.py#L1394) will get transformed multiple times (at least the same `FlyteSchema` instance will) by the type engine. Something to fix for the future. Signed-off-by: Yee Hing Tong <[email protected]>
Use proper VoidPromise in `create_and_link_node_from_remote` Signed-off-by: Hongxin Liang <[email protected]>
Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Remove plugins from flytekit core packaging Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Andres Gomez Ferrer <[email protected]> Signed-off-by: Hongxin Liang <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
…1240) * first commit Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow example tf record transformer Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * correct tfexample description Signed-off-by: Ryan Nazareth <[email protected]> * fix test_native.py Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow docs and reqs Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow docs and reqs1 Signed-off-by: Ryan Nazareth <[email protected]> * tensorflow import in init Signed-off-by: Ryan Nazareth <[email protected]> * fix failing tests Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow pinned version to reqs Signed-off-by: Ryan Nazareth <[email protected]> * pin grpcio-status to remove protobuf error Signed-off-by: Ryan Nazareth <[email protected]> * add suggested changes Signed-off-by: Ryan Nazareth <[email protected]> * redesign transformer Signed-off-by: Ryan Nazareth <[email protected]> * remove old script Signed-off-by: Ryan Nazareth <[email protected]> * fix type reference for TFREcordDataset Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * spacing and uppercase Signed-off-by: Ryan Nazareth <[email protected]> * redesign with tfdir and tfrecordfile subclass Signed-off-by: Ryan Nazareth <[email protected]> * fix conflicts and typos Signed-off-by: Ryan Nazareth <[email protected]> * address majority of comments Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * fix test with flytefile and metadata annotated Signed-off-by: Ryan Nazareth <[email protected]> * fix check for example records in directory Signed-off-by: Ryan Nazareth <[email protected]> * refactor and correct typing Signed-off-by: Ryan Nazareth <[email protected]> * lint Signed-off-by: Ryan Nazareth <[email protected]> * import annotated from typing_extensions Signed-off-by: Ryan Nazareth <[email protected]> * tweak to tests to test case when Config not passed in as type Signed-off-by: Ryan Nazareth <[email protected]> * add suggested changes Signed-off-by: Ryan Nazareth <[email protected]> * add task for tfrecord dir with no config in test Signed-off-by: Ryan Nazareth <[email protected]> * get filenames from local dir instead of remote Signed-off-by: Ryan Nazareth <[email protected]> Signed-off-by: Ryan Nazareth <[email protected]>
Bumps [numpy](https://github.com/numpy/numpy) from 1.21.6 to 1.22.0. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v1.21.6...v1.22.0) --- updated-dependencies: - dependency-name: numpy dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
wild-endeavor
requested review from
kumare3,
eapolinario and
pingsutw
as code owners
…ler for a given dataframe type. if so, when looking for an '' handler, we can use it Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
* Set flyteidl, grpcio, and grpcio-status new lower bounds Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Single out tensorflow tests and fix protobuf type transformer tests Signed-off-by: Eduardo Apolinario <[email protected]> * Comment out skl2onnx and tf2oonx in docs Signed-off-by: Eduardo Apolinario <[email protected]> * More requirements and dev-requirements shenanigans. Signed-off-by: Eduardo Apolinario <[email protected]> * Set env var in doc generation Signed-off-by: Eduardo Apolinario <[email protected]> * Add unit_test_codecov make target Signed-off-by: Eduardo Apolinario <[email protected]> * Aggregate code coverage data and run all unit tests Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
pingsutw
approved these changes
Dec 8, 2022
pingsutw
added a commit
that referenced
this pull request
Dec 9, 2022
* Set default format of structured dataset to empty Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * last error (#1364) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
eapolinario
pushed a commit
that referenced
this pull request
Feb 22, 2023
* Set default format of structured dataset to empty Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * last error (#1364) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
eapolinario
added a commit
that referenced
this pull request
Feb 23, 2023
* Force flyteidl==1.2.9 Signed-off-by: Eduardo Apolinario <[email protected]> * Sanitize query template input in sqlite task (#1359) Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * TypeTransformer for reading and writing from TensorFlowRecord format (#1240) * first commit Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow example tf record transformer Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * correct tfexample description Signed-off-by: Ryan Nazareth <[email protected]> * fix test_native.py Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow docs and reqs Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow docs and reqs1 Signed-off-by: Ryan Nazareth <[email protected]> * tensorflow import in init Signed-off-by: Ryan Nazareth <[email protected]> * fix failing tests Signed-off-by: Ryan Nazareth <[email protected]> * add tensorflow pinned version to reqs Signed-off-by: Ryan Nazareth <[email protected]> * pin grpcio-status to remove protobuf error Signed-off-by: Ryan Nazareth <[email protected]> * add suggested changes Signed-off-by: Ryan Nazareth <[email protected]> * redesign transformer Signed-off-by: Ryan Nazareth <[email protected]> * remove old script Signed-off-by: Ryan Nazareth <[email protected]> * fix type reference for TFREcordDataset Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * spacing and uppercase Signed-off-by: Ryan Nazareth <[email protected]> * redesign with tfdir and tfrecordfile subclass Signed-off-by: Ryan Nazareth <[email protected]> * fix conflicts and typos Signed-off-by: Ryan Nazareth <[email protected]> * address majority of comments Signed-off-by: Ryan Nazareth <[email protected]> * refactor Signed-off-by: Ryan Nazareth <[email protected]> * fix test with flytefile and metadata annotated Signed-off-by: Ryan Nazareth <[email protected]> * fix check for example records in directory Signed-off-by: Ryan Nazareth <[email protected]> * refactor and correct typing Signed-off-by: Ryan Nazareth <[email protected]> * lint Signed-off-by: Ryan Nazareth <[email protected]> * import annotated from typing_extensions Signed-off-by: Ryan Nazareth <[email protected]> * tweak to tests to test case when Config not passed in as type Signed-off-by: Ryan Nazareth <[email protected]> * add suggested changes Signed-off-by: Ryan Nazareth <[email protected]> * add task for tfrecord dir with no config in test Signed-off-by: Ryan Nazareth <[email protected]> * get filenames from local dir instead of remote Signed-off-by: Ryan Nazareth <[email protected]> Signed-off-by: Ryan Nazareth <[email protected]> * update ray plugin dependency (#1361) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Set default format of structured dataset to empty (#1159) * Set default format of structured dataset to empty Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * last error (#1364) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Adds CLI reference for pyflyte (#1362) * Adds pyflyte CLI reference guide Signed-off-by: Samhita Alla <[email protected]> * bump python version Signed-off-by: Samhita Alla <[email protected]> * bump python version Signed-off-by: Samhita Alla <[email protected]> * resolve docs error Signed-off-by: Samhita Alla <[email protected]> * set nested to none Signed-off-by: Samhita Alla <[email protected]> * remove flyteidl version constraint Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Samhita Alla <[email protected]> * Signaling (#1133) Signed-off-by: Yee Hing Tong <[email protected]> * Adding created and updated at to ExecutionClosure model (#1371) Signed-off-by: Yee Hing Tong <[email protected]> * Add Databricks config to Spark Job (#1358) Signed-off-by: Kevin Su <[email protected]> * Add overwrite_cache option the to calls of remote and local executions (#1375) Signed-off-by: H. Furkan Vural <[email protected]> Implemented cache overwrite feature is added on flytekit as well for the completeness. In order to support the cache eviction RFC, an overwrite parameter was added, indicating the data store should replace an existing artifact instead of creating a new one on local calls. * Remove project/domain from being overridden with execution values in serialized context (#1378) Signed-off-by: Yee Hing Tong <[email protected]> * Use TaskSpec instead of TaskTemplate for fetch_task and avoid network when loading module (#1348) Signed-off-by: Ketan Umare <[email protected]> * Register Databricks config (#1379) * Register databricks plugin Signed-off-by: Kevin Su <[email protected]> * Update databricks plugin Signed-off-by: Kevin Su <[email protected]> * register databricks Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * PodSpec should not require primary_container name (#1380) For Pod tasks, if the primary_container_name is not specified, it should default. Signed-off-by: Ketan Umare <[email protected]> * fix(pyflyte): change -d to -D for --destination-dir as -d is already for --domain (#1381) Co-authored-by: Eduardo Apolinario <[email protected]> * Handle Optional[FlyteFile] in Dataclass type transformer (#1393) * Add support for Optional to dataclass transformer Signed-off-by: Eduardo Apolinario <[email protected]> * Add one more test Signed-off-by: Eduardo Apolinario <[email protected]> * Add one more test Signed-off-by: Eduardo Apolinario <[email protected]> * Fix serialization of optional flyte types Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * add FastSerializationSettings to docs (#1386) Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Added more pod tests and an example pod task (#1382) * Added more pod tests and an example pod task Signed-off-by: Ketan Umare <[email protected]> * fixing test and name Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Ketan Umare <[email protected]> * Convert default dict to json string in pyflyte run (#1399) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * docs: update register help, non-fast version is supported (#1402) Signed-off-by: Patrick Brogan <[email protected]> * Update log level for structured dataset (#1394) Signed-off-by: Kevin Su <[email protected]> * Add Niels to code owners (#1404) Signed-off-by: Kevin Su <[email protected]> * Signal use (#1398) Signed-off-by: Yee Hing Tong <[email protected]> * User Documentation Proposal (#1200) Signed-off-by: Kevin Su <[email protected]> * Add support MLFlow plugin (#1274) * MLFlow plugin in progress Signed-off-by: Ketan Umare <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * update test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update readme Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * dwip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * change experiment name Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Add mlflow to index.rst Signed-off-by: Kevin Su <[email protected]> * use experiment name that user provided Signed-off-by: Kevin Su <[email protected]> * update doc-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Add backend plugin deployment Signed-off-by: Kevin Su <[email protected]> * generate doc for method Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * update docstring Signed-off-by: Niels Bantilan <[email protected]> * update docstring Signed-off-by: Niels Bantilan <[email protected]> * Update tracking.py Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> * fix remote API reference (#1405) Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> * Read structured dataset from a folder (#1406) * Read polars dataframe in a folder Signed-off-by: Kevin Su <[email protected]> * Read polars dataframe in a folder Signed-off-by: Kevin Su <[email protected]> * Load huggingface and spark plugin implicitly Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * remove _pyspark alias Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Update default config to work out-of-the-box with flytectl demo (#1384) Signed-off-by: Niels Bantilan <[email protected]> * Add dask plugin #patch (#1366) * Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add docs page Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add dask models Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <[email protected]> * Remove namespace Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update setup.py Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update docs for images Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update README.md Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <[email protected]> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <[email protected]> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <[email protected]> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <[email protected]> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <[email protected]> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Add support for overriding task configurations (#1410) Signed-off-by: Kevin Su <[email protected]> * Warning if git is not installed (#1414) * warning if git is not installed Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Flip the settings for channel and logger (#1415) Signed-off-by: Yee Hing Tong <[email protected]> * Preserving Exception in the LazyEntity fetch (#1412) * Preserving Exception in the LazyEntity fetch Signed-off-by: Ketan Umare <[email protected]> * updated lint error Signed-off-by: Ketan Umare <[email protected]> * more tests Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Ketan Umare <[email protected]> * [Docs] SynchronousFlyteClient API reference #3095 (#1416) Signed-off-by: Peeter Piegaze <[email protected]> Signed-off-by: Peeter Piegaze <[email protected]> Co-authored-by: Peeter Piegaze <[email protected]> Co-authored-by: Haytham Abuelfutuh <[email protected]> * Return error code on fail (#1408) * AWS batch return error code once it fails Signed-off-by: Kevin Su <[email protected]> * AWS batch return error code once it fails Signed-off-by: Kevin Su <[email protected]> * update tests Signed-off-by: Kevin Su <[email protected]> * Update tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * wrapping flyte entity in a task node in call to flyte node constructor, not sure if integration tests are actually running (#1422) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Sqlalchemy multiline query (#1421) * SQLAlchemyTask should handle multiline strings for query template Signed-off-by: Niels Bantilan <[email protected]> * sqlalchemy supports multi-line query Signed-off-by: Niels Bantilan <[email protected]> * update base sql task Signed-off-by: Niels Bantilan <[email protected]> * remove space Signed-off-by: Niels Bantilan <[email protected]> * fix snowflake tests Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * fix test Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> * Sklearn type transformer should be automatically loaded with import flytekit (#1423) * add flytekit.extras.sklearn to main __init__ import Signed-off-by: Niels Bantilan <[email protected]> * fix docs Signed-off-by: Niels Bantilan <[email protected]> * add temporary docs/requirements.txt for onnx plugins Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * Bump isort to 5.12.0 (#1427) Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Fixes guess type bug in UnionTransformer (#1426) Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Add `pod_template` and `pod_template_name` arguments for `PythonAutoContainerTask`, its downstream tasks, and `@task`. (#1425) * Add `pod_template` and `pod_template_name` arguments for `PythonAutoContainerTask`, its downstream tasks, and `@task` Signed-off-by: byhsu <[email protected]> * clean Signed-off-by: byhsu <[email protected]> * fix test Signed-off-by: byhsu <[email protected]> * Fix taskmetadata Signed-off-by: byhsu <[email protected]> * add kubernetes in setup.py Signed-off-by: byhsu <[email protected]> * address comments Signed-off-by: byhsu <[email protected]> * Regenerate requirements using python 3.7 Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: byhsu <[email protected]> * keep container validation Signed-off-by: byhsu <[email protected]> * bump idl version Signed-off-by: byhsu <[email protected]> * Regenerate requirements using python 3.7 Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate doc-requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> * fix Signed-off-by: byhsu <[email protected]> --------- Signed-off-by: byhsu <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Auto Backfill workflow (#1420) * Fix primitive decoder when evaluating Promise (#1432) Signed-off-by: Samhita Alla <[email protected]> * set maximum python version to 3.10 (#1433) * set maximum python version to 3.10 Signed-off-by: Niels Bantilan <[email protected]> * remove unneeded python version check Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * Revert "Remove project/domain from being overridden with execution values in serialized context (#1378)" (#1460) * Revert "Remove project/domain from being overridden with execution values in serialized context (#1378)" This reverts commit b3bfef5. * Import os Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Support checkpointing in local mode from cached tasks (#1457) * support checkpointing in local mode from cached tasks * clear cache before tests --------- Co-authored-by: Stef Nelson-Lindall <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Deprecate FlyteSchema (#1418) * Deprecate FlyteSchema Signed-off-by: Kevin Su <[email protected]> * Remove version Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Use scarf images (#1434) * Use scarf images Signed-off-by: Eduardo Apolinario <[email protected]> * Use scarf names in tests. Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * add undocumented objects/functions to flytekit api ref (#1502) * add reference_launch_plan to flytekit api ref Signed-off-by: Niels Bantilan <[email protected]> * import in init, add docstrings Signed-off-by: Niels Bantilan <[email protected]> * add more to references Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> * fix up docstrings Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Samhita Alla <[email protected]> * Use non-root user in default flytekit image (#1417) Signed-off-by: Kevin Su <[email protected]> * Fix PyTorch transformer (#1510) Signed-off-by: Samhita Alla <[email protected]> * Fix mypy errors (#1313) * wip Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Update type Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update dev-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Address comment Signed-off-by: Kevin Su <[email protected]> * upgrade torch Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Compile the workflow only at compile time (#1311) * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * support dynamic task Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lazy compile Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Get the origin type when serializing dataclass (#1508) * Get the origin type when serializing dataclass Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> * Fix bad merge Signed-off-by: Eduardo Apolinario <[email protected]> * Delay initialization of SynchronousFlyteClient in FlyteRemote (#1514) * Delay initialization of SynchronousFlyteClient in FlyteRemote Signed-off-by: Eduardo Apolinario <[email protected]> * Fix spark plugin flyteremote test. Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Set flytekit and flyteidl bounds in plugins tests Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Fix mypy errors (#1313)" This reverts commit 3798450. Signed-off-by: Eduardo Apolinario <[email protected]> * Fix requirements in dask and ray plugins Signed-off-by: Eduardo Apolinario <[email protected]> * Fix papermill tests requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Fix doc-requirements Signed-off-by: Eduardo Apolinario <[email protected]> * dask plugin requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Add dask plugin #patch (#1366)" This reverts commit 41a9c7a. Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Ryan Nazareth <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Patrick Brogan <[email protected]> Signed-off-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: Peeter Piegaze <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: byhsu <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ryan Nazareth <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: H. Furkan Vural <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: mcloney-ddm <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: pbrogan12 <[email protected]> Co-authored-by: bstadlbauer <[email protected]> Co-authored-by: Peeter Piegaze <[email protected]> Co-authored-by: Peeter Piegaze <[email protected]> Co-authored-by: Haytham Abuelfutuh <[email protected]> Co-authored-by: ByronHsu <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: Stef Lindall <[email protected]> Co-authored-by: Stef Nelson-Lindall <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yee Hing Tong [email protected]
TL;DR
Please replace this text with a description of what this PR accomplishes.
Type
Are all requirements met?
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/