Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[release v1.2] Backport v1.6 (#1645)
* Data subsystem (#1526) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Pluck retry from flytekit and into sagemaker (#1411) * Remove retry from flytekit's setup.py and regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Add to sagemaker Signed-off-by: Eduardo Apolinario <[email protected]> * Remove retry from sagemaker plugin requirements file Signed-off-by: Eduardo Apolinario <[email protected]> * Restore doc-requirements.txt Signed-off-by: eduardo apolinario <[email protected]> * Fix bad merge Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Update the pypi wait (#1554) Signed-off-by: Yee Hing Tong <[email protected]> * Stream Directories and Files using Flyte (#1512) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Make `FlyteFile` compatible with `Annotated[..., HashMethod]` (#1544) * fix: Make FlyteFile compatible with Annotated[..., HashMethod] See issue #3424 Signed-off-by: Adrian Rumpold <[email protected]> * tests: Add test case for FlyteFile with HashMethod annotation Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix: Use typing_extensions.Annotated for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix: Use `get_args` and `get_origin` from typing_extensions for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix(tests): Use fixture for local dummy file Signed-off-by: Adrian Rumpold <[email protected]> --------- Signed-off-by: Adrian Rumpold <[email protected]> * move FlyteSchema deprecation warning to initialization method (#1558) * move FlyteSchema deprecation warning to initialization method Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * add pod_template and pod_template_name arguments for ContainerTask (#1515) * add pod_template and pod_template_name arguments for ContainerTask Signed-off-by: Felix Ruess <[email protected]> * factor out _serialize_pod_spec into separate util function Signed-off-by: Felix Ruess <[email protected]> * model file changes, couple other changes Signed-off-by: Yee Hing Tong <[email protected]> * minor cleanup Signed-off-by: Felix Ruess <[email protected]> * add unit test for container_task pod_template Signed-off-by: Felix Ruess <[email protected]> * bump min version of flyteidl to 1.3.12 for pod template data config support Signed-off-by: Felix Ruess <[email protected]> * require flyteidl==1.3.12 in doc-requirements.txt Signed-off-by: Felix Ruess <[email protected]> --------- Signed-off-by: Felix Ruess <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Pass locally defined scopes to RemoteClientConfigStore (#1553) Signed-off-by: franco-bocci <[email protected]> * TypeTransformer for TensorFlow model (#1562) * TypeTransformer for TensorFlow model Signed-off-by: Samhita Alla <[email protected]> * clean up Signed-off-by: Samhita Alla <[email protected]> * clean up Signed-off-by: Samhita Alla <[email protected]> * fix lint Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> * Remove fsspec flytekit plugin from main Dockerfile (#1561) Signed-off-by: Yee Hing Tong <[email protected]> * Device auth flow / Headless auth (#1552) * Device auth flow Signed-off-by: Ketan Umare <[email protected]> * Device AuthFlow is now available in flytekit Signed-off-by: Ketan Umare <[email protected]> * unit tests Signed-off-by: Ketan Umare <[email protected]> * test added Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * Fixed test Signed-off-by: Ketan Umare <[email protected]> * Fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Fixed lint errors Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * url encode secret in client credentials flow (#1566) * url encode secret first Signed-off-by: Yee Hing Tong <[email protected]> * nit Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * Python run multiple files (#1559) * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * General Partial support in flytekit and multi-list support in flytekit (#1556) Signed-off-by: Ketan Umare <[email protected]> * fix: Silence keyring warnings by changing to debug (#1568) Co-authored-by: ggydush-fn <[email protected]> * Support GCP secrets (#1571) Signed-off-by: Yee Hing Tong <[email protected]> * Automatically remove unused import (#1574) * Automatically remove unused import Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Disallow partial lists in map tasks (#1577) * Disallow partial lists in map tasks Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Remove duplicate reporting logic (#1578) Signed-off-by: Yee Hing Tong <[email protected]> * [Core feature] Convert List[Any] to a single pickle file (#1535) * Convert List[Any] to a single pickle file * remove redundant code * keep batchSize only if type contain flytePickle * fix error * add batch support to translate_inputs_to_literals * add ci test Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve comment Signed-off-by: Yicheng-Lu-llll <[email protected]> * add more ci test Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * handle HashMethod case Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve format Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * add test_is_batchable Signed-off-by: Yicheng-Lu-llll <[email protected]> * Add BatchSize Signed-off-by: Kevin Su <[email protected]> * test_batch_size Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Improve authoring structure documentation (#1572) Signed-off-by: Samhita Alla <[email protected]> * Unify sqlalchemy Dockerfiles (#1585) * Unify sqlalchemy Dockerfiles Signed-off-by: eduardo apolinario <[email protected]> * Simplifyte sqlalchemy Dockerfile Signed-off-by: eduardo apolinario <[email protected]> * Set pythonpath to /root Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Pyflyte build imageSpec (#1555) * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * pyflyte build Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * Support serialize and package Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * move to plugin Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tested Signed-off-by: Kevin Su <[email protected]> * fixed tested Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * Add support passing yaml in pyflyte run Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feature/3506: Improve Type Conversion errors, use rich to prettify error messages (#1582) * improve input type conversion error Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * fix Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add rich Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * remove prototyping script, update loggers Signed-off-by: Niels Bantilan <[email protected]> * update __init__.py Signed-off-by: Niels Bantilan <[email protected]> * update logger Signed-off-by: Niels Bantilan <[email protected]> * update logger Signed-off-by: Niels Bantilan <[email protected]> * fix GE and pandera tests Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * Add support nested FlyteFile in pyflyte run (#1587) * Add support nested FlyteFile in pyflyte run Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Set cache_regions=True in the case of s3fs (#1592) * Set cache_regions=True in the case of s3fs * Linting Signed-off-by: eduardo apolinario <[email protected]> * Fix tests Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Set default ComSpec when running on Windows (#1595) * set default ComSpec when running on Windows Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feature: add activate-launchplan command to pyflyte (#1588) * add activate-launchplan command to pyflyte Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> * rework launchplan command Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> --------- Signed-off-by: Artem Petrov <[email protected]> * nit (#1596) Signed-off-by: Yee Hing Tong <[email protected]> * Add log streaming to papermill plugin (#1129) * checkpoint Signed-off-by: Mike Zhong <[email protected]> * Experimental implementation works well. Instead of messing with the class, we utilize the interpolation and context to dynamically generate a wrapper around the desired script. In the wrapper, we cd to the ctx.working_directory, export the env variables (handled by an additional method to convert dict to str), and then pass the arguments to the script Signed-off-by: Mike Zhong <[email protected]> * fix spacing Signed-off-by: Mike Zhong <[email protected]> * remove breakpoint Signed-off-by: Mike Zhong <[email protected]> * Output ctx.working_directory as single output Signed-off-by: Mike Zhong <[email protected]> * more doc strings Signed-off-by: Mike Zhong <[email protected]> * more comments Signed-off-by: Mike Zhong <[email protected]> * Added comments Signed-off-by: Mike Zhong <[email protected]> * Add tests and test files from other branch Signed-off-by: Mike Zhong <[email protected]> * minor fix, have function return the instance rather than create. It seems to get registered when flyte packages your project Signed-off-by: Mike Zhong <[email protected]> * fix tests Signed-off-by: Mike Zhong <[email protected]> * remove set flags not supported by sh Signed-off-by: Mike Zhong <[email protected]> * fix spellcheck lint errors Signed-off-by: Mike Zhong <[email protected]> * don't have a windows equivalent test script so bypassing those tests for now Signed-off-by: Mike Zhong <[email protected]> * Add typing to make_export_string_from_env_dict params Signed-off-by: Mike Zhong <[email protected]> * fixup doc string Signed-off-by: Mike Zhong <[email protected]> * Refactored the new behavior out into a new class. Did not change implementation details at all Signed-off-by: Mike Zhong <[email protected]> * Address linter issues and up test cov Signed-off-by: Mike Zhong <[email protected]> * Skip test on windows, no equivalent script Signed-off-by: Mike Zhong <[email protected]> * Run black and isort, address SC2236 Signed-off-by: Mike Zhong <[email protected]> * Refactored class name to remove _, make utility function require name so multiple uses in a workflow don't break Signed-off-by: Mike Zhong <[email protected]> * fix utility function Signed-off-by: Mike Zhong <[email protected]> * fix utility function call in tests Signed-off-by: Mike Zhong <[email protected]> * Fix isort on test_shell.py Signed-off-by: Mike Zhong <[email protected]> * adding logging settings to papermill plugin Signed-off-by: Calvin Leather <[email protected]> * set level to info logging * improved comments/docs --------- Signed-off-by: Mike Zhong <[email protected]> Signed-off-by: Calvin Leather <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: Mike Zhong <[email protected]> * Set lower bound for s3fs (#1598) * set lower bound for s3fs Signed-off-by: Kevin Su <[email protected]> * update doc-requirements.txt Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Implement horovod task in mpi plugin (#1575) * Add horovod task to mpi plugin Signed-off-by: byhsu <[email protected]> * Remove unused Signed-off-by: byhsu <[email protected]> * add test Signed-off-by: byhsu <[email protected]> * inherit from mpifunctiontask Signed-off-by: byhsu <[email protected]> * fix format Signed-off-by: byhsu <[email protected]> * fix format Signed-off-by: byhsu <[email protected]> --------- Signed-off-by: byhsu <[email protected]> Co-authored-by: byhsu <[email protected]> * add ImageRender class for Flyte Decks (#1599) Signed-off-by: esad <[email protected]> * Bump tensorflow from 2.10.0 to 2.11.1 in /plugins/flytekit-mlflow (#1586) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.10.0 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](tensorflow/tensorflow@v2.10.0...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Handle batchable lists of length 0 (#1600) Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * pyflyte run imperative workflow (#1597) * pyflyte run imperative workflow Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Backfill should fill in the right input vars (#1593) * Backfill fix - Backfill was using incorrect arguments - backfill should use the argument that user provides or none at all Signed-off-by: Ketan Umare <[email protected]> * Updated code Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * Pyflyte prettified (#1602) * Pyflyte prettified Signed-off-by: Ketan Umare <[email protected]> * added dependency Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * pyflyte run now supports json/yaml files (#1606) * pyflyte run now supports json files Signed-off-by: Ketan Umare <[email protected]> * added yaml support Signed-off-by: Ketan Umare <[email protected]> * fixed parsing Signed-off-by: Ketan Umare <[email protected]> * fixed windows test Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * Add is_inside func to ImageSpec (#1601) * Add is_inside func to ImageSpec Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * Add is_inside func to ImageSpec Signed-off-by: Kevin Su <[email protected]> * Add tests Signed-off-by: Kevin Su <[email protected]> * rename Signed-off-by: Kevin Su <[email protected]> * rename Signed-off-by: Kevin Su <[email protected]> * check execution mode Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Measuring the time taken for each component when executing a task (#1581) * add time measurement framework Signed-off-by: Yicheng-Lu-llll <[email protected]> * add time measurement framework Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> --------- Signed-off-by: Yicheng-Lu-llll <[email protected]> * Skip the image building process if the check for its existence fails (#1614) * Fixed tests Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * add lru cache Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Enable torch elastic training (torchrun) (#1603) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> This PR brings [torch elastic training (`torchrun`)](https://pytorch.org/docs/stable/elastic/run.html) to the pytorch plugin: ```python from flytekitplugins.kfpytorch import Elastic @task( task_config=Elastic( replicas=4, nproc_per_node=4, ... ), ... ) def train(...): ... ``` flyteorg/flyte#3614 * update sphinx youtube plugin (#1619) * update sphinx youtube plugin Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Read polar dataframe without copying to local (#1618) Signed-off-by: Kevin Su <[email protected]> * Add default spark image (#1616) * Add default spark image Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Lazy load modules (#1590) * lazy load module Signed-off-by: Kevin Su <[email protected]> * lazy load module Signed-off-by: Kevin Su <[email protected]> * import Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * keep structured dataset in flytekit init Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tess Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tests Signed-off-by: Kevin Su <[email protected]> * fixed tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * move import pandas to __init__ Signed-off-by: Kevin Su <[email protected]> * use lazy import loader instead Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * regular import Signed-off-by: Kevin Su <[email protected]> * fixed tests Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Override node name with a name containing underscore (#1608) * override node_name with a name containing underscore Signed-off-by: Kevin Su <[email protected]> * add comment Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * External Plugin Service (grpc) (#1524) External Plugin Service * Include traceback in errors from admin (#1621) Signed-off-by: Yee Hing Tong <[email protected]> * Add support for copying all the files in source root (#1622) * Add support for copying all the files in source root Signed-off-by: Kevin Su <[email protected]> * Add tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * fix PipelineModel transformer issue 3648 (#1623) Signed-off-by: esad <[email protected]> * update GH workflow for external plugin service (#1624) Signed-off-by: Kevin Su <[email protected]> * [Issue-3617] Enables FlyteFiles, FlyteDirectors, and StructuredDatasets inputs in papermill plugin (#1612) * Add support for using a list as an input for a subworkflow (#1605) Signed-off-by: Kevin Su <[email protected]> * Improve task decorator type hints with overload (#1631) Without the overload, the decorated function does not have the proper type of PythonFunctionTask, leading to spurious type errors when attempting to register the task on a FlyteRemote object Signed-off-by: Matthew Hoffman <[email protected]> * add annotation option for serialization (#1615) * add annotation option for serialization * add support for guessing type * formatting * test both directions * remove guessing for annotated types * clarify usage in test * Update tests/flytekit/unit/core/test_type_engine.py --------- Co-authored-by: Eli Bixby <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Delete removed data persistence classes from docs (#1633) Signed-off-by: Niels Bantilan <[email protected]> * Improve workflow decorator type hints with overload (#1635) Previously, the workflow decorator is hinted as always returning a WorkflowBase, which is not true when _workflow_function is None; similar to #1631, we propose using typing.overload to differentiate the return type of workflow based on the value of _workflow_function Signed-off-by: Matthew Hoffman <[email protected]> * Fix mypy errors (#1313) * wip Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Update type Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update dev-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Address comment Signed-off-by: Kevin Su <[email protected]> * upgrade torch Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * type-protobuf<4 Signed-off-by: eduardo apolinario <[email protected]> * Allow for python 3.7 Signed-off-by: eduardo apolinario <[email protected]> * Fix protobuf transformer merge error * Fix bad merge of s/_task_models/task_models/ * Add maxsize=128 to @lru_cache to cover python 3.7 Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> * Regenerate doc-requirements.txt Signed-off-by: eduardo apolinario <[email protected]> * Add missing call to lru_cache Signed-off-by: eduardo apolinario <[email protected]> * Bump version and regenerate requirements in a few plugins Signed-off-by: eduardo apolinario <[email protected]> * Revert "Fix mypy errors (#1313)" This reverts commit af49155. Signed-off-by: Eduardo Apolinario <[email protected]> * Remove docs Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> * Fix type of get_type_for_output_var Signed-off-by: eduardo apolinario <[email protected]> * Remove tensorflow test_model * Replace use of shutil.copytree Signed-off-by: eduardo apolinario <[email protected]> * Other use of copytree Signed-off-by: eduardo apolinario <[email protected]> * Stringify paths in test_script_mode Signed-off-by: eduardo apolinario <[email protected]> * os.path.isfile instead in imagespec Signed-off-by: eduardo apolinario <[email protected]> * Remove tests Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> * Remove flaky test Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Felix Ruess <[email protected]> Signed-off-by: franco-bocci <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Artem Petrov <[email protected]> Signed-off-by: Mike Zhong <[email protected]> Signed-off-by: Calvin Leather <[email protected]> Signed-off-by: byhsu <[email protected]> Signed-off-by: esad <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Adrian Rumpold <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Felix Ruess <[email protected]> Co-authored-by: Franco Bocci <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: ggydush <[email protected]> Co-authored-by: ggydush-fn <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Artem <[email protected]> Co-authored-by: Calvin Leather <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: ByronHsu <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: peridotml <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Eli Bixby <[email protected]> Co-authored-by: Eli Bixby <[email protected]>
- Loading branch information
26 people
authored
May 17, 2023
1 parent
f7b59bc
commit 4fd5d81