Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pickle as a default Transformer rather than a special case of function signatures #1661

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

elibixby
Copy link
Contributor

@elibixby elibixby commented May 28, 2023

TL;DR

Right now only "top level" missing types (those defined directly in a function) are pickled if they don't have a registered TypeTransformer. This PR alters that behavior to anywhere TypeTransformers are used.

Type

  • Bug Fix
  • [X ] Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

The special casing for pickling as a serialization/deserialization method leads to some strange behavior where, e.g


@task
def my_task(foo: Foo):
   ...

Is valid, because that type is converted, but e.g. Foo inside of a dataclass etc is not. Because the conversion to the placeholder FlytePickle type happens in a function which is only called once on top level type, and only traverses the origin stack, rather than being used anywhere TypeTransformers are used. This PR fixes that problem, while also significantly simplifying the logic to support it.

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

@kumare3
Copy link
Contributor

kumare3 commented Jan 2, 2024

this seems to be abandoned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants