Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Make FlyteFile, FlyteDirectory, and StructuredDataset pickleable #6144

Open
2 tasks done
cosmicBboy opened this issue Jan 6, 2025 · 3 comments
Open
2 tasks done
Labels
bug Something isn't working

Comments

@cosmicBboy
Copy link
Contributor

Describe the bug

Currently, FlyteFile and FlyteDirectory objects with a non-default _downloader function (i.e. a _downloader that isn't the default noop function) cannot be pickled.

      File "/opt/micromamba/envs/runtime/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
        ForkingPickler(file, protocol).dump(obj)
    AttributeError: Can't pickle local object 'FlyteDirToMultipartBlobTransformer.async_to_python_value.<locals>._downloader'

Message:

    AttributeError: Can't pickle local object 'FlyteDirToMultipartBlobTransformer.async_to_python_value.<locals>._downloader'

This error is raised when, e.g. users specify FlyteDirectory/File task inputs together with the Elastic plugin, which relies on pickling the inputs when running the task function code in parallel.

Expected behavior

This should just work.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@cosmicBboy cosmicBboy added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Jan 6, 2025
@wild-endeavor
Copy link
Contributor

@cosmicBboy does this issue impact structureddataset?

@cosmicBboy
Copy link
Contributor Author

Oh, yeah I think it does

@wild-endeavor
Copy link
Contributor

mind putting in the fix if it does?

@eapolinario eapolinario changed the title [BUG] Make FlyteFile and FlyteDirectory pickleable [BUG] Make FlyteFile, FlyteDirectory, and StructuredDataset pickleable Jan 16, 2025
@eapolinario eapolinario removed the untriaged This issues has not yet been looked at by the Maintainers label Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Backlog
Development

No branches or pull requests

3 participants