Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core Feature] [flytekit] Run dispatch_execute and trigger type engine during local task execution #1032

Closed
wild-endeavor opened this issue May 13, 2021 · 0 comments
Labels
enhancement New feature or request flytekit FlyteKit Python related issue needs discussion
Milestone

Comments

@wild-endeavor
Copy link
Contributor

wild-endeavor commented May 13, 2021

Motivation: Why do you think this is important?

Generally we made an effort during the development of flytekit to make the local testing experience as similar to what happens in a real backend run as possible. This was not done in one case - local task execution.

In local task execution, it was originally thought to be a feature that when an @task decorated function was run, say inside a unit test, flytekit steps completely out of the way and the raw function is just run.

More and more, this seems like a mistake.

  1. This makes unit test behavior differ potentially. When a task is run in a unit test today, we just run the function. But if you wrap the task into a workflow that does nothing except call the task, then flytekit does not step out of the way: the type engine is triggered and inputs and outputs are translated between Python and Flyte literal types.
  2. Types that rely on the TypeEngine getting invoked (like FlyteFile, FlyteDirectory, and FlyteSchema) behave differently given 1.
  3. There doesn't seem to be good reason to diverge from mantra of maintain as much similarity as possible between unit testing and live production run.

Goal: What should the final outcome look like, ideally?

The behavior for running a task locally by itself and running a task locally inside a workflow should be the same.

@wild-endeavor wild-endeavor added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers flytekit FlyteKit Python related issue needs discussion labels May 13, 2021
@kumare3 kumare3 modified the milestones: 0.15.0, 0.16.0 Jun 7, 2021
@EngHabu EngHabu modified the milestones: 0.16.0, 0.17.0 Aug 2, 2021
@wild-endeavor wild-endeavor removed the untriaged This issues has not yet been looked at by the Maintainers label Aug 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request flytekit FlyteKit Python related issue needs discussion
Projects
None yet
Development

No branches or pull requests

3 participants