Feb 2022 - v0.31.0 Release (please read reminder)
wild-endeavor
released this
08 Mar 22:00
·
1394 commits
to master
since this release
Flytekit v0.31.0
Reminders
Please be aware of a couple things coming down the pipe soon...
- This is (hopefully) the penultimate release for flytekit before the 1.0 release. We have a couple major changes pending, so we plan on releasing v0.32.0 within the next few days as well. (There is a complete overhaul of flytekit's configuration system in the works, and a brand new type forthcoming (unions).)
- As of v0.32.0, we will flip the Structured Dataset flag to true by default. Please ensure that your backend is on Propeller version v0.16.14 and Admin version v0.6.78 or later (aka Flyte release v0.19.1 or later) otherwise you will get compiler errors if you try to mix
FlyteSchema
tasks withpd.DataFrame
tasks. (But also, you should upgrade to theStructuredDataset
type away fromFlyteSchema
.) - We are still planning on releasing flytekit 1.0.0 in mid-April (along with all other Flyte components).
Main changes
Core
StructuredDataset
(andFlyteSchema
) literals can now be cached based on a hash of the dataframe's contents, rather than the storage location. This is helpful if you have a non-cached task that produces a dataframe that a cached task consumes. With this feature, if the parent task produces the same data in the dataframe, the second task will see that as a cache hit. Also applicable to dynamic tasks, see the issue for more information.- To turn on this feature just
typing.Annotate
with a hash function and make sure you're on Propeller v0.16.31 or later (Flyte release version v0.19.3). Documentation forthcoming.
- To turn on this feature just
- New AWS Batch Task type. This allows users to run a task on AWS Batch instead of on a K8s cluster.
FlyteRemote
v0.30.0
added support for calling tasks and launchplans fetched from Admin viaFlyteRemote
in new@workflow
s. This release changes behind the scenes how they work and also adds support for fetched workflows.
Others
- We realized there was a shortcoming in the
StructuredDatasetDecoder
interface and added a fieldcurrent_task_metadata
that represents the metadata of the currently running task, since it may be more or less specific than the incoming task. - A bunch of authentication related configuration objects have been removed since Admin has for a long time now been publishing OAuth2 metadata (and our own metadata).
- Now all you should need is:
since the pkce flow is assumed.
[platform] url=flyte.corp.net
- For the client credentials flow, you just have to add:
[credentials] client_id=flytepropeller client_secret=your.password auth_mode=client_credentials # or basic still works scopes=all
- Now all you should need is:
- The user-facing logger, (i.e.
flytekit.current_context().logging
) is now going to be a flytekit logger (seeflytekit.loggers
). Originally it was just passing through the main Pythonlogging
library, so if you hadbasicConfig()
before you'll need to update that, or just manually switch to logging. We also swapped out flytekit internal usage of the mainlogging
library in favor of specific flytekit loggers. - New encoder/decoder in
fsspec
plugin support forStructuredDataset
.
Auto Generated Changelog
What's Changed
- Assign input and output to FlyteWorkflowExecution by @pingsutw in #842
- Add reference entities to FlyteTask and FlyteLaunchPlan by @wild-endeavor in #850
- Fix fast registration error by @wild-endeavor in #851
- Add support for local execute in pod task by @ggydush-fn in #852
- Add anonymous retry by @wild-endeavor in #854
- [Core feature] Add Raw AWS Batch Task by @pingsutw in #782
- Add structured dataset encoder/decoder in fsspec plugin by @pingsutw in #849
- Delete unnecessary auth configuration by @wild-endeavor in #858
- Fixed format alias in Flytekit docs by @SmritiSatyanV in #844
- Bump idl by @wild-endeavor in #862
- Updated authoring.rst by @SmritiSatyanV in #863
- Updated authoring.rst by @SmritiSatyanV in #866
- Change docs for HTMLPage type to say HTMLPage instead of PNGImage by @NotMatthewGriffin in #868
- Revisit StructuredDatasetDecoder interface by @pingsutw in #865
- Quick cleanup of contributing.rst by @wild-endeavor in #870
- Add GCS protocol in the structured dataset by @pingsutw in #869
- Make fetched entities callable within workflows by @wild-endeavor in #867
- Caching of offloaded objects by @eapolinario in #762
- FlyteRemote fetch of conditional nodes by @wild-endeavor in #772
- Removed root logger by @pingsutw in #871
- Remove extraneous calls to set logging level by @wild-endeavor in #876
- Fix to support sphinx doctest with task decorator by @ggydush-fn in #873
- [Docs] Link to data doc in FlyteFile by @wild-endeavor in #877
- Fixed lint errors by @pingsutw in #878
New Contributors
- @ggydush-fn made their first contribution in #852
- @SmritiSatyanV made their first contribution in #844
- @NotMatthewGriffin made their first contribution in #868
Full Changelog: v0.30.1...v0.31.0