Skip to content

Feb 2022 - v0.31.0 Release (please read reminder)

Compare
Choose a tag to compare
@wild-endeavor wild-endeavor released this 08 Mar 22:00
· 1394 commits to master since this release
457f323

Flytekit v0.31.0

Reminders

Please be aware of a couple things coming down the pipe soon...

  • This is (hopefully) the penultimate release for flytekit before the 1.0 release. We have a couple major changes pending, so we plan on releasing v0.32.0 within the next few days as well. (There is a complete overhaul of flytekit's configuration system in the works, and a brand new type forthcoming (unions).)
  • As of v0.32.0, we will flip the Structured Dataset flag to true by default. Please ensure that your backend is on Propeller version v0.16.14 and Admin version v0.6.78 or later (aka Flyte release v0.19.1 or later) otherwise you will get compiler errors if you try to mix FlyteSchema tasks with pd.DataFrame tasks. (But also, you should upgrade to the StructuredDataset type away from FlyteSchema.)
  • We are still planning on releasing flytekit 1.0.0 in mid-April (along with all other Flyte components).

Main changes

Core

  • StructuredDataset (and FlyteSchema) literals can now be cached based on a hash of the dataframe's contents, rather than the storage location. This is helpful if you have a non-cached task that produces a dataframe that a cached task consumes. With this feature, if the parent task produces the same data in the dataframe, the second task will see that as a cache hit. Also applicable to dynamic tasks, see the issue for more information.
    • To turn on this feature just typing.Annotate with a hash function and make sure you're on Propeller v0.16.31 or later (Flyte release version v0.19.3). Documentation forthcoming.
  • New AWS Batch Task type. This allows users to run a task on AWS Batch instead of on a K8s cluster.

FlyteRemote

  • v0.30.0 added support for calling tasks and launchplans fetched from Admin via FlyteRemote in new @workflows. This release changes behind the scenes how they work and also adds support for fetched workflows.

Others

  • We realized there was a shortcoming in the StructuredDatasetDecoder interface and added a field current_task_metadata that represents the metadata of the currently running task, since it may be more or less specific than the incoming task.
  • A bunch of authentication related configuration objects have been removed since Admin has for a long time now been publishing OAuth2 metadata (and our own metadata).
    • Now all you should need is:
      [platform]
      url=flyte.corp.net
      
      since the pkce flow is assumed.
    • For the client credentials flow, you just have to add:
      [credentials]
      client_id=flytepropeller
      client_secret=your.password
      auth_mode=client_credentials  # or basic still works
      scopes=all
      
  • The user-facing logger, (i.e. flytekit.current_context().logging) is now going to be a flytekit logger (see flytekit.loggers). Originally it was just passing through the main Python logging library, so if you had basicConfig() before you'll need to update that, or just manually switch to logging. We also swapped out flytekit internal usage of the main logging library in favor of specific flytekit loggers.
  • New encoder/decoder in fsspec plugin support for StructuredDataset.

Auto Generated Changelog

What's Changed

New Contributors

Full Changelog: v0.30.1...v0.31.0