Skip to content

Flytekit v0.30.0
Compare
Choose a tag to compare
@eapolinario eapolinario released this 03 Feb 22:47
· 1449 commits to master since this release
v0.30.0
9dcf320
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Flytekit v0.30.0

Important Notes:

  • This release introduces a new type (see the section on Structured Dataset below). Naturally this means changes to the IDL, which means that tools that work with Flyte entities encode the new type will also need to be updated. For most readers of this note, that means FlyteRemote (which comes with flytekit).
    • Note: This feature is turned off by default (will be on by default at the 1.0.0 release) because this new type needs a Propeller and Admin upgrade. Turn it on via the env var FLYTE_SDK_USE_STRUCTURED_DATASET=TRUE. See more below.
  • This release also removes the legacy API (see more below). Because of this change, if you bump up your flytekit version, you'll also need to bump all your flytekitplugins- to the same version as well, otherwise you might see missing imports.
  • Lastly, this release skips a few version numbers from 0.26 to 0.30 for future-proofing.

Notable Changes

New Type - Structured Dataset

To handle the new type, you will need to be on Propeller version v0.16.14 and Admin version v0.6.78 or later. Because of this backend upgrade restriction, we've decided to feature-gate this new type. To turn this feature on, you will need to set the environment variable:

FLYTE_SDK_USE_STRUCTURED_DATASET=TRUE

Note that we'll be turning on this feature by default at the 1.0.0 release of flytekit (and Flyte generally) in Q2.

If turned on, dataframe types (e.g. pandas or pyspark and others) will be converted into the new StructuredDataset type and literal.

Please see the PR description for the full details, extension interfaces, and design, as well as the upcoming OSS meeting (Tuesday Feb. 8th) for a deeper discussion. The design doc also is here.

Removal of legacy API

We've removed the old legacy API. This helped remove over 20k lines of code, reduces complexity, and makes the ongoing maintenance load lighter. As part of this cleanup however, some functionality in flyte-cli and (pyflyte) will be going away. Most of this functionality though has been in flytectl for some time. Please let us know if you have any problems.

flyte-cli

  • launch-task
  • execute-launch-plan
  • watch-execution
  • relaunch-execution

These were all commands that relied on the old typing system that was removed.

pyflyte

  • register and fast-register
  • launchplan commands

These were commands that have been deprecated for some time because they communicated with the Flyte control plane (which has been taken over by flytectl and/or relied on the old typing system that was removed.)

Auto-generated Notes

What's Changed

New Contributors

Full Changelog: v0.26.0...v0.30.0

Contributors

eapolinario, cosmicBboy, and 8 other contributors
Assets 2
Loading