This repository has been archived by the owner on Oct 9, 2023. It is now read-only.
Cache Serialize API #326
Merged
Cache Serialize API #326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
…th name change Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
hamersaw
force-pushed
the
feature/cache-reservation-api
branch
from
b62adfe to
0d288d5
Compare
Signed-off-by: Daniel Rammer <[email protected]>
kumare3
reviewed
Nov 19, 2021
kumare3
reviewed
Nov 19, 2021
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
EngHabu
previously approved these changes
Nov 30, 2021
go.mod
Outdated
| @@ -33,3 +33,5 @@ require ( | |||
| ) | |||
|
|
|||
| replace github.com/aws/amazon-sagemaker-operator-for-k8s => github.com/aws/amazon-sagemaker-operator-for-k8s v1.0.1-0.20210303003444-0fb33b1fd49d | |||
|
|
|||
| replace github.com/flyteorg/flyteplugins => github.com/hamersaw/flyteplugins v0.6.2-0.20211129132447-34fdd4b8118e | |||
Signed-off-by: Daniel Rammer <[email protected]>
EngHabu
approved these changes
Dec 1, 2021
eapolinario
pushed a commit
to eapolinario/flytepropeller
that referenced
this pull request
Aug 9, 2023
* added cache reservation logic to task execution Signed-off-by: Daniel Rammer <[email protected]> * setting reservation heartbeat to workflow reeval duration Signed-off-by: Daniel Rammer <[email protected]> * releasing reservation on completion Signed-off-by: Daniel Rammer <[email protected]> * fixed lint errors Signed-off-by: Daniel Rammer <[email protected]> * updated task metadata field to DiscoverySerializable in accordance with name change Signed-off-by: Daniel Rammer <[email protected]> * not allowing cache serialization without specifying cachable as well Signed-off-by: Daniel Rammer <[email protected]> * added datacatalog unit tests Signed-off-by: Daniel Rammer <[email protected]> * added handler unit tests Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * updated flyteidl and flyteplugins versions - change before merging Signed-off-by: Daniel Rammer <[email protected]> * updated flyteidl and flyteplugin version to reflect latest changes Signed-off-by: Daniel Rammer <[email protected]> * remove flyteidl replace in go.mod and updating to latest version Signed-off-by: Daniel Rammer <[email protected]> * moving release reservation to node finalize function Signed-off-by: Daniel Rammer <[email protected]> * updated go.mod and added missing docs on public types Signed-off-by: Daniel Rammer <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Implemented cache reservations reducing the amount of duplicate work for long running cachable tasks. If enabled, a task acquires a reservation which is released upon completed (after the results are written to the cache). All other identical instances (ie. same task and inputs) wait for the reservation to expire or the cached results to become available.
Type
Are all requirements met?
Complete description
RFC - flyteorg/flyte#1461
Tracking Issue
fixes flyteorg/flyte#267
fixes flyteorg/flyte#420
fixes flyteorg/flyte#872
Follow-up issue
NA