-
Notifications
You must be signed in to change notification settings - Fork 674
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Revert "Revert "Ensure token is refreshed on Unauthenticated (#239)" … (
#294) …(#292)" This reverts commit ad19bf1. ## Overview Fixes the racy behavior in auth token cache using mutexes and condition variables. In a nutshell, the change does the following. * Allows all parallel token requesters to try to acquire a lock * One of the requester gets the lock and rest get waitlisted and wait for notification from the requester who acquired the lock * The requester who acquired the lock refreshes the token and save to the token cache * He also notifies the waitlisted requesters to now go and read the saved token cache and proceed with the request The test plan below summarizes how this was tested ## Test Plan https://buildkite.com/unionai/org-staging-sync/builds/2311 https://buildkite.com/unionai/managed-cluster-staging-sync/builds/3461 Serverless onboarding workshop https://buildkite.com/unionai/serverless-workshop-staging-deploy-and-run/builds/44#018fa797-269b-4c08-9809-f6561f8af777 This was tested locally aswell using integration test and was ran using 20 parallel routines ``` go test -v -tags=integration -run '^TestAPISuite$/^Test_RandomNameWorkflow$' ``` * All routines get unauthenticated in the first attempt of using inmemory cached token provider ``` cat t |grep "If it's an unauthenticated error, we will attempt to establish an authenticated context" |wc -l 20 ``` * All routines try to lock ``` cat t |grep "try lock" |wc -l python 20 ``` * One of the routine is able to acquire the lock ``` cat t |grep "Locked : true" |wc -l python 1 ``` * Rest fail to lock ``` cat t |grep "Locked : false" |wc -l python 19 ``` * The ones who fail to lock go into waiting state . i.e they get added to condition variables waitlist ``` cat t |grep "Waiting" |grep -v "Coming" |wc -l python 19 ``` eg : log for ref ``` {"json":{"src":"token_cache_inmemory.go:77"},"level":"info","msg":"Coming out of Waiting","ts":"2024-05-22T23:52:47-07:00"} ``` * The one who has locked goes and saves the token ``` cat t |grep "Saved" |wc -l python 1 ``` * Broadcasts to all the others waiting using single notification which notifies all the waiters. ``` cat t |grep "Broadcasted" |wc -l python 1 ``` * Post which they come out of waiting ``` cat t |grep "Coming out of Waiting"|wc -l python 19 ``` And go ahead and read the token Log file for reference [t.txt](https://github.com/unionai/flyte/files/15413364/t.txt) ## Rollout Plan (if applicable) Merge this change and merge the cloud PR ## Upstream Changes Should this change be upstreamed to OSS (flyteorg/flyte)? If so, please check this box for auditing. Note, this is the responsibility of each developer. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F). - [X] To be upstreamed ## Issue *TODO: Link Linear issue(s) using [magic words](https://linear.app/docs/github#magic-words). `fixes` will move to merged status, while `ref` will only link the PR.* ## Checklist * [ ] Added tests * [ ] Ran a deploy dry run and shared the terraform plan * [ ] Added logging and metrics * [ ] Updated [dashboards](https://unionai.grafana.net/dashboards) and [alerts](https://unionai.grafana.net/alerting/list) * [ ] Updated documentation
- Loading branch information
1 parent
092dfe8
commit 99ebd79
Showing
38 changed files
with
430 additions
and
142 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.