AWS SSO Sync/Login receiving 429 - TooManyRequestsException

[peteawood]

Describe the bug
After a successful login to my AWS SSO Integration, an HTTP 429 Unknown Code is displayed and nothing displays in the Session list. Initiating a Sync operation also exhibits the same behaviour. This is new behaviour without me upgrading my version of Leapp nor changing my local configuration. Prior to the 24th May Leapp worked as expected.

Leapp Version
0.12.1

To Reproduce
Steps to reproduce the behavior:

1. Correctly configure an AWS Single Sign On Integration
2. Login to the Integration and authenticate/allow Leapp.
3. The error will display
4. Try using the Sync menu option
5. The error will display again

Expected behavior
After a sync or login I expect the relevant roles and accounts I have access to to appear in the session window.

Screenshots

Desktop (please complete the following information):

• OS: Windows
• OS Version 21H2 (19044.1706)
• Leapp Version 0.12.1

Additional context
My first thought was that this is due to an increase in the accounts I have access to and Leapp is not throttling the amount of ListAccountRoles API calls being made. Judging by CloudTrail entries:

Problem occurring (post 24th May)

~227 ListAccountRoles calls made in 3 seconds
~59 429 errors logged (majority in the final second)

Problem not occurring (prior 24th May)

~177 ListAccountRoles calls made in 2 seconds
0 429 errors logged

The amount of roles/accounts I have access to has not increased by 50 (maybe increased 2-5). Looking into a few of the CloudTrail entries I can see that ListAccountRoles is being called more than once for at least 1 account where I only have 1 role.

I was struggling to class this as a bug or a feature request as it's not clear where the fault is but at the moment the SSO Integration (and therefore Leapp) is unusable for me so settled on bug. Relevant log entry attached.

LeappLogExcerpt.txt

[andreacavagna01]

Interesting Bug! thanks for the clarification. It's a limitation added by AWS Single Sign-On in the last week.
We will look at it and I think is something that can be solved by slowing down the request number of API calls to AWS SSO sdk.

Thanks for reporting it

[peteawood]

Thanks @andreacavagna01! I did an extremely basic fix (pausing for 5 seconds every 25 calls) which seemed to resolve it so would agree with the approach 👍 Although maybe something with more sensible figures and that potentially remains responsive rather than my "let it hang until completed" approach! 😄

[andreacavagna01]

Subscribe a PR and we can discuss the better way to do it

[peteawood]

Apologies Andrea, I will do a PR but a combination of Leapp temporarily ceasing to work on my machine entirely and the issue seemingly resolved (or at least happening a lot less) means I haven't had a chance/the urgency to work on it. Just wanted to update so it looks less like an abandoned issue :)

[andreacavagna01]

Is the problem still happening?

[peteawood]

Yes but not to the same degree. I still see Throttling Errors in CloudTrail but they are fewer and the SSO Login eventually succeeds (although it does take a while with no feedback).

Since Leapp is usable again I'm happy if you would prefer to close this bug report and I can open a feature request for some sort of progress bar during SSO Integration logins which should also aim to address the throttling problem?

[jessepiccolo]

This is a issue for me as well. I have 180+ AWS accounts for SSO and only growing.

[jessepiccolo]

@peteawood could you please show how you added the pause? This bug killed the app for me... Just doesn't work with a lot of accounts.

[peteawood]

Sure although a) this is from memory as I binned it once it started working again and b) it's not pretty, it does look like the app has hung you have to wait so long! 😃

I replaced

leapp/packages/core/src/services/aws-sso-integration-service.ts

Line 226 in d5eb452

private async getSessions(integrationId: string, accessToken: string, region: string): Promise<SsoRoleSession[]> {

with

private async getSessions(integrationId: string, accessToken: string, region: string): Promise<SsoRoleSession[]> 
{
    const accounts: AccountInfo[] = await this.listAccounts(accessToken, region);
    const waitFor = delay => new Promise(resolve => setTimeout(resolve, delay));

    const promiseArray: Promise<SsoRoleSession[]>[] = [];
    var counter = 0;
    
    for(const account of accounts)
    {
      counter++;
      if(counter % 25 == 0)
      {
        await waitFor(5000);
      }
      
      promiseArray.push(this.getSessionsFromAccount(integrationId, account, accessToken, region));
    }

    return new Promise((resolve, _) => {
      Promise.all(promiseArray).then((sessionMatrix: SsoRoleSession[][]) => {
        resolve(sessionMatrix.flat());
      });
    });
  }

The 25 limit and 5 second pause were the first values I tried and seemed to work. I would much prefer just handling the throttling errors and some exponential back off/retry along with the UI updating to show progress but at the time I just wanted it working again 😄

[andreacavagna01]

I'll open a Branch linked to this issue in which we can discuss about the best solution

[andreacavagna01]

fixed in the PR and will be released soon

[egauk]

This is an issue for me running 0.13.1, 200+ AWS accounts

[tscully49]

Same here, still problem with 0.13.1 and we are only loading ~75 accounts. It unfortunately is making the app unusable for new users in the company.

[pethron]

Sorry to hear that, we'll check again. I'll reopen the issue to keep track of everything.

[ericvilla]

Hi everyone! We've just released 0.13.2 version of the Desktop App. It solves the issue reported in the current issue and in the #308 one. @egauk, @tscully49, @jessepiccolo, @peteawood I invite you to test it with your AWS SSO (ops... Identity Center) organization. Let me know if it works now. I'll re-open the issue if you still encounter problems. Thank you!

[Kimnor]

So I understand this is closed however I am getting this error on my mac and I am on version 18.3. I cant give an exact date when this started because I have ignored it for so long. Just thought I would report that I still see this error.

[andreacavagna01]

Thanks for reporting it! We have already fixed this problem in version 0.18.5. Download it and let me know if this is fixed