AWS Session Manager - Slow while write / pasting code

[CD10000]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues, especially the pinned issues.

Exception report

N/A

Screenshot

Hard to demonstrate, since it will look like just slow typing.

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.7
PSEdition                      Core
GitCommitId                    7.3.7
OS                             Linux 6.1.52-71.125.amzn2023.aarch64 #1 SMP Tue Sep 12 21:41:10 UTC 2023
Platform                       Unix


OS: AL2023 arm based.
PSReadLine Version: 2.3.3

Steps to reproduce

I started from a fresh install.

1. Spin up EC2 w/ AL2023 ARM base processor (unsure if its ARM related, doubt it, but throwing it out there)

2. Install powershell as such:

   sudo dnf install -y libicu
sudo curl -L -o /tmp/powershell.tar.gz https://github.com/PowerShell/PowerShell/releases/download/v7.3.7/powershell-7.3.7-linux-arm64.tar.gz
sudo mkdir -p /opt/microsoft/powershell/7
sudo tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7
sudo chmod +x /opt/microsoft/powershell/7/pwsh
sudo ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh

3. Log in to the EC2 via Session Manager

4. Run pwsh

5. Start Typing or paste code

Expected behavior

Not lag behind or take a long time to paste

Actual behavior

Very slow to type. Very slow to paste.

If you uninstall PSReadLine, it all works fine.

[StevenBucher98]

Have you noticed this kind of slow down in any other environment or is it just this specific one? I dont think it is an ARM problem as I seem to have no issues on my ARM based machine.

[CD10000]

I am getting this in most of my environments quite frankly. But they are all mostly running AL2023.

At first I thought I was perhaps contaminating the environments somehow, thats why I created a fresh EC2 w/ AL2023 and installed just pwsh and it does it there too.

[StevenBucher98]

Is it only with Copy/pasting? or do you have general typing lag? Also do you notice it more when you press ctrl+c or right click in the terminal (I am unfamiliar with AL2023 so not sure if this is possible or not 😅 )

[CD10000]

Noticable when typing also. No diff when ctrl+c or right clicking (right click doesn't do anything in the console really)

[StevenBucher98]

We tried reproducing this error but did not have any noticeable lag... Are there any network issues that may be contributing to this issue? Also, if you install 2.2.6 do you notice the lag goes away?

[CD10000]

Same issue w/ 2.2.6.

More in depth steps to reproduce:

1. Launch EC2 (AMI ami-03f6c2c562b3df715 for this sake).
2. Give Role to EC2 that contains SSM Perms (AmazonSSMManagedInstanceCore, I also give AmazonSSMPatchAssociation but I don't think its necessary for Session Manager)
3. Use Session Manager to get Console Access
4. Install pwsh, my steps are:

sudo dnf install -y libicu
sudo curl -L -o /tmp/powershell.tar.gz https://github.com/PowerShell/PowerShell/releases/download/v7.3.7/powershell-7.3.7-linux-arm64.tar.gz
sudo mkdir -p /opt/microsoft/powershell/7
sudo tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7
sudo chmod +x /opt/microsoft/powershell/7/pwsh
sudo ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh

6. Launch pwsh
7. Start Typing

[StevenBucher98]

Hmm we were unable to reproduce this issue still, we tried it with PowerShell 7.3.6. I am pretty doubtful that it has to do with the update from 7.3.6-->7.3.7 but may be worth trying if you are still having lag issues.

[daxian-dbw]

@CD10000 Can you run Set-PSReadLineOption -PredictionSource None after starting pwsh in the above repro, and see if that makes any difference?

[CD10000]

Exact same thing. This is really odd.

[daxian-dbw]

Thanks. That at least rules out the history prediction from the root cause. It's really bizarre.

[CD10000]

I have tried this on a regular AL2023 x64 (not arm) and it gives me the same trouble.

I have noticed this, if I copy a long-ish string, it will paste almost all of it real quick except for the last say 10 chars, which then slows down and goes slow.

Doesn't matter if the string is 100 chars or 1000 chars, only the last little bit will slow down. Multi line is a nightmare.

If I hammer down on my keyboard and write a lot gibberish, it will sorta keep up with me until the last 10-20ish character, a bit like pasting.

---

I just tried on a Windows machine and it works fine!!! I'll ensure everything is up to date on Mac and reply back.

Update: I have updated the mac book pro to the latest and still same problem. Odd.

[CD10000]

RECAP:

Using session manager,

pwsh 7.3.x +
Mac Os: Issue occurs
Windows: Fine

pwsh 7.2.x +
Mac Os: Fine
Windows: Fine

[daxian-dbw]

What do you mean by "using session manager + macOS/windows"? Is it a macOS/Windows VMs running in AWS and you got the console access through the Session Manager? (I have no idea what the Session Manager is TBH)?

Does this happen to you locally on a macOS, Windows, or Linux machine?

[CD10000]

Session manager is a utility provided by AWS to access the terminal from your browser.

When you launch an ec2, you can use Session Manager to access the terminal instead of say... ssh via putty.

And no, There is an ec2 (VM) running AL2023 (which is linux, fedora based).

So I (on a Mac) will access Session Manager via (Chrome, Firefox or Safari) to connect to the EC2 running AL2023.
That gives me the issue.

If I do the same thing e.g.
So I (on a Windows machine) will access Session Manager via (Chrome, Firefox) to connect to the EC2 running AL2023.
No slow down issue here.

---

If I do these steps but running pwsh 7.2.x, I have no issues. If pwsh 7.3.x+ is installed on the ec2, I get the issue when connecting via Session Manager from my Mac.

To re-add info from previous post, if I am running 7.3.x on AL2023 (ec2), and remove the module PSReadLine, I don't have the issue at all.

[daxian-dbw]

Thank you for those additional information, very helpful to me to understand the scenario.

FYI, we don't have direct access to AWS, so @StevenBucher98 emailed a contact we have in AWS and asked help to reproduce the issue. But they cannot reproduce it with your repro steps. Here is the reply we got:

I attempted a quick repro - launched two new Amazon Linux 2023 instances running the ARM architecture. Followed the install details from that issue for both PowerShell 7.3.6 and 7.3.6, then updated to PSReadLine 2.3.3 (using Install-Module -Name PSReadLine -Scope CurrentUser -Force).

Both of those work as I expect, with no noticeable lag (other than standard network lag from connecting to a remote system). I tested connections through Sessions Manager from a CLI, and from the AWS Console using Firefox and Chrome. They work fine whether I'm using the built-in version of PSReadLine, or the latest one from the PowerShell Gallery.

It's very bizarre. Given that you are connecting to the VM console with a browser, it's hard to reason why it makes a difference between doing it from macOS vs. Windows. It feels like the issue lies in Session Manager instead of PowerShell.

If I do these steps but running pwsh 7.2.x, I have no issues. If pwsh 7.3.x+ is installed on the ec2, I get the issue when connecting via Session Manager from my Mac.

Can you please try one more thing: install 7.2.x on the ec2 VM, and then upgrade PSReadLine to v2.3.4 (the latest stable). Then try again from macOS and Windows and see if it still works fine.

[CD10000]

I will give that a shot. Will report back.

Would it be possible to ask your AWS contact to attempt this same thing but try with a Mac ? I didn't know this made a difference at the time.

I have a contact that has a Mac too, will have them try.

[CD10000]

Installed pwsh 7.2.6, installed PSReadLine 2.3.4 and now PSReadLine doesn't seems to be working. Can't tab, don't see any colors.

[daxian-dbw]

Can't tab, don't see any colors.

From both macOS and Windows? Does it work fine when connecting from Windows?

BTW, @StevenBucher98 and I chatted about this offline and he will bring up the new info to our AWS contact and ask to try reproducing it again from macOS.

[CD10000]

Both have no colors, tab, etc.

[CD10000]

I asked someone who has a Mac to remote via Session Manager to the same ec2 instance and have the same issues I have.

To recap:

Mac -> Any Browsers -> Session Manager to ec2 -> pwsh 7.3.x w/ PSReadLine Module loaded = Issue
Mac -> Any Browsers -> Session Manager to ec2 -> pwsh 7.3.x w/ PSReadLine Module unloaded = OK
Mac -> Any Browsers -> Session Manager to ec2 -> pwsh 7.2.x w/ PSReadLine Module loaded = OK
Windows -> Any Browsers -> Session Manager to ec2 -> pwsh 7.3.x w/ PSReadLine Module loaded = OK
Windows -> Any Browsers -> Session Manager to ec2 -> pwsh 7.2.x w/ PSReadLine Module loaded = OK

[daxian-dbw]

@StevenBucher98 Can you please ask your contact at AWS to take another look at this issue?
Given the issue happens when connecting to the VM console with a browser and works fine when the client OS is Windows, the root cause may lie in their session manager.

[StevenBucher98]

Adding response from our contact at AWS

I have tried to reproduce this and I still can't. I've tried using Edge, Chrome and Firefox from my Intel Macbook Pro, and using Amazon Linux 2023 instances running either:

• PowerShell 7.2.15 with PSReadLine 2.3.4
• PowerShell 7.3.8 with PSReadLine 2.2.6
• PowerShell 7.3.8 with PSReadLine 2.3.4

No matter what I do, the experience is smooth and just like you'd expect. I can copy paste large blobs of text, etc.

No issues I can see. I could only guess at reasons, but I do this frequently enough with Session Manager and I've never experienced what is being stated in the issue.
I can't find an issue with a default, newly launched Amazon Linux 2023 EC2 Instance running PowerShell.

I am starting to think it may be something specific to your session manager? Not sure how to resolve from this point

[daxian-dbw]

@CD10000 Now we don't think the root cause for your issue is in the PowerShell or PSReadLine layer.
Please directly contact the owner team of the AWS Session Manager to report this inconsistent behavior you observed between using macOS vs. Windows as the client OS. You will likely get more help from them given that we don't have the environment for investigation.

[CD10000]

For some closure, this issue seems to have been resolved in Session Manager.