.NET Core remote debug via SSH isn't working as advertised

[roblourens]

microsoft/vscode-remote-release#2074 (comment)

from @adaviding

Issue Description

I am following the ssh debug scenario in your documentation.

I cannot remote debug using the SSH technique (hereafter known as "method 1").

Possibly related to #3347.

Steps to Reproduce

I am following the ssh debug scenario in your documentation.

I am having a problem with method 1 (remote debugging via ssh).

I have gotten this working with method 2 (remote debugging via docker exec), so you know most of the pieces setup correctly.

Here is the launch.json content which shows how both methods are configured within my instance of VSCode. (Both methods are configured similarly with some minor differences, as shown in your documentation.)

{
    // Method 1 -- Not working
    // This is not currently working.  We are supposed to be able to connect to a remote debugger using this config via ssh
    // This is taken from https://github.com/OmniSharp/omnisharp-vscode/wiki/Attaching-to-remote-processes#configuring-ssh-attach-with-launchjson
    "name": ".NET Core SSH Attach",
    "type": "coreclr",
    "request": "attach",
    "processId": "${command:pickRemoteProcess}",
    "pipeTransport": {
        "pipeProgram": "ssh",
        "pipeArgs": [ "-T", "-i", "C:/Users/ding/.ssh/id_rsa", "-o", "StrictHostKeyChecking=no", "root@localhost", "-p", "10022"],
        //"pipeArgs": [ "-T", "root@localhost", "-p", "10022"],
        "debuggerPath": "/var/vsdbg/vsdbg",
        "pipeCwd": "${workspaceRoot}",
        "quoteArgs": true
    },
    "sourceFileMap": {
        "/home/ExampleAccount/ExampleProject": "${workspaceRoot}"
    }
},
{
    // Method 2 -- Working
    // This is how we remote debug an application running in a docker container.
    // This is taken from https://github.com/OmniSharp/omnisharp-vscode/wiki/Attaching-to-remote-processes#configuring-docker-attach-with-launchjson
    "name": ".NET Core Docker Attach",
    "type": "coreclr",
    "request": "attach",
    "processId": "${command:pickRemoteProcess}",
    "pipeTransport": {
        "pipeProgram": "docker",
        "pipeArgs": [ "exec", "-i", "docker-images_debug_test_1" ],
        "debuggerPath": "/var/vsdbg/vsdbg",
        "pipeCwd": "${workspaceRoot}",
        "quoteArgs": false
    },
    "sourceFileMap": {
        "/home/ExampleAccount/ExampleProject": "${workspaceRoot}"
    }
}

I have also setup and configured openssh-server (running within the docker container) to the point that you can ssh into the container from the host machine by executing a simple command using either Command Prompt or GitBash on the host machine:

ssh -i C:/Users/ding/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected] -p 10022

Expected Behavior

Debugger can attach via SSH.

Actual Behavior

When I try to connect a remote debugger using method 1 (via ssh), it doesn't actually connect; it just hangs.

Logs

remote-attach log

In the OUTPUT tab (with "remote-attach" selected) I can see a message:

Executing: ssh -T -i C:/Users/ding/.ssh/id_rsa -o StrictHostKeyChecking=no root@localhost -p 10022 "sh -s" < C:\Users\ding\.vscode\extensions\ms-vscode.csharp-1.21.9\scripts\remoteProcessPickerScript

At this point I should see some kind of UI which allows me to pick the PID but it doesn't appear. When I forcibly stop the docker container I immediately see additional text within the OUTPUT tab:

stderr: Connection to localhost closed by remote host.
Error Message: Command failed: ssh -T -i C:/Users/ding/.ssh/id_rsa -o StrictHostKeyChecking=no root@localhost -p 10022 "sh -s" < C:\Users\ding\.vscode\extensions\ms-vscode.csharp-1.21.9\scripts\remoteProcessPickerScript
Connection to localhost closed by remote host.

Environment information

VSCode version: 1.41.1
C# Extension: 1.21.9

Dotnet Information

.NET Core SDK (reflecting any global.json): Version: 3.1.100 Commit: cd82f021f4

Runtime Environment:
OS Name: Windows
OS Version: 10.0.17763
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.100\

Host (useful for support):
Version: 3.1.0
Commit: 65f04fb6db

.NET Core SDKs installed:
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.400 [C:\Program Files\dotnet\sdk]
2.1.401 [C:\Program Files\dotnet\sdk]
2.1.403 [C:\Program Files\dotnet\sdk]
2.1.504 [C:\Program Files\dotnet\sdk]
2.2.107 [C:\Program Files\dotnet\sdk]
2.2.109 [C:\Program Files\dotnet\sdk]
3.1.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension	Author	Version
cpptools	ms-vscode	0.26.2
csharp	ms-vscode	1.21.9
remote-wsl	ms-vscode-remote	0.41.6

[gregg-miskelly]

Can you try manually running your ssh command and pasting in the lines of the script (C:\Users\ding.vscode\extensions\ms-vscode.csharp-1.21.9\scripts\remoteProcessPickerScript) and see if things get stuck at some point?

Note that everyone is probably on vacation, so expect slow replies.

[adaviding]

From GitBash:

$ ssh -i C:/Users/ding/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected] -p 10022 'uname && if [ "$(uname)" = "Linux" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= ; elif [ "$(uname)" = "Darwin" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= -c; fi'
Linux
      aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    1 dotnet                                            dotnet /var/dotnet_proj/Debug/netcoreapp2.2/dotnet_proj.dll
   10 sshd                                              /usr/sbin/sshd
   18 sshd                                              sshd: root [priv]
   20 sshd                                              sshd: root@notty
   21 bash                                              bash -c uname && if [ "$(uname)" = "Linux" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= ; elif [ "$(uname)" = "Darwin" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= -c; fi
   31 ps                                                ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args=

BTW: My docker container is based on fedora 29 (in case it matters).

[gregg-miskelly]

@adaviding does that command complete? (ex: you get your prompt back) or does it hang?

[adaviding]

Yes it does. Let me show you the additional context from my GitBash console (down below). The second prompt (at the bottom) appears immediately.

BTW, there is a Warning the first time the command is executed. (I'm not sure if that would affect anything.) I am showing the warning in the snippet below.

DIng@SEA-DING-03 MINGW64 /c/tmp/my_repo (master)
$ ssh -i C:/Users/ding/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected] -p 10022 'uname && if [ "$(uname)" = "Linux" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= ; elif [ "$(uname)" = "Darwin" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= -c; fi'
Warning: Permanently added '[127.0.0.1]:10022' (RSA) to the list of known hosts.
Linux
      aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    1 dotnet                                            dotnet /var/dotnet_proj/Debug/netcoreapp2.2/dotnet_proj.dll
   11 sshd                                              /usr/sbin/sshd
   23 sshd                                              sshd: root [priv]
   25 sshd                                              sshd: root@notty
   26 bash                                              bash -c uname && if [ "$(uname)" = "Linux" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= ; elif [ "$(uname)" = "Darwin" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= -c; fi
   36 ps                                                ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args=

DIng@SEA-DING-03 MINGW64 /c/tmp/my_repo (master)
$

[rberrelleza]

I'm running into the issue. My configuration works fine on MacOS, but it hangs on Windows. I believe this might have to do with the fact that the extension is using script piping, which AFAIK doesn't work on windows.

This is the command that the extension is running (and hangs)

ssh -T hello-world.okteto "sh -s" < "C:/Users/ramiro/.vscode/extensions/ms-dotnettools.csharp-1.21.18/scripts/remoteProcessPickerScript"

Running SSH and then the content of the script manually works fine.

This is on Windows 10, with OpenSSH's ssh client

[tungphuong]

I am also facing the same issue. Could you please let me know any option to work around?

[adaviding]

My workaround was to use a different IDE (Rider or Visual Studio) because of this one issue. Unfortunate.

[VoX]

@tungphuong

I ran into this as well on Windows 10 using the Microsoft version of the OpenSSH ssh client. I found two workarounds, either install and specify the full path to a different cygwin based build of the OpenSSH ssh client (I tested with the one that comes with the Windows git installer). Or edit the remoteProcessPickerScript file that the extension attempts to pipe to the ssh connection and add a new line with just the exit command and then one final blank line. Ensure the line endings are set to LF.

C:\Users\user\.vscode\extensions\ms-dotnettools.csharp-1.22.1\scripts\remoteProcessPickerScript

uname && if [ "$(uname)" = "Linux" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= ; elif [ "$(uname)" = "Darwin" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= -c; fi
exit

If you go the script editing route you will probably need to re-edit it every time the extension gets updated. I believe the root cause is how cmd.exe treats the end of file when redirecting to stdin.

[tungphuong]

@VoX Many thanks, it works with me on the second.
But for the first option, could you please give me know more detail?

[alanmbarr]

An alternative to Vox's approach is this

uname && if [ "$(uname)" = "Linux" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= ; exit; elif [ "$(uname)" = "Darwin" ] ; then ps -axww -o pid=,comm=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,args= -c; fi

with a exit; added after the first if block which should avoid having to change the file lineendings. i'm exploring if this is safe to do but seemed to work for my windows machine and the script seems to run the same in linux