Does CreatePseudoConsole accept Socket handle?

[skyline75489]

I brought this up to Dustin yesterday. Feel like should also open a thread here.

Background

We know that on Linux file descriptor is a "universal" thing that can be used both for socket & file. However, on Windows file handle(HANDLE) & socket handle (SOCKET) are two different things. Despite the fact that socket handle from a Winsock provider can be used with other non-Winsock functions, we can't do the other way around, that is passing a file handle into a function that accepts socket handle.

I'm trying to port tmux on Windows which uses libevent a lot. Most of Libevent's APIs utilizes file descriptor, which make it suitable for both file & networking programming on Linux. In tmux, the IO of tty device (/dev/tty) is also managed by libevent using the same FD interface. Underneath the interface, libevent uses recv on Linux (again, works for both file & socket), and uses WSARecv on Windows (which only works for SOCKET handle).

ConPTY Interface

The CreatePseudoConsole interface is defined as this:

HRESULT WINAPI CreatePseudoConsole(
    _In_ COORD size,
    _In_ HANDLE hInput,
    _In_ HANDLE hOutput,
    _In_ DWORD dwFlags,
    _Out_ HPCON* phPC
);

The example in the documentation uses HANDLE created by CreatePipe. The documentation does not mention restriction about passing a SOCKET handle.

Question

So here's the problem. I can not use HANDLE created by CreatePipe because it won't work in libevent. I may be able to use SOCKET instead, which works in libevent. But I don't know if it works in ConPTY.

I've tried to pass TCP/UDP sockets like this:

// conpty_fd is a TCP/UDP socket.
hr = CreatePseudoConsole(coord, conpty_fd, conpty_fd, 0, &hPC);

CreateProcess does work. But the process exits immediately after creation (cmd.exe, powershell.exe).

I've also tried this:

// pipe is created by `CreatePipe`
hr = CreatePseudoConsole(coord, pipe, conpty_fd, 0, &hPC);

CreateProcess still works, and the process survives. But I'm not seeing anything written into the socket.

[zadjii-msft]

Well, this is my API so I should know the answer here 😅 Unfortunately, I don't really know what makes SOCKETs and HANDLEs different on Windows. Guess I maybe should have investigated that more in 2018 😬

@miniksa usually does know more about these sorts of things. Any ideas?

[miniksa]

Nothing in particular is checking the type of the HANDLE that is coming through to the psuedoconsole threads, as far as I'm aware. I believe they're just calling ReadFile and WriteFile on whatever handle they're given. The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement.

You could.... replace your conhost.exe in system32 with a build of openconsole.exe, set up a post-mortem debugger in VS or WinDBG, and use the DebugLaunch flag (see srvinit.cpp ~ line 246) to make it break on startup into your debugger and trace around to see what might be going wrong.... that's what I'd do if I had time to dig at the moment.

[eryksun]

The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement.

Winsock IFS sockets created with socket and accept are "\Device\Afd" files opened in asynchronous mode. See, for example, the documentation of the hFile parameter of ReadFile[Ex] and WriteFile[Ex].

[miniksa]

The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement.

Winsock IFS sockets created with socket and accept are "\Device\Afd" files opened in asynchronous mode. See, for example, the documentation of the hFile parameter of ReadFile[Ex] and WriteFile[Ex].

Welp, there we go then. It just takes doing #262 to be able to deal with an OVERLAPPED and it'd probably work.

[skyline75489]

Thanks guys for the comments. You guys are awesome.

Is it possible for me to workaround this by making the socket blocking?

[eryksun]

Thanks guys for the comments. You guys are awesome.

Is it possible for me to workaround this by making the socket blocking?

I haven't experimented with using sockets with conpty. But you can get a synchronous-mode socket by calling WSASocketW directly, which allows creating a socket without the WSA_FLAG_OVERLAPPED flag. At the system I/O level, this corresponds to opening the file object in FILE_SYNCHRONOUS_IO_NONALERT mode. accept(s, ...) and WSAAccept(s, ...) will return a socket with the same synchronous or asynchronous mode as s.

[skyline75489]

Thanks Eryk. You’re amazing! It works actually. Socket with conpty works as long as the socket has no overlapped attribute. My next question would be is it possible to WSARecv on a file handle? For example, I need to pass the stdhandle into libevent. Inside libevent it will use WSA apis to manipulate the handle. I think it doesn't work as WSA api would complains about the file socket not being a valid socket. I wonder if there’s some kind of bridge that can connect file handle & socket handle. 获取 Outlook for iOS<https://aka.ms/o0ukef>

[miniksa]

My next question would be is it possible to WSARecv on a file handle?

What do you mean, specifically? Are you saying that you're doing GetStdHandle(STD_OUTPUT_HANDLE) and the return type is HANDLE while WSARecv wants a SOCKET but you know that the object is a socket for another reason?

[eryksun]

My next question would be is it possible to WSARecv on a file handle? For example, I need to pass the stdhandle into libevent. Inside libevent it will use WSA apis to manipulate the handle. I think it doesn't work as WSA api would complains about the file socket not being a valid socket. I wonder if there’s some kind of bridge that can connect file handle & socket handle.

Short story: Winsock socket operations, such as recv and select, do not work on a non-socket file. They will fail with the last error set to WSAENOTSOCK.

Very long story: I'll outline the problem and the context, as far as I know, but this doesn't include any simple way to wrap a non-socket file for use with socket functions...

A socket is a handle for a file object opened on the "Afd" device (created by the "Ancillary Function Driver"), which is associated with Winsock protocol and connection state. (The latter is the reason for WSADuplicateSocket. Technically, the socket context is registered with AFD, from which it can be retrieved later. So it's possible to use handle inheritance and DuplicateHandle. The main problem is/was Winsock layered service providers, but those are deprecated.)

A file object is created when a device object is opened. The driver for the device is sent an IRP_MJ_CREATE request that includes any remaining path in the namespace of the device. Exactly how the remaining path should be parsed is up to the device.

You're probably used to accessing files on a volume device (e.g. "C:\Windows\explorer.exe" might be "\Device\HarddiskVolume2\Windows\explorer.exe"). For a volume device that's mounted by a filesystem, the system uses the volume parameter block (VPB) of the device object to refer to the filesystem device that controls access to the volume and its namespace. The filesystem driver typically supports either opening the entire volume or a regular file or directory named by a path in the filesystem (e.g. "\Windows\explorer.exe"). The operations supported on the file object are entirely up to what's directly supported and allowed by the filesystem device.

Any other device type usually manages its own namespace and operations. It can implement a filesystem or a fixed set of virtual files. For example, "\Device\NamedPipe\" (note the trailing backslash) is an NPFS (named-pipe filesystem) directory that contains named pipes, "\Device\Mup" (Multiple UNC Provider) mounts UNC shares in its namespace (with the remaining path managed by a filesystem redirector device, e.g. for SMB shares), and the console's "ConDrv" device supports virtual files such as "\Device\ConDrv\Input". The AFD driver implements an "Endpoint" virtual file to create an enpoint. Winsock calls NtCreateFile on the path "\Device\Afd\Endpoint" in order to create the file object for a socket.

The I/O manager provides a set of common I/O functions for file objects. For example, NtReadFile sends an IRP_MJ_READ request to the driver that's associated with the file object. The base set of services is extended by IOCTL and FSCTL device and filesystem control operations -- some standard and some custom for a particular device or filesystem. These extended requests are respectively sent by calling NtDeviceIoControlFile and NtFsControlFile, which sends an IRP_MJ_DEVICE_CONTROL or IRP_MJ_FILE_SYSTEM_CONTROL request to the driver.

You probably see where this is going. The "receive" operation for a socket WSARecv is the IOCTL code 0x12017, which is supported only by the AFD driver. The IOCTL code is comprised of (FILE_DEVICE_NETWORK << 12 | 5 << 2 | METHOD_NEITHER), where FILE_DEVICE_NETWORK (18) is for any network device type such as an NDIS device (but AFD is actually of type FILE_DEVICE_NAMED_PIPE), the function code for the receive operation is 5, and METHOD_NEITHER (3) tells the I/O manager that it should neither buffer nor lock/map user buffers in system space.

0x12017 is a non-standard code. The standard format for a control code has the 16-bit device type in the upper word, the 2-bit required access left shifted by 14 bits, the 12-bit function code left shifted by 2 bits, and the buffering method in the lower 2 bits. When parsed normally, 0x12017 looks like a code for function 2053 on a beep-type device (FILE_DEVICE_BEEP). That said, since AFD is a private driver for Winsock, no one cares about its weird IOCTL codes.

If a non-socket file handle is passed to a Winsock function such as recv or WSARecv, it won't even get as far as sending the receive device control request. Since Winsock doesn't have any association with the handle, it tries to look up the (presumably) previously stored socket context using the control code 0x12043. This code won't be supported by any driver other than AFD, so the request will fail with STATUS_INVALID_DEVICE_REQUEST, and Winsock will fail the call with the last error set to WSAENOTSOCK.

[skyline75489]

Thanks @eryksun so much for such a detailed writeup! I have no word but to thank you for the effort. And I’ve learned a lot from it.

Unfortunately this “file handle to socket handle” issue will be a huge blocker for tmux to work on Windows. I’ll close this issue now. I think people in the future will have the reference they need if they are tackling similar issues.

[skyline75489]

@miniksa yeah basically I’m trying to do the impossible, to use the StdHandle as if it is on Linux and recv/select on it.