std: /dev/ptmx in MSYS2 is not recognized as a tty in std::io::IsTerminal

[petrochenkov]

How to reproduce:

./x t tests/run-make --test-args emit-to-stdout

That test (tests\run-make\emit-to-stdout\Makefile) checks that rustc's behavior with options like --emit obj is different depending on whether stdout is tty or not.
That test uses /dev/ptmx as stdout and fails when run under MSYS2.

cc @ChrisDenton

[petrochenkov]

we@WIN-GQ6AS7OLS34 MINGW64 ~
$ rustc --emit obj=- hello.rs 1>/dev/ptmx

we@WIN-GQ6AS7OLS34 MINGW64 ~
$ rustc --emit obj=- hello.rs
error: option `-o` or `--emit` is used to write binary output type `obj` to stdout, but stdout is a tty

error: aborting due to 1 previous error

Cannot check the behavior on Linux right now, but if nobody complains about failing run-make tests, then /dev/ptmx is probably considered a tty there.
UPD: Yep, /dev/ptmx is a tty on Linux and rustc refuses to --emit obj into it.

[ChrisDenton]

Does it work with MSYS=enable_pcon set in msys2.ini?

For msys2 without pseudoconsole support there's no way to really know if something is a tty or not so we use heuristics based on the pipe name, these may need updated. If msys2 is using the Windows pseudoconsole then it should just work as it should be detected as a normal Windows console handle (unless /dev/ptmx is special in some other way).

[petrochenkov]

Does it work with MSYS=enable_pcon set in msys2.ini?

No, it doesn't help.

For msys2 without pseudoconsole support there's no way to really know if something is a tty or not so we use heuristics based on the pipe name, these may need updated

I did assume it was some missing heuristic in fn msys_tty_on.

[ChrisDenton]

Ah, actually the problem is this check:

rust/library/std/src/sys/windows/io.rs

Lines 100 to 112 in efb3f11

	// At this point, we *could* have a false negative. We can determine that this is a true
	// negative if we can detect the presence of a console on any of the standard I/O streams. If
	// another stream has a console, then we know we're in a Windows console and can therefore
	// trust the negative.
	for std_handle in [c::STD_INPUT_HANDLE, c::STD_OUTPUT_HANDLE, c::STD_ERROR_HANDLE] {
	let std_handle = c::GetStdHandle(std_handle);
	if !std_handle.is_null()
	&& std_handle != handle
	&& c::GetConsoleMode(std_handle, &mut out) != 0
	{
	return false;
	}
	}

The trouble is even if enable_pcon is set (which it turns out it is by default now) /dev/ptmx is still a pipe instead of a pseduoconsole handle. So unless all std handles are redirected, it will return negative.

[ChrisDenton]

I created a fix in #119664 by simply removing the above code. This decreases the chances of a false negative at the cost of increasing the chances of a false positive. But I think simply inspecting the pipe name is overall the most reliable way to detect an msys2 tty.

[lygstate]

Why /dev/ptmx is a pipe, can that be checked by environment variable?

[ChrisDenton]

You'd have to ask msys2 why that is. Maybe it's a bug or maybe there's some reason for it. I don't know.

can that be checked by environment variable

I don't think so.

[petrochenkov]

You'd have to ask msys2 why that is. Maybe it's a bug or maybe there's some reason for it. I don't know.

@mati865, maybe you know?

[mati865]

Unfortunately I don't know much about Cygwin's internals.

[Biswa96]

Unfortunately I am not familiar with Rust language. I assumed the rust code looks like following C code.

#include <stdio.h>
#include <windows.h>

int main(void)
{
    DWORD mode;
    printf("%d\n", GetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), &mode));
    printf("%d\n", GetConsoleMode(GetStdHandle(STD_OUTPUT_HANDLE), &mode));
    printf("%d\n", GetConsoleMode(GetStdHandle(STD_ERROR_HANDLE), &mode));
}

That program prints 1 (TRUE) for all handles in msys2 environment with conpty enabled.
With MSYS=disable_pcon environment variable, that program prints 0 (FALSE) for all handles.

[ChrisDenton]

@Biswa96 if that program is called isconsole then doing the following:

./isconsole 2>/dev/ptmx

prints:

1
1
0

[Biswa96]

@tyan0 may provide some information about that.

[tyan0]

/dev/ptmx is a device for native msys2 (cygwin) programs. I guess you are using native windows rust compiler. In that case, the program built is also native windows program.

/dev/ptmx is a tty for native msys2 program, however it is just a pipe for native windows program. So, the result is as the author reported.

You can confirm the behaviour with the code:

#include <stdio.h>
#include <unistd.h>
int main() {
    printf("%d\n", isatty(0));
    printf("%d\n", isatty(1));
    printf("%d\n", isatty(2));
    return 0;
}

If you compile this using /usr/bin/gcc and run ./a.exe 2>/dev/ptmx, the result is:

1
1
1

however, with /mingw64/bin/gcc, the result is:

64
64
0

I do not think there is much meaning to check behaviour with /dev/ptmx for native windows program, because it does not provide any meaningful functions for native windows program.

[Biswa96]

I am bit curious why a test for native Windows platform runs a command like program > /dev/ptmx. Should not that test be excluded in Windows?

[ChrisDenton]

Test file is here: https://github.com/rust-lang/rust/blob/master/tests/run-make/emit-to-stdout/Makefile

On Windows our "run-make" tests use msys2 (or git bash, but same difference) so the Makefile is cross-platform, This particular test is testing that emitting binary data to a tty correctly fails. This is definitely something that should be tested on Windows. I'm not totally sure why /dev/ptmx is used for this but I'd assume it's so that the tty is independent of the one we're using for normal output (otherwise it would be annoying if the test fails and binary data is actually written to the tty).

See also #111626 which introduced the test. cc @pjhades

[pjhades]

This particular test is testing that emitting binary data to a tty correctly fails.

Yes, that's the purpose of adding such (failure) tests.

I am bit curious why a test for native Windows platform runs a command like program > /dev/ptmx. Should not that test be excluded in Windows?

I'm not totally sure why /dev/ptmx is used for this but I'd assume it's so that the tty is independent of the one we're using for normal output (otherwise it would be annoying if the test fails and binary data is actually written to the tty).

If I remember correctly, in the test I initially tried

obj: $(OUT)
	$(RUSTC) --emit obj=- $(SRC) 2>$(OUT)/$@ || true
	diff $(OUT)/$@ emit-obj.stderr

but found that the tests would then fail, because stdout was already redirected in the test environment, so the expected error didn't happen. I used /dev/ptmx because that was the one that worked on all the platforms.