Re-enable ANSI console color for Windows 10.16257 and later

[glenn-slayden]

In Windows 10 build 16257 (released last year) and later, ANSI color and escape sequences for the built-in cmd.exe console ("conhost") are subject to the following registry key:

[HKEY_CURRENT_USER\Console]
"VirtualTerminalLevel"=dword:00000001

See MSDN blogs for more information. There's also some official MSDN docs and a bunch of articles on StackOverflow, such as here and here. There's also some critical information on this subject in this github comment.

This new capability means that the following lines of code in youtube-dl are overly-restrictive:

https://github.com/rg3/youtube-dl/blob/f9f10268c1816dcf1fc0db9d8128008f73a154c7/youtube_dl/extractor/common.py#L782

https://github.com/rg3/youtube-dl/blob/f9f10268c1816dcf1fc0db9d8128008f73a154c7/youtube_dl/YoutubeDL.py#L596

https://github.com/rg3/youtube-dl/blob/f9f10268c1816dcf1fc0db9d8128008f73a154c7/youtube_dl/YoutubeDL.py#L608

On Windows 10, the value of compat_os_name is "nt". Because the lines listed above subsume all three cases where youtube-dl references the --no-color command-line option, the end result is that the --no-color option always asserted on Windows (that is, regardless of not specifying --no-color).

Relaxing the three conjunctions shown above, by removing the and compat_os_name != 'nt' term from each, allows for ANSI color on Windows 10 (assuming the system-wide default is 'enabled' as specified by the registry key mentioned above).

[glenn-slayden]

[HKEY_CURRENT_USER\Console]
Registry setting to enable ANSI sequence processing on Windows 10.16257 and later:

[Hrxn]

Do you perhaps know if such a setting also exists for PowerShell?

[glenn-slayden]

I don't, but it's possible it might just work now. As I noted, the 3 instances of ... and compat_os_name != 'nt' ... are blocking ANSI color on any Windows machine. As a test, you can just remove those 3 and give it a try. Doing just that (alone with VirtualTerminalLevel enabled in the registry) enables color for me on Windows 10.0.16299.125.

Glenn

[glenn-slayden]

...since I have those changes here locally I just gave it a try and yes, it does work for PowerShell too.

(And hmm, why am I never using PowerShell in general?...)

[glenn-slayden]

On further consideration, and even better idea would be to have youtube-dl explicitly enable/disable ANSI color on Windows using the Win32 API SetConsoleMode. This would restore the ability for the user to control of the feature on Windows—via the existing --no-color command line option in youtube-dl—and without requiring the user to fiddle with any registry settings.

I implemented this change in a updated version of the youtube-dl function _windows_write_string from the file .../youtube-dl/utils.py. The following is a complete replacement for that function.

Beyond cleanup/modernization overhaul, the substantive change in this code is that ANSI VT mode will be enabled, if necessary, for the stdout and stderr handles (the state persists for each, so the calls to SetConsoleMode will generally only happen once, upon first use).

def _windows_write_string(s, out):
    """ Returns True if the string was written using special methods,
    False if it has yet to be written out."""
    # Adapted from http://stackoverflow.com/a/3259271/35070

    from ctypes import byref, POINTER, windll, WINFUNCTYPE
    from ctypes.wintypes import BOOL, DWORD, HANDLE, LPWSTR, LPVOID

    FILE_TYPE_CHAR = 0x0002
    FILE_TYPE_REMOTE = 0x8000
    ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004

    GetStdHandle = compat_ctypes_WINFUNCTYPE(
        HANDLE,
        DWORD)(('GetStdHandle', windll.kernel32))

    GetFileType = compat_ctypes_WINFUNCTYPE(
        DWORD,
        HANDLE)(('GetFileType', windll.kernel32))

    GetConsoleMode = compat_ctypes_WINFUNCTYPE(
        BOOL,
        HANDLE,
        POINTER(DWORD))(('GetConsoleMode', windll.kernel32))

    SetConsoleMode = compat_ctypes_WINFUNCTYPE(
        BOOL,
        HANDLE,
        DWORD)(('SetConsoleMode', windll.kernel32))

    WriteConsoleW = compat_ctypes_WINFUNCTYPE(
        BOOL,
        HANDLE,
        LPWSTR,
        DWORD,
        POINTER(DWORD),
        LPVOID)(('WriteConsoleW', windll.kernel32))

    try:
        fileno = out.fileno()
    except AttributeError:
        # If the output stream doesn't have a fileno, it's virtual
        return False
    except io.UnsupportedOperation:
        # Some strange Windows pseudo files?
        return False

    if fileno == 1:
        h = GetStdHandle(-11)
    elif fileno == 2:
        h = GetStdHandle(-12)
    else:
        return False

    if h is None or h == HANDLE(-1):
        return False

    if (GetFileType(h) & ~FILE_TYPE_REMOTE) != FILE_TYPE_CHAR:
        return False

    mode = DWORD()
    if not GetConsoleMode(h, byref(mode)):
        return False

    if (mode.value & ENABLE_VIRTUAL_TERMINAL_PROCESSING) == 0:
        SetConsoleMode(h, mode.value | ENABLE_VIRTUAL_TERMINAL_PROCESSING)

    def next_nonbmp_pos(s):
        try:
            return next(i for i, c in enumerate(s) if ord(c) > 0xffff)
        except StopIteration:
            return len(s)

    written = DWORD(0)
    while s:
        count = min(next_nonbmp_pos(s), 1024)

        ret = WriteConsoleW(
            h, s, count if count else 2, byref(written), None)
        if ret == 0:
            raise OSError('Failed to write string')
        if not count:  # We just wrote a non-BMP character
            assert written.value == 2
            s = s[1:]
        else:
            assert written.value > 0
            s = s[written.value:]
    return True

If someone wants to merge in this code, the following lines should also be changed to remove the and compat_os_name != 'nt' exclusion from each.

https://github.com/rg3/youtube-dl/blob/f9f10268c1816dcf1fc0db9d8128008f73a154c7/youtube_dl/extractor/common.py#L782

https://github.com/rg3/youtube-dl/blob/f9f10268c1816dcf1fc0db9d8128008f73a154c7/youtube_dl/YoutubeDL.py#L596

https://github.com/rg3/youtube-dl/blob/f9f10268c1816dcf1fc0db9d8128008f73a154c7/youtube_dl/YoutubeDL.py#L608

Many thanks if someone is willing to help pull this code, since I am not yet fully adept at GitHub.

[glenn-slayden]

ping. Does anybody want to help push the changes I included above? It has been working super-great for me locally.