Use UTF-8 encoding for log file.

[methane]

Fixes #10071.

[uranusjr]

Is there a way to only do this to new log files? If there is an existing file written not using UTF-8, this will I think pip will crash with this patch since it won’t be able to open the existing file for appending.

[DiddiLeija]

Is there a way to only do this to new log files? If there is an existing file written not using UTF-8, this will I think pip will crash with this patch since it won’t be able to open the existing file for appending.

Agree. There are many ways to make this crash. But if you just write the new files with UTF-8, but giving the option to dismiss that and read another kind of encoding, that's fine for me.

Example: On logging.py, if you use a try/except to attempt the UTF-8 encoding, but giving the option to ignore (and use a warning to tell the user, if you want to) could help.

[methane]

Is there a way to only do this to new log files?

It is very difficult. We need to check the encoding of existing file.
But note that the default log file is /dev/null. It is very rare.

If there is an existing file written not using UTF-8, this will I think pip will crash with this patch since it won’t be able to open the existing file for appending.

pip won't crash.

>>> with open("x.txt", "w", encoding="cp932") as f:
...     f.write("こんにちは\n")
...
6
>>> with open("x.txt", "a", encoding="utf-8") as f:
...     f.write("こんにちは\n")
...
6
>>> with open("x.txt", "rb") as f:
...     print(f.read())
...
b'\x82\xb1\x82\xf1\x82\xc9\x82\xbf\x82\xcd\n\xe3\x81\x93\xe3\x82\x93\xe3\x81\xab\xe3\x81\xa1\xe3\x81\xaf\n'
>>>

[uranusjr]

pip won't crash.

Interesting! I did not know this. So encoding errors only happen lazily when something actually needs to be encoded/decoded (i.e. on read()). TIL.

[DiddiLeija]

Interesting! I did not know this.

Me neither. But after you explain it, I really like the idea.

[pradyunsg]

Thanks @methane! ^>^