UnicodeDecodeError when running pipenv install on Windows

[pansila]

Issue description

I'm installing a pipenv virtual env from a virtualenv project that has a requirement.txt, but it doesn't matter, I can't run any pipenv commands, even pipenv --support.

Expected result

Installation should be successful.

Actual result

pipenv install or pipenv --support

Traceback (most recent call last):
File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\panzi\AppData\Local\Programs\Python\Python36\Scripts\pipenv.exe_main.py", line 5, in
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv_init.py", line 47, in
from .cli import cli
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli_init_.py", line 3, in
from .command import cli
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli\command.py", line 7, in
import crayons
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\patched\crayons.py", line 49, in
is_powershell = "powershell" in shellingham.detect_shell()[0]
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham_init_.py", line 22, in detect_shell
shell = get_shell(pid, max_depth=max_depth)
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 100, in get_shell
processes = dict(_iter_process())
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 78, in _iter_process
info = {'executable': str(pe.szExeFile.decode('utf-8'))}
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2: invalid continuation byte

Steps to replicate

pipenv install

---

After some debugging, I found it's a decoding bug in Python36\Lib\site-packages\pipenv\vendor\shellingham\nt.py

I added a try catch to capture the exception and try to decode it in my locale encoding, then it succeeded. Is there a more elegant way to fix it?

def _iter_process():
   ...
        try:
            info = {'executable': str(pe.szExeFile.decode('utf-8'))}
        except UnicodeDecodeError:
            info = {'executable': str(pe.szExeFile.decode('GBK'))}

[jxltom]

Thanks for your effort! It may relate with https://github.com/sarugaku/shellingham. @uranusjr do you mind to have a look for this?

[uranusjr]

Similar to #2820, but on Windows. Thanks for the debugging effort! I think I know what is going on here. Will fix in the next release.

[uranusjr]

By the way, @pansila can you provide the value of sys.getfilesystemencoding() of the Python you install Pipenv in?

[pansila]

@uranusjr It's 'utf-8'. And furthermore, sys.getdefaultencoding() also gives 'utf-8'.

The problem is that there are some processes with Chinese names, so when I tried to decode them with 'GBK', pipenv succeeded to run.

I'm using a Windows 10 with English as the default language, but I installed some Chinese softwares. It might be difficult to guess all encodings that processes use in this case.

[uranusjr]

Thanks! I believe my solution is correct, then.

[techalchemy]

It shouldn’t actually be necessary — but going to and from mbcs will be lossy, since utf-16 uses surrogates and if you fail to encode a surrogate using the python mbcs encoder you can only drop the pair.

The correct behavior is to aim for utf-8 where possible but especially on python 3.x where the surrogatepass handler was added explicitly for this purpose. Also, you can use the native Unicode filesystem APIs on windows.

If you want to make this easier I just finished implementing this in full in vistir (not sure if you want to depend on that in shellingham) but the final implementation fixes an edge case that was picked up separately by hypothesis

Source: sarugaku/vistir#54
https://www.python.org/dev/peps/pep-0529/
Also, I wrote the original implementation of the shell handler for windows

[uranusjr]

We can afford to be lossy here since all we want to know is whether an executable is a shell. If the name fails to be decoded, it is definitely not a shell.

I’ve released 1.2.8 for Shellingham. This can be closed after the vendor upgrades for the next release.

[techalchemy]

thats ok with me then, although possibly not actually necessary? do we have a way to test it?

[techalchemy]

@pansila if you add an import:

from ctypes import create_unicode_buffer

and then you make the following changes:

         executable = create_unicode_buffer(pe.szExeFile)

         info = {'executable': executable.value}

I'd be a lot more comfortable letting windows handle this itself than randomly using mbcs for it, since python doesn't actually encode and decode mbcs correctly (which is why they switched to UTF8)

Does this also work? I'd love a test case (i.e. the code points) for the characters that are breaking -- you can get those with ord("character") -- that way we can test for this a bit more thoroughly

[pansila]

@techalchemy It fails with another exception.

pipenv --support

Traceback (most recent call last):
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\panzi\AppData\Local\Programs\Python\Python36\Scripts\pipenv.exe\__main__.py", line 5, in <module>
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\__init__.py", line 47, in <module>
    from .cli import cli
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli\__init__.py", line 3, in <module>
    from .command import cli
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli\command.py", line 7, in <module>
    import crayons
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\patched\crayons.py", line 49, in <module>
    is_powershell = "powershell" in shellingham.detect_shell()[0]
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\__init__.py", line 22, in detect_shell
    shell = get_shell(pid, max_depth=max_depth)
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 106, in get_shell
    processes = dict(_iter_process())
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 79, in _iter_process
    executable = create_unicode_buffer(pe.szExeFile)
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\ctypes\__init__.py", line 291, in create_unicode_buffer
    raise TypeError(init)
TypeError: b'[System Process]'

[callmexss]

thanks a lot, I think you should pull a request to fix this error.

[techalchemy]

This will be caught in vendored dependency updates as Tzu-ping mentioned

[swasher]

Still no success, under Windows7 russian, without russian letters in paths...

> mkdir test && cd test
> pipenv install
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 280: invalid continuation byte
Failed to create virtual environment.

Full output

> pipenv install
Creating a virtualenv for this project…
Pipfile: C:\Users\swasher\PycharmProjects\test\Pipfile
Using c:\users\swasher\appdata\local\programs\python\python37\python.exe (3.7.3) to create virtualenv…
[=== ] Creating virtual environment...Already using interpreter c:\users\swasher\appdata\local\programs\python\python37\python.exe
Using base prefix 'c:\\users\\swasher\\appdata\\local\\programs\\python\\python37'
New python executable in C:\Users\swasher\.virtualenvs\test-tQmITBdN\Scripts\python.exe
Installing setuptools, pip, wheel...
done.

Failed creating virtual environment
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\pipenv\cli\command.py", line 254, in install
[pipenv.exceptions.VirtualenvCreationException]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 1741, in do_install
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 574, in ensure_project
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 506, in ensure_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 935, in do_create_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       extra=[crayons.blue("{0}".format(c.err)),]
[pipenv.exceptions.VirtualenvCreationException]: Traceback (most recent call last):
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 903, in call_subprocess
  line = line.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 280: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\swasher\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
  "__main__", mod_spec)
File "c:\users\swasher\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
  exec(code, run_globals)
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 2610, in <module>
  main()
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 834, in main
  symlink=options.symlink,
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 1129, in create_environment
  install_wheel(to_install, py_executable, search_dirs, download=download)
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 976, in install_wheel
  _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 1066, in _install_wheel_with_search_dir
  call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
File "C:\Users\swasher\AppData\Roaming\Python\Python37\site-packages\virtualenv.py", line 905, in call_subprocess
  line = line.decode(fs_encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 280: invalid continuation byte

Failed to create virtual environment.

> pipenv support 

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'C:\\Users\\swasher\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'

Python location: 'c:\\users\\swasher\\appdata\\local\\programs\\python\\python37\\python.exe'

Python installations found:

• 3.7.3: C:\Users\swasher\AppData\Local\Programs\Python\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '7',
 'platform_system': 'Windows',
 'platform_version': '6.1.7601',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

• ACLOCAL_PATH
• ALLUSERSPROFILE
• ANSICON
• ANSICON_DEF
• APPDATA
• CMDER_ROOT
• COMMONPROGRAMFILES
• COMPUTERNAME
• COMSPEC
• CONFIG_SITE
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• CONEMUANSI
• CONEMUANSILOG
• CONEMUARGS2
• CONEMUARGS
• CONEMUBACKHWND
• CONEMUBASEDIR
• CONEMUBASEDIRSHORT
• CONEMUBUILD
• CONEMUCFGDIR
• CONEMUCONFIG
• CONEMUDIR
• CONEMUDRAWHWND
• CONEMUDRIVE
• CONEMUHWND
• CONEMUHOOKS
• CONEMUISADMIN
• CONEMUPID
• CONEMUPALETTE
• CONEMUSERVERPID
• CONEMUTASK
• CONEMUWORKDIR
• CONEMUWORKDRIVE
• DISPLAY
• EXEPATH
• FP_NO_HOST_CHECK
• HOME
• HOMEDRIVE
• HOMEPATH
• HOSTNAME
• INFOPATH
• LANG
• LOCALAPPDATA
• LOGONSERVER
• MANPATH
• MINGW_CHOST
• MINGW_PACKAGE_PREFIX
• MINGW_PREFIX
• MSYSTEM
• MSYSTEM_CARCH
• MSYSTEM_CHOST
• MSYSTEM_PREFIX
• NUMBER_OF_PROCESSORS
• OLDPWD
• ORIGINAL_PATH
• ORIGINAL_TEMP
• ORIGINAL_TMP
• OS
• PATH
• PATHEXT
• PKG_CONFIG_PATH
• PLINK_PROTOCOL
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMFILES
• PROMPT
• PS1
• PSMODULEPATH
• PUBLIC
• PWD
• PROGRAMDATA
• PROGRAMFILES(X86)
• PROGRAMW6432
• SESSIONNAME
• SHELL
• SHLVL
• SSH_ASKPASS
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TERM
• TMP
• TMPDIR
• USERDOMAIN
• USERNAME
• USERPROFILE
• VAGRANT_HOME
• VBOX_MSI_INSTALL_PATH
• WINDIR
• _
• __COMPAT_LAYER
• WINDOWS_TRACING_FLAGS
• WINDOWS_TRACING_LOGFILE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: C:\Users\swasher\bin;C:\cmder\bin;C:\cmder\vendor\bin;C:\Program Files\Git\bin;C:\cmder\vendor\git-for-windows\mingw64\bin;C:\cmder\vendor\git-for-windows\usr\local\bin;C:\cmder\vendor\git-for-windows\usr\bin;C:\cmder\vendor\git-for-windows\usr\bin;C:\cmder\vendor\git-for-windows\mingw64\bin;C:\cmder\vendor\git-for-windows\usr\bin;C:\Users\swasher\bin;C:\cmder\vendor\conemu-maximus5\ConEmu\Scripts;C:\cmder\vendor\conemu-maximus5;C:\cmder\vendor\conemu-maximus5\ConEmu;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\WindowsPowerShell\v1.0;C:\HashiCorp\Vagrant\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\nodejs;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\swasher\AppData\Local\Programs\Python\Python37\Scripts;C:\Users\swasher\AppData\Local\Programs\Python\Python37;C:\cmder\bin;C:\Program Files\heroku\bin;C:\Users\swasher\AppData\Roaming\npm;C:\Users\swasher\AppData\Local\atom\bin;C:\Users\swasher\AppData\Roaming\Python\Python37\Scripts;C:\Users\swasher\AppData\Local\GitHubDesktop\bin;C:\cmder;C:\cmder\vendor\git-for-windows\usr\bin\vendor_perl;C:\cmder\vendor\git-for-windows\usr\bin\core_perl
• SHELL: C:\cmder\vendor\git-for-windows\usr\bin\bash.exe
• LANG: ru_RU.UTF-8
• PWD: C:/Users/swasher/PycharmProjects/test

---

Contents of Pipfile ('C:\Users\swasher\PycharmProjects\test\Pipfile'):

UPDATE
My issue resolved. The reason is python was blocked by firewall at unusual path -
C:\Users\swasher\.virtualenvs\flask-vue-heroku-o7FqSK_s\Scripts. With new firewall rule is everything okay.

[10-5]

Same error when running pipenv install, on Windows 10 (Chinese). Caused by some network errors in Chinese. Fixed by adding an environment variable: PYTHONLEGACYWINDOWSFSENCODING=mbcs

[helping083]

Fixed this by going to - "c:\Python38-32\lib\site-packages\virtualenv.py" and changing a fs_encoding variable to = 'GBK'

[pansila]

When can we include this fix? There are no releases for almost a year since the last pipenv release. sarugaku/shellingham@3813720

[LaBlazer]

When can we include this fix? There are no releases for almost a year since the last pipenv release. sarugaku/shellingham@3813720

I would also need a real fix for this, not a hack...

[H-Madrid]

thanks, the problem has been solved.

[kopes18]

Every time I got this error reboot system solves, but the error keep repeat after a while.
how did you solve it? @H-Madrid

[kopes18]

error output:
File "c:\users\xxzz\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 78, in _iter_process info = {'executable': str(pe.szExeFile.decode('utf-8'))} UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2: invalid continuation byte
the error is at this line info = {'executable': str(pe.szExeFile.decode('utf-8'))}
I think change 'utf-8' to system encoding will fix the bug?

[geelaro]

Fix this by replacing shellingham/nt.py [L85-L89]
to local C:\Users\LEE\AppData\Local\Programs\Python\Python38\Lib\site-packages\pipenv\vendor\shellingham\nt.py[#L78]

[LKI]

For Windows10 users, possible solution is change system-wide default encoding to utf-8

Hint: this may break some old application which is incompatible with utf-8.

[JalinWang]

This problem has been solved after I upgrade pipenv to the latest version. I think this issue should be closed @uranusjr

[uranusjr]

@JalinWang Thanks for the reminder!

[EqualMa]

This problem has been solved after I upgrade pipenv to the latest version. I think this issue should be closed

@JalinWang Which accurate version is the latest version? I am still facing this issue in 2020.8.13

[JakSamJakSam]

Same error when running pipenv install, on Windows 10 (Chinese). Caused by some network errors in Chinese. Fixed by adding an environment variable: PYTHONLEGACYWINDOWSFSENCODING=mbcs

Thant you!!!

[ekersten]

Is there any solution for this? Still getting the error after setting the environment variable: PYTHONLEGACYWINDOWSFSENCODING=mbcs on version 2021.5.29

[GitRon]

Nothing of the mentioned worked for me... I am using a work-around:

pip install -U pip pipenv pipenv-to-requirements
pipenv run pipenv_to_requirements -f
pip install -r requirements.txt
pip install -r requirements-dev.txt
rm requirements.txt
rm requirements-dev.txt

I put this inside a script so it's convenient to execute. Maybe this helps somebody else...

[thtrummer]

For me this problem was caused by the fake Windows Python App Execution Alias (the one which tries to redirect to the Windows store). It dumps a non UTF-8 error message which causes the problem.