OSError: [WinError 8] Not enough memory

[marcdexet-cnrs]

Hi, I get this odd error

How to to reproduce

• Use tobix/pywine:3.10 image
• Create a project test_mem
• Add in test_mem the test_mem.py

import click

@click.group()
def cli():
    pass

@cli.command("hello", help="Help message")
@click.option("--name", "-n", type=str)
def hello(name):
    print(f"Hello {name}")

if __name__ == "__main__":
    cli()

• Run docker as docker run --rm -ti -v C:\temp\test_nem:/src tobix/pywine:3.10 bash
• Inside docker :

wine python -m pip install click
wine pyinstaller --onefile test_mem.py

This generates the following fault

  File "C:\Python39\lib\site-packages\PyInstaller\building\api.py", line 784, in assemble
    os.rename(build_name, self.name)
OSError: [WinError 8] Not enough memory: 'Z:\\src\\dist\\test_mem.exe.notanexecutable' -> 'Z:\\src\\dist\\test_mem.exe'

complete trace is below

• Computer : Windows 10 Professionnel 21H1
• 16 Go of memory

When I run exactly the same command on my computer (without docker image), it runs perfectly.

Thank you.

Complete trace

537 INFO: PyInstaller: 4.10
538 INFO: Python: 3.10.4
622 INFO: Platform: Windows-10-10.0.18362
631 INFO: wrote Z:\src\test_mem.spec
746 INFO: UPX is available.
782 INFO: Extending PYTHONPATH with paths
['Z:\\src']
2288 INFO: checking Analysis
2323 INFO: Building because hookspath changed
2324 INFO: Initializing module dependency graph...
2403 INFO: Caching module graph hooks...
2508 INFO: Analyzing base_library.zip ...
11641 INFO: Processing pre-find module path hook distutils from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
11652 INFO: distutils: retargeting to non-venv dir 'C:\\Python39\\lib'
14324 INFO: Caching module dependency graph...
15152 INFO: running Analysis Analysis-00.toc
15170 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by C:\Python39\python.exe
15199 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-runtime-l1-1-0.dll
15211 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-heap-l1-1-0.dll
15240 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-math-l1-1-0.dll
15251 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-stdio-l1-1-0.dll
15261 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-locale-l1-1-0.dll
15292 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-time-l1-1-0.dll
15304 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-convert-l1-1-0.dll
15319 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-environment-l1-1-0.dll
15332 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-string-l1-1-0.dll
15400 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-filesystem-l1-1-0.dll
15440 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-process-l1-1-0.dll
15467 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-core-path-l1-1-0.dll
15487 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-conio-l1-1-0.dll
15568 INFO: Analyzing Z:\src\test_mem.py
15588 INFO: Processing module hooks...
15589 INFO: Loading module hook 'hook-difflib.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
15595 INFO: Loading module hook 'hook-distutils.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
15599 INFO: Loading module hook 'hook-distutils.util.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
15604 INFO: Loading module hook 'hook-encodings.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16215 INFO: Loading module hook 'hook-heapq.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16222 INFO: Loading module hook 'hook-lib2to3.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16829 INFO: Loading module hook 'hook-multiprocessing.util.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16834 INFO: Loading module hook 'hook-pickle.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16839 INFO: Loading module hook 'hook-sysconfig.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16843 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16846 INFO: Loading module hook 'hook-xml.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
16981 INFO: Loading module hook 'hook-_tkinter.py' from 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks'...
18138 INFO: checking Tree
18172 INFO: Building because root changed
18173 INFO: Building Tree Tree-00.toc
18924 INFO: checking Tree
18949 INFO: Building because root changed
18949 INFO: Building Tree Tree-01.toc
19545 INFO: checking Tree
19570 INFO: Building because root changed
19571 INFO: Building Tree Tree-02.toc
19620 INFO: Looking for ctypes DLLs
19644 INFO: Analyzing run-time hooks ...
19650 INFO: Including run-time hook 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_inspect.py'
19656 INFO: Including run-time hook 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_subprocess.py'
19661 INFO: Including run-time hook 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgutil.py'
19677 INFO: Including run-time hook 'C:\\Python39\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_multiprocessing.py'
19692 INFO: Looking for dynamic libraries
20581 WARNING: Excluding Wine built-in DLL: C:\windows\system32\api-ms-win-crt-utility-l1-1-0.dll
21368 INFO: Looking for eggs
21370 INFO: Using Python library C:\Python39\python310.dll
21371 INFO: Found binding redirects:
[]
21409 INFO: Warnings written to Z:\src\build\test_mem\warn-test_mem.txt
21560 INFO: Graph cross-reference written to Z:\src\build\test_mem\xref-test_mem.html
21620 INFO: checking PYZ
21640 INFO: Building because toc changed
21640 INFO: Building PYZ (ZlibArchive) Z:\src\build\test_mem\PYZ-00.pyz
22445 INFO: Building PYZ (ZlibArchive) Z:\src\build\test_mem\PYZ-00.pyz completed successfully.
22502 INFO: checking PKG
22527 INFO: Building because toc changed
22527 INFO: Building PKG (CArchive) test_mem.pkg
22584 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\python310.dll
24430 INFO: Disabling UPX for C:\Python39\VCRUNTIME140.dll due to CFG!
24462 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_multiprocessing.pyd
24607 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_decimal.pyd
24853 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_socket.pyd
25085 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\select.pyd
25230 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_overlapped.pyd
25407 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_ssl.pyd
25600 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_asyncio.pyd
25758 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_queue.pyd
25907 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\pyexpat.pyd
26125 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\unicodedata.pyd
26635 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_hashlib.pyd
26815 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_lzma.pyd
26996 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_bz2.pyd
27174 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\_ctypes.pyd
27365 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\libcrypto-1_1.dll
28631 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\libssl-1_1.dll
29055 INFO: Executing - upx --lzma -q C:\users\root\AppData\Local\pyinstaller\bincache01_py310_64bit\libffi-7.dll
29857 INFO: Building PKG (CArchive) test_mem.pkg completed successfully.
29874 INFO: Bootloader C:\Python39\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
29874 INFO: checking EXE
29882 INFO: Building EXE because EXE-00.toc is non existent
29882 INFO: Building EXE from EXE-00.toc
29907 INFO: Copying bootloader EXE to Z:\src\dist\test_mem.exe.notanexecutable
30029 INFO: Copying icon to EXE
30029 INFO: Copying icons from ['C:\\Python39\\lib\\site-packages\\PyInstaller\\bootloader\\images\\icon-console.ico']
30108 INFO: Writing RT_GROUP_ICON 0 resource with 104 bytes
30109 INFO: Writing RT_ICON 1 resource with 3752 bytes
30109 INFO: Writing RT_ICON 2 resource with 2216 bytes
30110 INFO: Writing RT_ICON 3 resource with 1384 bytes
30110 INFO: Writing RT_ICON 4 resource with 37019 bytes
30111 INFO: Writing RT_ICON 5 resource with 9640 bytes
30111 INFO: Writing RT_ICON 6 resource with 4264 bytes
30112 INFO: Writing RT_ICON 7 resource with 1128 bytes
30142 INFO: Copying 0 resources to EXE
30143 INFO: Emedding manifest in EXE
30145 INFO: Updating manifest in Z:\src\dist\test_mem.exe.notanexecutable
30237 INFO: Updating resource type 24 name 1 language 0
30268 INFO: Appending PKG archive to EXE
Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python39\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "C:\Python39\lib\site-packages\PyInstaller\__main__.py", line 124, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "C:\Python39\lib\site-packages\PyInstaller\__main__.py", line 58, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "C:\Python39\lib\site-packages\PyInstaller\building\build_main.py", line 803, in main
    build(specfile, distpath, workpath, clean_build)
  File "C:\Python39\lib\site-packages\PyInstaller\building\build_main.py", line 725, in build
    exec(code, spec_namespace)
  File "Z:\src\test_mem.spec", line 23, in <module>
    exe = EXE(pyz,
  File "C:\Python39\lib\site-packages\PyInstaller\building\api.py", line 510, in __init__
    self.__postinit__()
  File "C:\Python39\lib\site-packages\PyInstaller\building\datastruct.py", line 155, in __postinit__
    self.assemble()
  File "C:\Python39\lib\site-packages\PyInstaller\building\api.py", line 784, in assemble
    os.rename(build_name, self.name)
OSError: [WinError 8] Not enough memory: 'Z:\\src\\dist\\test_mem.exe.notanexecutable' -> 'Z:\\src\\dist\\test_mem.exe'

[TobiX]

I cannot reproduce the problem. How much memory has your Docker installation? If you are using Docker Desktop with WSL2, maybe the docker VM hasn't allocated enough memory?

[blueshack112]

@TobiX facing the same issue recently with pyinstaller. Fresh docker install.

How could I check what memory is it allocated?

This is the output of docker system df command

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE

Images          13        7         6.071GB   5.163GB (85%)
Containers      10        7         29.51MB   26.27MB (89%)
Local Volumes   12        4         165.5MB   41.39MB (25%)
Build Cache     60        0         82.73MB   82.73MB

[blueshack112]

@TobiX just had an observation: This error only emerges when the distpath is set to a mounted volume. In my case a windows directory is mounted to /var/www/data/. If I move the file /tmp/test/code.py and then run pyinstaller, it works perfectly

[mundodisco8]

I have exactly the same problem. Pulled the image, updated everything to latest and wasn't able to build. I added my code to the docker instance via a volume, from my Windows machine. Copied it somewhere else (/temp) and then it completed successfully.

[lsangild]

This seems to have something to do with the Windows mapping of data into WSL.
If the mounted folder is located on my Windows partition and mapped into a docker running under WSL, I get the mentioned error.
If I copy the folder into the WSL instance, and then mounts it into the docker container, there is no issues.

[blueshack112]

@lsangild I'm not very well-versed with docker's internal architecture, but that does seem to be the issue. The read/write permissions are also not a problem because I can copy the .exe file from /tmp/pyinstaller-temp to my mounted folder after the pyinstaller command without any problems.

For now, I've simply updated my command to do:

1. Copy the target file to /temp/pyinstaller-temp
2. Run Pyinstaller on the temp file
3. Copy the .exe back to the mounted directory (/var/www/data in my case)

[TobiX]

Do I understand that correctly:

• You are running Windows Python
• On Wine
• On Linux
• In a VM running on Windows

While passing through a Windows directory all the way?

• On NTFS
• Exposed to Linux via 9p (WSL2) or DrvFS (WSL1)
• Which wine assumes is a POSIX-complient file system
• And Wine maps back to NTFS semantics

I'm surprised this only breaks in such a corner case. I'd suggest reporting this problem to Microsoft, since it seems like DrvFs isn't POSIX compliant enough to satisfy Wine's requirements for a filesystem...

[lsangild]

Yes, that is correctly understood. I am not sure if WSL2 is actually a VM or how it should be described.

The use case is that I need to build a Windows executable for customers. Normally this is done on our Linux build server, skipping the WSL step, but for consistency I tried to build it on my Windows PC through WSL with Docker and encountered the issue.

[TobiX]

Yes, WSL2 is a Linux VM running in Hyper-V and all Windows directories are mounted into the VM using the 9p protocol, which isn't very performant and apparently this whole layercake of filesystems (docker/podman puts an overlayfs over all this stuff) is slightly broken in the end. I see nothing I would be able to fix here, sorry.

[mundodisco8]

I understand that you can't do anything as they issue is not in your code, but could you try to capture it somehow and print a different error? When I got the "out of memory" error I spent quite a lot of time trying to figure out how that was even possible. It wasn't until I found this issue that I tried to copy the project to a different path and it worked.

Also, if that matters, I didn't run my docker from WSL, but from Windows, with the project path passed to the docker container as a volume. Why running docker when I could have compiled in Windows in the first place? I did this because I wanted to test the docker image before converting it to a pipeline in my CICD system.

[TobiX]

I understand that you can't do anything as they issue is not in your code, but could you try to capture it somehow and print a different error?

I cannot, that's some strange error code generated by Wine(?), which Python just reports. I could probably put a fat WARNING into the default entrypoint of my images if any 9p mount is detected... Thinking about it...

Also, if that matters, I didn't run my docker from WSL, but from Windows

Yeah, that uses WSL2 behind the scenes, so it's the same problem, but a different way to get to it.

[TobiX]

For reference: This does not happen with other 9p servers, so the culprit is pretty much Microsoft's implementation of the protocol.

[mundodisco8]

Understood, thanks for the explanation!

…

On Fri, 26 May 2023, 23:57 Tobias Gruetzmacher, ***@***.***> wrote: For reference: This does not happen with other 9p servers, so the culprit is pretty much Microsoft's implementation of the protocol. — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACADOE3T24YUCD6QBMBC2EDXIEYONANCNFSM5T6QJBTA> . You are receiving this because you commented.Message ID: ***@***.***>

[TobiX]

Here is the smoking gun (this is from tracing wine with strace):

The left is on WSL2 9p, the right on ext4. It seems there is a race condition after a rename, where the target file doesn't exist yet, but Wine(?) tries to query attributes, but this fails.

For (my) reference:

• os.rename: https://github.com/python/cpython/blob/34e93d3998bab8acd651c50724eb1977f4860a08/Modules/posixmodule.c#L5491
• MoveFileExW: https://source.winehq.org/git/wine.git/blob/HEAD:/dlls/kernelbase/file.c#l2491
• https://source.winehq.org/git/wine.git/blob/HEAD:/dlls/ntdll/unix/file.c#l4766
• https://source.winehq.org/git/wine.git/blob/HEAD:/server/fd.c#l2655 (there's the STATUS_NO_MEMORY)

[TobiX]

I've prepared a warning message in #43 - If anyone wants to test it before I merge it, please report back here. Otherwise I will merge it the near future 😄