single folder output runs correctly, but --onefile output fails on linux x64

[imavo]

Using pyinstaller 6.8.0 with python 3.8.10 on ubuntu 20.04 x64.

My use case involves python call a 'C' language extension which requires a directory tree (from a 3rd party) to be added via --add-data and includes many .so shared libraries in that directory-tree.

My pyinstaller output on linux runs correctly with the single-folder approach.

But with --onefile it fails (unhandled exception in 3rd party code) and only because of LD_LIBRARY_PATH and PATH problems it seems.

I can circumvent by setting those environment variables externally before running the --onefile output of pyinstaller but I seek to avoid setting such variables externally.

I prefer if such variables get set dynamically while the built onefile runs. I try to do this in my python code, when the script-starts, conditionally changing both LD_LIBRARY_PATH and PATH as needed by searching for the relevant directories inside the /tmp/_MEIxxxx tree. This allows the one-folder approach to work, but is not sufficient for the --onefile output.

questions:

(1) At runtime, does the second-process (and any of its children) inherit on linux x64 the LD_LIBRARY_PATH (and PATH) changes made by the first process?

(2) Is there any method to supply entries to the LD_LIBRARY_PATH (and PATH) that would be passed on to the child-process(es) by the bootloader?

(3) Is the order of the entries on the LD_LIBRARY_PATH significant, in other words, must the /tmp/_MEIxxxx/ directory appear first on the LD_LIBRARY_PATH? ( I prefix two additional directories from /tmp/_MEIxxxx ).

[rokm]

(1) At runtime, does the second-process (and any of its children) inherit on linux x64 the LD_LIBRARY_PATH (and PATH) changes made by the first process?

Yes, the parent modifies LD_LIBRARY_PATH to include top-level application directory (the _MEIxxx directory in onefile build, or the _internal directory in onedir build), and this is inherited by the child process. If that were not the case, the child process could not find any bundled dependencies (since a process cannot change its own library search paths, unlike on Windows).

We generally do not modify PATH on POSIX platforms, though (I say "generally", because this might be done in some run-time hook for a particular package).

(2) Is there any method to supply entries to the LD_LIBRARY_PATH (and PATH) that would be passed on to the child-process(es) by the bootloader?

No.

(3) Is the order of the entries on the LD_LIBRARY_PATH significant, in other words, must the /tmp/_MEIxxxx/ directory appear first on the LD_LIBRARY_PATH? ( I prefix two additional directories from /tmp/_MEIxxxx ).

The order of entries does affect the discovery of shared libraries (in case of copies being available in multiple locations), but for your purposes, it should not matter.

---

Can you provide more specific details about the issue? Because if shared libraries are located in an arbitrary directory tree, then they would not be discoverable via LD_LIBRARY_PATH under unfrozen python, either (beacuse like I said, a process cannot dynamically modify its search paths). So either there are RPATHs at play (which should also work in frozen application, as long as they are relative and as long as relative location relationships are preserved) or there are other tricks involved (like "preloading" the shared libraries via ctypes).

[imavo]

@rokm Thank you for those answers. I will continue investigating. I suspect it is only the LD_LIBRARY_PATH that is the trouble. Logically if manually setting it externally allows it to work, something must be missing from the onefile bundle, but I cannot detect it as yet.

[rokm]

I doubt LD_LIBRARY_PATH is trouble per-se, because it wouldn't work within python, either. That's the reason why extensions shipped with wheels that use bundled .so libraries typically use rpaths.

[rokm]

Is the 3rd party package in question publicly available, or is it private/proprietary?

[imavo]

The 'c' extension is the ibm_db library, which is publicly available (with source) here on github, but that in turn dynamically links to a proprietary closed-source component from IBM called clidriver which by default gets added to the site-packages tree.

The ibm_db extension works correctly with python , at least for my use cases, and it needs the LD_LIBRARY_PATH mods always for my usage .

When I bundle , I specify --add-data /path/to/clidriver:./clidriver . The python script that I bundle searches the bundle temp directory for clidriver and conditionally prefixes 2 of the subdirectories to the LD_LIBRARY_PATH with fully-qualified paths (to mimic what I would ordinarily do if running python with that extension).

I continue to investigate, and I've eliminated the PATH as the cause, only the LD_LIBRARY_PATH seems to matter.

[rokm]

The ibm_db extension works correctly with python , at least for my use cases, and it needs the LD_LIBRARY_PATH mods always for my usage .

Which sub-directories are you adding to LD_LIBRARY_PATH, then? Because extensions and shared libraries that come with linux PyPI wheels for ibm-db seem to be using rpath (for most part?). The binaries shipped with in bin directory do not, though, and might require LD_LIBRARY_PATH if they are used via sub-processes (but in that case, setting LD_LIBRARY_PATH from within the python process should suffice).

[imavo]

Adding clidriver/lib and clidriver/lib/icc (at runtime, relative to the bundled location). But when running not-bundled, relative to the site-packages ( i.e. site-packages/clidriver/lib and site-packages/clidriver/lib.icc ). The diff -r of the directories compare well.

Tracing of clidriver suggests the fault is loading one of the libraries from clidriver/lib/icc (when I've not set LD_LIBRARY_PATH externally before running the bundled binary).

I've also made a mistake previously in this thread, now both the single-folder and --onefile methods both fail in the same manner when LD_LIBRARY_PATH is not externally set, so at least they are consistent.

Since I've made one mistake, I probably made others. I will take a break from this in case of tired eyes causing a snafu and look at it again with fresh eyes.

[imavo]

objdump -x on the libraries supplied with clidriver ( as shipped with ibm_db release 3.2.3 wheel) showed a mix of RPATH settings (mostly $ORIGIN, some $ORIGIN/../../ibm_db.libs/, others with no RPATH, and no RUNPATH usage, and one library with a possibly junk setting for RPATH).

export LD_DEBUG=libs , when run with the working case (python myscript.py) and compared with the failing case (bundled myscript) showed that LD_LIBRARY_PATH as set by the script before import ibm_db was being ignored ( hence my assumption was false).

export LD_DEBUG=all , again comparing working and failing cases, found other missing libraries in the failing case. Ditto for observing strace output for file actions failing.

I removed the code from my testscript for adjusting LD_LIBRARY_PATH or PATH, as it had no effect.

The directory ibm_db.libs gets created and populated by pip install ibm_db and the ibm_db.libs is in the site-packages directory along with clidriver and appears in the bundled root directory automatically presumably after pyinstaller analysis.

I switched from --add-data to --add-binary, and used multiple of these --add-binary options until I had all the required files in the bundle at the ibm_db.libs location and subdirectories.

I will check later if using Tree class is easier to operate with multiple python versions.

The test script now runs successfully both when bundled to a single folder and also when bundled with --onefile. It worked with ubuntu 20.04 and 22.04 and python 3.8.10, 3.9.5, 3.10.12.

I will search for a more elegant method that is easier to operate with multiple python versions, but for now I have a useable workaround.

For now I consider this question answered because no issues were found with pyinstaller for this case.
Thanks for advice.

[rokm]

The most elegant (and canonical) way would be to create a hook for ibm_db that looks up the package path, then checks if ibm_db.libs exists in the same directory, and collect files from it.

Typically, PyInstaller.utils.hooks.collect_dynamic_libs helper would be used for this, but in this case, ibm_db.libs is not a valid package name, so a custom implementation is required (and even if it could be used, the search patterns would need to be overridden, to include versioned .so files).

[imavo]

@rokm will investigate a hook for ibm_db when I get time and report back. As mentioned, clidriver is a moving target. Thanks.

[imavo]

Added pull request 765 on pyinstaller/pyinstaller-hooks-contrib containing an analysis hook and runtime hook for module ibm_db