Fixing Python linking error #149
Conversation
This commit is an initial draft at fixing #147. The error is due to the fact that pyo3 has linked the libpython from the compilation and not trying to use libpython that is available in `LD_LIBRARY_PATH`. The current only solution to disable the linking is to use the `extension-module` flag. This requires to make the python `runtime-node` packaged in a python library. The python `runtime-node` should also be fully compatible with the other operators in case we want hybrid runtime. The issue that i'm facing is that. Due to the packaging, I have to deal with the `GIL` that is present from the start of `dora-runtime` node. This however makes the process single threaded wich is impossible. So, I have to disable it, but when I do, I have a race condition: ```bash Exception ignored in: <module 'threading' from '/usr/lib/python3.8/threading.py'> Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 1373, in _shutdown assert tlock.locked() AssertionError: ``` The issue is the same as PyO3/pyo3#1274 To fix this issue, I'm going to look again at the different step to make this work. But this is my current investigation.
By making the stopping loop the first loop, we can avoid using `pyo3/allow_threads`, which seems buggy.
haixuanTao
changed the title
There was a problem hiding this comment.
Thanks for the updates!
As I understand it, the runtime now only works with a single Python operator. If a Python operator is combined with other operators, Python or Rust, it might not spawn the additional operators (because it waits until the Python operator finishes first). Is that correct? If so, I think we should forbid spawning Python operators together with other operators for now and throw an error in that case.
Otherwise, this PR seems good to go.
Yes, we could engineer a way to make this resilient but they might be some edge cases and deadlock that could happen so let's forbid spawning python operators with other operators. |
This commit is an initial draft at fixing #147. The error is due to the fact that pyo3 has linked the libpython from the compilation and not trying to use libpython that is available in
LD_LIBRARY_PATH.The current only solution to disable the linking is to use the
pyo3/extension-moduleflag. This requires to make the pythonruntime-nodepackaged in a python library. The pythonruntime-nodeshould also be fully compatible with the other operators in case we want hybrid runtime.The issue that i'm facing is that. Due to the packaging, I have to deal with the
GILthat is present from the start ofdora-runtimenode. This however makes the process single threaded wich is impossible.So, I have to disable it, but when I do, I have a race condition:
The issue is the same as PyO3/pyo3#1274
To fix this issue, I'm going to look again at the different step to make this work.
But this is my current investigation.
Fixes #147