add Python 3.9 support #1253
Comments
hanxiao
added
status/blocked-help-needed
|
On running |
|
This issue is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 4 days |
|
Hi, I am working on a PR for this issue. I have found out the issue why jina fails to work on Python 3.9 jina -vf runs successfully now. jina hello-world fails. |
|
I built protobuf from source for cpp and python to help in resolving the error: |
|
To investigate the error encountered while running jina hello-world, I went through the code flow for the entire error stack trace. Copy pasting it here for ease of access: This error occurs because of getting the value of event variable as None and then trying to access anchor attribute from a NoneType variable. Also, attaching the github links for exact line numbers in jina for easily accessing the codebase for tracing the code flow to understand the error stack: ruamel related github line number links for the final part of the error stack trace: |
|
Hey @gamesterrishi , Thank you very much for your collaboration. We are not doing any special thing in this case, we are trying to load python objects from the file https://github.com/jina-ai/jina/blob/master/jina/resources/logging.default.yml. I am not sure if there is some syntax here that |
|
Thanks @JoanFM for your comment. I had a similar initial assumption as you about thinking that ruamel might not be ready for Python 3.9. I investigated it further and checked the codebase to for relevant package wheels to get more idea about supported Python versions: https://sourceforge.net/projects/ruamel-yaml/ https://www.wheelodex.org/projects/ruamel-yaml/ Project: | ruamel.yaml Requires-Dist: | ruamel.yaml.clib (>=0.1.2); platform_python_implementation == "CPython" and python_version < "3.9" https://www.wheelodex.org/projects/ruamel-yaml-clib/ Project: ruamel.yaml.clib So I tried installing this wheel and executed jina hello-world again. The error that I was encountering got resolved and it ran properly and printed all the logs too. However, the only issue which I now face is that I get the following line in the log: Any help around why the current build could be 0 is highly appreciated. I'll attach log ouput after doing some more investigation. |
Hey @gamesterrishi That is not a problem, EMPTY is the enum name whose value is 0, so no problem about it. I guess some change on how enum values are printed |
|
Hey @gamesterrishi, thanks for solving this. You can look at |
|
Hey @JoanFM and @bhavsarpratik, thanks for your comments. Before I submit my PR, I'd like to share my proposal for solving this issue and confirming if it's the best way for us to proceed towards solving it. In summary, jina is unable to support Python 3.9 currently due to lack of official support for Python 3.9 in two of its dependencies: protobuf and raumel Installing the latest wheel for ruamel.yaml.clib solves the ruamel issue. As per my screenshots above, making use of the following PR allows us to successfully make protobuf work for Python 3.9: Please let me know your thoughts on this, and then I can proceed accordingly. Thanks. |
Hey @gamesterrishi, Thank you very much for the investigation. This is very helpful to understand clearly where we stand with this issue. I think the idea of supporting a different tag for this without official protobuf or ruamel support is a good idea, but it would put too much mantainance cost on our side. I would rather wait for protobuf to have a stable version supporting it. As per I think a good step forward would be to work on the migration in #1471 |
|
Hey @JoanFM , thanks for your sharing your thoughts. I was in fact looking for the differences between ruamel and PyYAML while trying to resolve the ruamel issue, since I have been solely making use of PyYAML for handling any yaml related operations in Python in my work, and was curious to know as to why ruamel was being used in jina instead of PyYAML. The following command was sufficient for me to resolve the ruamel compatibility: This shows that the latest version for ruamel.yaml.clib is officially supported, however the latest ruamel package does not include this latest version as its subdependency. I am not sure if installing a subdependency for a jina dependency separately should be considered as unstable or not. I just wanted to point this out before moving onto another issue. Also, attaching some screenshots too.
|
To be fair, ruamel.yaml supports YAML 1.2 spec whereas PyYAML only supports YAML 1.1. |
|
the PyYAML part is done in #1495 but it seems that Protobuf does not support Python 3.9 yet. I tested locally in docker and it gives Related issues:
@gamesterrishi how did you manage to run hello-world in Python 3.9? |
|
Hey @hanxiao, Thanks for being so quick on the PyYAML migration implementation - it's amazing that now that jina will support PyYAML completely, since its a better and more stable choice in the long run as per the reasons that you have mentioned in:
I came to know about it's differences with ruamel while working on this issue, as I initially felt that if there was no way to get ruamel working with Python 3.9, I would have to try on getting things working by replacing it with PyYAML. https://yaml.readthedocs.io/en/latest/pyyaml.html
I'll share the list of commands with you in accordance with this, which would allow you to test things in a Docker container. I'd be happy if we could then try to incorporate it as a temporary fix for early support of Python 3.9 in jina. |
|
Hey @hanxiao, Please find the list of commands: (Commands to be executed are bold formatted)
https://github.com/protocolbuffers/protobuf/blob/master/src/README.md (C++ Installation Instructions - Unix) To build protobuf from source, the following tools are needed: Get the source by "git clone" of the git repository for the fix. Make sure you have also cloned the submodules and generated the configure script: To build and install the C++ Protocol Buffer runtime and the Protocol Buffer compiler (protoc) execute the following: cd .. Check out contents of some folders. The protobuf that's already installed is present in /usr/local/lib/python3.9/site-packages/google/protobuf/ https://github.com/protocolbuffers/protobuf/tree/master/python (Python Installation Instructions - Unix) The pure python performance is slow. For better performance lets install python c++ implementation. Now, we need to uninstall already existing protobuf installed by pip and copy the contents of our newly compiled protobuf to the target Python directory.
I understand that maintaining releases has to be entirely automated and manageable. But do let me know if we could work out something that's maintainable and works. I believe that including a special tag Docker image in jina Docker Hub registry can work out, or we can add these instructions in the docs for installation for users who want to make use of jina in Python 3.9. Any other suggestions are welcome. |
|
This issue is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 4 days |
Describe the feature
Jina can not run on Python 3.9. We need community support on this to point out the issues, the out-of-date dependencies on Python 3.9.
Your proposal
@alexcg1 as a kick-off, can you add your log here?
Environment
Screenshots
The text was updated successfully, but these errors were encountered: