Jep does not pip install with numpy support for Ubuntu 24.04, Python 3.12, pip 24.0

[svensk-pojke]

When pip-installing numpy followed by jep on a new virtual env, the installed version of jep does not support numpy (jep.JEP_NUMPY_ENABLED == 0). I suspect this is somehow related to pyproject.toml vs setup.py, which seems to be determined by the version of pip being used, but have not been able to prove it.

I discovered this by upgrading an existing system using Mint 21.3 (Ubuntu 22.04), Jep 4.2.0 and Python 3.10, which supports numpy just fine. Switching between python 3.10 and 3.12 verified the problem. As a result of jep.JEP_NUMPY_ENABLED being 0, Java arrays are no longer translated into numpy arrays and arrive in python as Jep.NDArrays.

• OS Platform, Distribution, and Version: Linux Mint 22.0 (Ubuntu 24.04)
• Python Distribution and Version: 3.12.3
• Java Distribution and Version: Openjdk-11
• Jep Version: 4.2.0
• Python packages used (e.g. numpy, pandas, tensorflow): numpy-2.1.1

Log of venv creation, numpy and jep pip install:
jep_build.txt

Test program:
TestJepNumpySupport.java.txt

Test output:
test_output.txt

Any guidance is appreciated and I am more than happy to test or debug any possible solutions.
Thanks!
Erik

[bsteffensmeier]

Does #562 solve this for you?

[svensk-pojke]

Hi Ben, I will build and test that shortly. I just checked, and it is not specific to numpy 2, When I first caught the problem the install was using numpy 1.26.4. I'll post me results shortly. Thanks!

[svensk-pojke]

I downloaded jep, switched to dev_4.2 and installed jep in a fresh venv with numpy 2.1.1 and it still fails:

(venv)user@xxxx/mnt/My2TBDisk/asdt/24.04$  cd /mnt/My2TBDisk/asdt/24.04 ; /usr/bin/env /usr/lib/jvm/java-11-openjdk-amd64/bin/java @/tmp/cp_6gw3flufj1mqkhtttktkvubg4.argfile TestJepNumpySupport 
python version = 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0]
pip version = 24.0
numpy version = 2.1.1
jep version = 4.2.1
jep.JEP_NUMPY_ENABLED = 0
test_result = false

I double checked with numpy==26.4 and got the same result.

[svensk-pojke]

Here's an interesting test. Still using 24.04, but I used python 3.10 instead, and then tweaked the version of pip in use. The failure occurs when going from pip 23 to 24.

(venv) user@xxxx:/mnt/My2TBDisk/asdt/24.04$  cd /mnt/My2TBDisk/asdt/24.04 ; /usr/bin/env /usr/lib/jvm/java-11-openjdk-amd64/bin/java @/tmp/cp_cct0wsxiqrgwllet3sr5fu36r.argfile TestJepNumpySupport 
python version = 3.10.15 (main, Sep  7 2024, 18:35:38) [GCC 13.2.0]
pip version = 23.0.1
numpy version = 1.26.4
jep version = 4.2.0
jep.JEP_NUMPY_ENABLED = 1
test_result = true

(venv) user@xxxx:/mnt/My2TBDisk/asdt/24.04$  cd /mnt/My2TBDisk/asdt/24.04 ; /usr/bin/env /usr/lib/jvm/java-11-openjdk-amd64/bin/java @/tmp/cp_cct0wsxiqrgwllet3sr5fu36r.argfile TestJepNumpySupport 
python version = 3.10.15 (main, Sep  7 2024, 18:35:38) [GCC 13.2.0]
pip version = 24.2
numpy version = 1.26.4
jep version = 4.2.0
jep.JEP_NUMPY_ENABLED = 0
test_result = false

[svensk-pojke]

One other nugget. When installing using the old version of pip, it gives this deprecation warning:

DEPRECATION: jep is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

This makes me wonder if it fails when it builds a wheel for the install (Building wheel for jep (pyproject.toml)).

[svensk-pojke]

Hi Ben. I've been debugging this using the --verbose flag for pip install and adding print statements to setup.py to see if I can figure out what is happening. What I found is very strange. setup.py is failing on the try block starting at line 30. It is the import of numpy that fails, so it never even gets to the new path you added in dev_4.2. I am totally perplexed, as my steps have been:

1. create and activate the venv
2. pip install numpy
3. cd to fresh checkout of jep on dev_4.2 branch
4. run pip install . --verbose
   It blows up on the import numpy statement with this error: ModuleNotFoundError: No module named 'numpy'. Normally this is caught by the try block and it simply prints "numpy not found, running without numpy support"

This makes no sense to me at all, as I can manually start python in the venv and import numpy without any problem.

I am going to test some more and I will update you with what I find. Thanks.

[svensk-pojke]

If it helps, here's the stack trace:

  Running command Getting requirements to build wheel
  Traceback (most recent call last):
    File "/mnt/My2TBDisk/asdt/24.04/venv-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/mnt/My2TBDisk/asdt/24.04/venv-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/mnt/My2TBDisk/asdt/24.04/venv-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-_n9fcn1d/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=[])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-_n9fcn1d/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-_n9fcn1d/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 503, in run_setup
      super().run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-_n9fcn1d/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 318, in run_setup
      exec(code, locals())
    File "<string>", line 31, in <module>
  ModuleNotFoundError: No module named 'numpy'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /mnt/My2TBDisk/asdt/24.04/venv-3.12/bin/python3 /mnt/My2TBDisk/asdt/24.04/venv-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpbb41dqm5
  cwd: /mnt/My2TBDisk/asdt/24.04/jep
  Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

[svensk-pojke]

Update: I have been able to get past this problem by pip installing both "wheel" and "setuptools" before pip installing jep. I am still not able to build with numpy support, but I have a lead and am chasing that down now.

[bsteffensmeier]

I have not been able to recreate this problem in my test environment. I've tried both python 3.12 and 3.13-rc with pip 24.2. I do need to install setuptools before jep but without setuptools it doesn't disable numpy support, it just fails completely. I do not need to install wheel.

[svensk-pojke]

My test case builds fine with either setuptools or wheel (or both) missing and then does not support numpy. It does not crash as in your case. That could easily be attributed to any number of things, so I am not going to worry about it.

I still cannot get my simple test case to work with numpy support, so I think I have made some kind of error. But, the good news is: my original application (which is way more complicated) now works fine! And, in that case, all I needed was to add setuptools, just like you did, I did not need to add wheel.

OK, I think all that is required is setuptools and everything seems to work fine. I will hold off upgrading to numpy 2.X until you all do another release (4.3?). Thanks very much for your help! -Erik