Fix ModuleNotFoundError: No module named 'numpy' on build

[kounelisagis]

The following error occurs when trying to build. Fixed by adding numpy in build dependencies.

Traceback (most recent call last):
  File "/home/vsts/.local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
    main()
  File "/home/vsts/.local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/home/vsts/.local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 308, in get_requires_for_build_sdist
    return hook(config_settings)
  File "/tmp/build-env-dg88woc1/lib/python3.10/site-packages/setuptools/build_meta.py", line 328, in get_requires_for_build_sdist
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-dg88woc1/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-dg88woc1/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 9, in <module>
ModuleNotFoundError: No module named 'numpy'

Also update HISTORY.

[ihnorton]

In order for this to work correctly, we need to use numpy >= 1.25 to ensure that we build against an ABI-compatible version of NumPy (otherwise will get ABI mismatch errors at import time)

• see discussion here: Status of the project and foreseeable future ? scipy/oldest-supported-numpy#76
• and in README for this deprecated package

I believe we also need to make it conditional for python 3.8 like this:

TileDB-Py/pyproject.toml

Line 36 in e89521b

"numpy==1.17.*,<2.0 ; python_version == '3.8' and platform_machine != 'aarch64'",

so the build-system requires would look like:

"numpy==1.17.*,<2.0 ; python_version == '3.8' and platform_machine != 'aarch64'",
"numpy>=1.25 ; python_version >= '3.9"

[ihnorton]

cc @sgillies in case any comments, since you worked on cleaning up pyproject deps here in https://github.com/TileDB-Inc/TileDB-Py/pull/1963/files

[kounelisagis]

cc @sgillies in case any comments, since you worked on cleaning up pyproject deps here in https://github.com/TileDB-Inc/TileDB-Py/pull/1963/files

It seems like that PR broke the build procedure. Maybe the problem is the removal of:

[tool.setuptools.dynamic]
dependencies = {file = "requirements_wheel.txt"}

But just adding numpy in pyproject.toml doesn't seem to fix it. It then requires cmake and adding cmake in pyproject.toml won't fix the error: https://gist.github.com/kounelisagis/28f520e52d98e1c04f4f00573b310ace

[sgillies]

I'm sorry about the breakage. Maybe we can work on having our tests build and install a wheel instead of using python setup.py develop. I've started doing that on some of my open source projects and it helps find such problems early.

I'll check this out. We may want to start requiring numpy 2 in pyproject.toml for forward compatibility.

[sgillies]

@kounelisagis I'm on the dev branch, commit 8355035, and if I add numpy and cmake to pyproject.toml

[build-system]
requires = ["setuptools>=64", "wheel", "pybind11", "Cython", "cmake", "numpy<2"]
build-backend = "setuptools.build_meta"

Then I get an sdist and a wheel if I run python -m build.

(venv) $ python -m build
...
Successfully built tiledb-0.23.5.dev95.tar.gz and tiledb-0.23.5.dev95-cp311-cp311-macosx_11_0_universal2.whl

cibuildwheel invokes pip wheel and that works for me, too.

(venv) $ python -m pip wheel .
Processing /Users/sean.gillies/projects/TileDB-Inc/TileDB-Py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy<2.0,>=1.23.2 (from tiledb==0.23.5.dev95)
  Using cached numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl.metadata (114 kB)
Using cached numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl (14.0 MB)
Saved ./numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl
Building wheels for collected packages: tiledb
  Building wheel for tiledb (pyproject.toml) ... done
  Created wheel for tiledb: filename=tiledb-0.23.5.dev95-cp311-cp311-macosx_11_0_universal2.whl size=10996493 sha256=08bb8134b44e6bafbbb03aaa64b6d9f009bc270805463246b1b25d6f469aef88
  Stored in directory: /Users/sean.gillies/Library/Caches/pip/wheels/18/2c/f0/23f614e22e251d62d022ba76f9eb2f572a782fe2dbb8f051a5
Successfully built tiledb

I think that should be all that's needed. @ihnorton's comments about Python specific numpy versions look correct.

[ihnorton]

I'm sorry about the breakage.

I don't think there was breakage in any release, slightly unclear to me what that's referring to - no worries.

Maybe we can work on having our tests build and install a wheel

We do run tests on the wheels with cibuildwheel (but that probably won't catch a numpy ABI mismatch).

[kounelisagis]

@ihnorton's suggestion works after @dudoslav's help. NumPy was needed. No need to add cmake. Again, there was a problem with the windows wheel setup. A fix is coming for all the CI.