Status of the project and foreseeable future ?

[neutrinoceros]

As I'm sure you guys know, Numpy 1.25 introduced a mechanism to control backward compatibility at build time, which, quoting

makes the use of oldest-supported-numpy unnecessary

So, what is the future of this project ? I can imagine it will stop receiving updates at some point in the near future, but will it still be relevant when CPython 3.12.0 final is released (currently scheduled for early October) ?

I figure there would already be some internal discussions about it, but maybe this issue can serve as a reference for maintainers of downstream code.

[rgommers]

Thanks for starting the discussion @neutrinoceros. We have only touched on this topic lightly, so this is a good place for a more structural discussion.

To answer your concrete questions:

• There is no plan to some updating this package for now; it's heavily used and it's not much work to maintain.
• The relevance of this project is not coupled to CPython 3.12.0

What I would say is that we should encourage users of this package to instead adopt a direct numpy build dependency over time, so this package can fade away (it's an elaborate hack for the lack of features in Python packaging regarding ABI compatibility after all).

There are a couple of other things to think about:

• If users change their approach now, they have to again make updates for NumPy 2.0 - and while the plans for that release are clear, they haven't been implemented yet. I'd probably suggest to wait until that release is out.
  • At that point it should be possible to have a numpy>=2.0 build requirement and a numpy>=1.23.2 or similar runtime requirement (the ABI shim in NumPy 2.0 should allow that).
• The NumPy 1.25.0 feature isn't used much yet - it'd be great to see projects use it, and build against 1.25.0 and have a lower runtime requirement. That also needs to be tested in regular CI of projects, which is not something that is done anywhere yet AFAIK.
• Right now, just using numpy>=1.25.0,<2 as a build-time requirement may not translate too well to some other packaging systems which don't have isolated builds where build-time and runtime requirements can be different. Neither does the pinning strategy of course - it would be nice to summarize how this works for distro packagers (e.g., xref Too-strict version restrictions on NumPy break distribution builds scipy#18767).
• It is possible that some problems with the new approach will appear. For example, Cython may have too-strict checks on numpy version somewhere. Or adding const to existing signatures used to be fine, but now it may no longer be. @seberg the latter potential issue just came to mind, did you think about constness differences between 1.25 and older versions of the C API? E.g. MAINT: const correctness for the generalized ufunc C API numpy/numpy#23847 is recent. I think const having been added to any public API between the lowest-exported and build-time versions is a breaking change now.

For myself, I'd prefer to first experiment with this in a few other packages (e.g., PyWavelets and SciPy) before making any changes to this package.

[seberg]

Or adding const to existing signatures used to be fine, but now it may no longer be.

why not? Our public API is all C, so while inconvenient, C++ people can just cast. Since they can enforce compiling with the new numpy (at least rather quickly), hacks should be simpler, as you don't need both versions side-by-side for as long?

[rgommers]

I wasn't thinking about C++, only C and maybe Cython. I checked a bit more and it's indeed okay for C (ABI-wise) to go from a const parameter at build time to a non-const one at runtime (a case we did not previously allow, but now do). It's the 2->1 case in https://stackoverflow.com/questions/5083765/does-changing-fmystruct-a-to-fconst-mystruct-a-breaks-api-abi-in-c.

We recently had problems with this exact thing in Cython (see the README of https://github.com/lysnikolaou/const-cython), that's why it came to mind. But that's for signatures in .pxd files.

[neutrinoceros]

If users change their approach now, they have to again make updates for NumPy 2.0

Can you please clarify what you mean here @rgommers ?

[seberg]

Once NumPy 2.0 is out, you may want to bump the build requirement to NumPy >=2.0 rather than something like numpy>=1.25. That is because building for both 1.x and 2.x will require building with a 2.x to ensure you are use the newest headers.

(This is necessary to ensure that downstream must adjust to API changes. That allows us to do ABI breaks as long as they are disguised as API changes; in the simplest case, removing a function/macro.)

[neutrinoceros]

Interesting. Would you recommend pinning numpy<=2.0 as a runtime requirement in libraries that have numpy as a build time dependency, while numpy 2.0 isn't out ?

[rgommers]

Yes, that's recommended. <2.0 rather than <=2.0.

[neutrinoceros]

Thank you !

[Czaki]

Also, numpy==1.25.0 is python 3.9+ so any project that supports 3.8 still requires this package.

[rgommers]

Also, numpy==1.25.0 is python 3.9+ so any project that supports 3.8 still requires this package.

You can also take over the relevant 3.8 constraints from the setup.cfg file in this repo into your own pyproject.toml instead of relying on oldest-supported-numpy.

[rgommers]

Copying this comment from @matejsp because it belong in this issue:

One idea for this project future and easier usage/upgrades would be to replace all versions (that are 1.19.x & >=py3.9)

numpy==1.23.3,>=1.25.0<2; python_version=='3.9' and platform_system=='OS400' and platform_machine!='loongarch64'

ALTERNATIVE: I don't know if you can have multiple ranges that work with setup tools (maybe >= and then exclude all version up to 1.25.*:

numpy>=1.23.3,!=1.24.*,<2; 
or
numpy>=1.19.3,!=1.20.*,!=1.21.*,!=1.22.*,!=1.23.*,!=1.24.*,<2 ; python_version=='3.9' and platform_system not in 'OS400' and platform_machine not in 'arm64|loongarch64'

For most platforms you would then just pick best numpy version with best OS compatibility and STILL retain backwards compatibility :) Could be win win :D

[rgommers]

I just released a new version and closed all open PRs and almost all open issues. I think we can try a next release which uses more flexible constraints and targets numpy 1.25/1.26, making use of those versions targeting an older C API by default. That will resolve several "build from source" issues where the oldest numpy version that we target now no longer builds from source for some reason, but we must keep targeting it because (a) it's the first release with wheels and (b) backwards compatibility. So far that has held up well - at least no complaints regarding ABIs/versioning since 1.25.0 was released in June'23 AFAIK.

I think I'd go for this flavor (if the pin now is ==1.19.3 for some platform):

numpy>=1.19.3,!=1.20.*,!=1.21.*,!=1.22.*,!=1.23.*,!=1.24.*,<2

[seberg]

For projects which use 3.9 anyway, is there actually much of a reason to specify any bound at all (unless they have some requirement at build time)? A wheel build should presumably pick up the 1.25 or 1.26.
If for some reason we pick up an earlier version, maybe it is OK to assume that whatever is being compiled doesn't have to be compatible even earlier versions? (Since building against those versions is fine, you just don't get the maximum compatibility range.)

[rgommers]

If for some reason we pick up an earlier version, maybe it is OK to assume that whatever is being compiled doesn't have to be compatible even earlier versions?

We do get real bug reports for such situations. I just closed one from scikit-learn, where they had issues because building against 1.23.x (as in this repo) resulted in issues with folks who installed 1.22.x from source on a technically not-yet-supported Python version. We really should try to avoid this kind of thing - it's not hard to do so I believe.

For the time being, projects will be supporting 1.21 - 1.24 versions commonly as runtime dependency, and those versions will run into compat issues.

Also, I think we should keep a <2 bound until we are sure that 2.0 will work.

[seberg]

Well, sklearn is "special" since they support Python 3.8 so they clearly need oldest-support-numpy! I think it is also a good choice here to just do it as you did.

But if we rely on Python >3.9 and oldest-support-numpy becomes less interesting, I am not sure that there is any reasonable situation where:

1. you don't pick NumPy 1.25 or 1.26 at build time (since the newest are preferred anyway), but
2. also are shipping the build to people who may have an older NumPy install.
   So while maybe not perfect, I am not sure there is a very relevant failure mode. OTOH, maybe it just doesn't matter, because there is also nothing that gets solved by just removing the pins.

[rgommers]

There are lots of projects that still support Python 3.8. Once 3.9 is the minimum, I think the advice is to not use oldest-supported-numpy at all - simply use numpy>=1.25.0,<2.0 instead. Maybe that is what you meant? If so, I agree - I was thinking here about constraints to keep in this package.

[seberg]

What I was wondering was whether it isn't OK to just skip the numpy>=1.25, because I can't see a realistic failure mode. I doubt there is any reasonable way to end up building wheels with <1.25 even without a requirement.

[rgommers]

Oh I'm sure there's a failure mode that people are going to hit - the only question is how often. Example, when building everything from source (whether due to a pip setting or on a platform with no wheels):

• Run pip install pkg1 pkg2 # pkg1 depends on numpy, pkg2 on numpy<=1.24
• now pkg1 is built against numpy 1.24, and the produced wheel for it will be cached by pip without that information included
• Later, in a different env with numpy 1.23.x installed, run pip install pkg1 - voila, problem.

[rgommers]

There's also things like pip download or transitive dependencies where all involved packages get built in a single build env rather than each in their own separate isolated build envs.