Sphinx does not work with docutils 0.21.1

[rikardn]

Describe the bug

When trying to use the latest version of docutils together with Sphinx 7.2.6 I get the following error:

The conflict is caused by:
    The user requested docutils==0.21.1
    sphinx 7.2.6 depends on docutils<0.21 and >=0.18.1

This is a repetition of #11414

The sdist on PyPI for docutils 0.21 was incomplete so the 0.21.1 is an important fix.

Perhaps again you could consider not capping the docutils version since it hinders downstream users to handle dependency issues like this. This stance is further explained in the following blog post (linking to the TL;DR): https://iscinumpy.dev/post/bound-version-constraints/#tldr

How to Reproduce

In requirements.txt:

docutils==0.21.1
sphinx==7.2.6

Environment Information

Not needed

Sphinx extensions

No response

Additional context

No response

[jayaddison]

Hi @rikardn - thanks for the suggestion; from my perspective I tend to think that Sphinx is following a reasonably good practice here.

I do agree that there are sometimes good reasons not to range-limit dependency versions, but the choice of whether to do so (either for setup-based dependency specs, and/or for package metadata requirements) can vary contextually by source project, and over time.

In the case of Sphinx, a fairly well-deployed project that wants to cautiously upgrade the core reStructuredText-processing docutils component, I think it's OK to have a version restriction that indicates more-or-less what the maintainers believe is well-supported and should work for users. None of us have infinite time, and constraining version ranges also has the effect of reducing the number of combinatorial possibilities for bug interactions, and from unexpected interactions from relatively-new code.

Even so: I do believe that early testing of dependencies is useful, especially from users who are familiar with projects. So I would encourage you to upgrade, but I would not encourage Sphinx to upgrade everyone's defaults until the project itself is comfortable to do so.

(security upgrades can be something of an exception to this rule, although even then I think there can be reasons to be careful, and sometimes to backport changes to prior versions)

To achieve your upgrade, I'd note that removing the docutils line entry from the requirements.txt file, and running both pip install -r requirements.txt and then pip -U docutils<0.22' would be one way to do that, albeit with an extra step. It is true that pip` will continue to warn you about the conflict, but the ecosystem doesn't prevent you from doing it - ultimately it is a set of source code files that are placed into a filesystem directory structure.

My understanding from reading a relevant pip issue thread at pypa/pip#8076 is that allowing relaxation of some dependency specifications is a feature-request that is under consideration, and that it would theoretically be possible in future for an additional file format to allow overrides of certain package constraints - but I'd tend to sympathise with the thinking that maintenance, bugreport and debugging overhead could arise if those override files became used (even if rarely, given sufficiently-complex cases).

Those kinds of overheads tend to apply ecosystem-wide - during learning about systems, during packaging and release, during maintenance, during development, bugreporting, testing, investigations, auditing, continuous integration, and so on - so even apparently-small improvements that seem to address a reasonable-sounding use-case can in fact cause problems that drag on an entire community. That's a frustration that I frequently find when trying to get small changes for things I care about introduced in other systems, and it can be annoying, but simplicity is extremely beneficial in software engineering. Dependency versioning and compatibility for code is, I think, a fairly intrinsically hard problem - perhaps the API-versioning side of it is mostly-tractable given strongly-enough typed languages, but runtime behaviour is also a consideration, and, to my knowledge, that's difficult to express or constrain using version numbers.

On a theoretical-possibilities note, I'd also theorize that if a future docutils upgrade did break Sphinx in some way -- maybe causing a bunch of frustration and time lost by people who install that, their userbase, and then Sphinx/docutils maintainers to investigate under some sense of pressure -- then a hypothetically-likely (in my mind) short-term remediation would be to reintroduce a version range constrant, and then we'd be back at the current status-quo.

So: please do upgrade if you're ready, but I'm fairly skeptical about whether Sphinx should remove the version range.