Outdated versions of sphinx and other dependencies being installed (with pyproject.toml), search stuck

[EwoutH]

Problem description

We've encountered an issue with Read the Docs configuration where search is not functioning properly, likely due to incompatible versions of sphinx, sphinx-rtd-theme, and other dependencies being installed. Our dependencies are specified in an optional docs dependency in a pyproject.toml file, and our .readthedocs.yaml includes this:

[project.optional-dependencies]
docs = ["sphinx", "sphinx-rtd-theme", "readthedocs-sphinx-ext", "nbsphinx", "myst", "pyscaffold", "myst-parser"]

Then, our .readthedocs.yaml includes this:

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

However, our Read the Docs build log first installs sphinx<2 and sphinx-rtd-theme<0.5, and only after that, in a separate step, our actual .[docs] optional dependencies, which include sphinx-6.1.3 and sphinx-rtd-theme-1.2.0. This issue is detailed in the PR quaquel/EMAworkbench#242.

We've also tried adding a docs/requirements.txt file specifically for Read the Docs, but it still installs outdated versions of the dependencies. We're following the documentation, but search still doesn't work.

Expected behavior

The Read the Docs build should install the correct versions of sphinx, sphinx-rtd-theme, and readthedocs-sphinx-ext as specified in the pyproject.toml file, and search should function correctly.

Steps to reproduce

1. Set up a pyproject.toml file with optional docs dependencies as shown above.
2. Configure .readthedocs.yaml as shown above.
3. Create a Read the Docs build.
4. Observe the installed versions of sphinx, sphinx-rtd-theme and other dependencies

Environment details

• Project: quaquel/EMAworkbench
• Build URL: https://readthedocs.org/projects/emaworkbench

Additional context

• Our PR quaquel/EMAworkbench#242
• Possible related issue: readthedocs/sphinx_rtd_theme#1452

[stsewd]

However, our Read the Docs build log first installs sphinx<2 and sphinx-rtd-theme<0.5,

Hi, this is to keep backwards compatibility with old projects, see https://docs.readthedocs.io/en/stable/build-default-versions.html#python. I can remove your project from that "feature", so it installs the latest version by default.

But your main problem about the search not working is because of readthedocs/sphinx_rtd_theme#1452, please follow the workaround from readthedocs/sphinx_rtd_theme#1452 (comment).

[stsewd]

I can remove your project from that "feature", so it installs the latest version by default.

Done https://readthedocs.org/projects/emaworkbench/builds/20258938/

[stsewd]

Also seeing your PR quaquel/EMAworkbench#242, note that search options work over the final HTML pages produced by your build https://docs.readthedocs.io/en/stable/config-file/v2.html#search-ranking, not the source files or files that are in your repo and aren't included in the final build of your docs.

[EwoutH]

Thanks a lot for getting back!

Hi, this is to keep backwards compatibility with old projects, see https://docs.readthedocs.io/en/stable/build-default-versions.html#python. I can remove your project from that "feature", so it installs the latest version by default.

Done https://readthedocs.org/projects/emaworkbench/builds/20258938/

Thanks!

One small note, it still installs setuptools<58. This a problem for us, since our docs dependencies are defined in a pyproject.toml. Optional requirements are only properly support from 63 and up. Is it possible to install a later version of setuptools in Readthedocs?

But your main problem about the search not working is because of readthedocs/sphinx_rtd_theme#1452, please follow the workaround from readthedocs/sphinx_rtd_theme#1452 (comment).

Thanks, this fixes our problem!

We started teaching a course today, so over a hundred students thank you!

CC @quaquel.

[stsewd]

One small note, it still installs setuptools<58. This a problem for us, since our docs dependencies are defined in a pyproject.toml. Optional requirements are only properly support from 63 and up. Is it possible to install a later version of setuptools in Readthedocs?

I have forgotten about that, looks like this was done to break projects using setup.py install #8659, I remember that we talked about doing something smarter if this was a problem in the future (checking if the project is using setup.py install, and only install that version for those projects, and install the latest version for all others). So, probably that day has arrived :D

As a workaround, you can install the required version of setuptools just before installing your requirements.

https://github.com/quaquel/EMAworkbench/blob/1a3c678125415eea6f15bc27eec3bf9475033a08/.readthedocs.yaml#L24-L26

python:
  install:
    - requirements: docs/setuptools.txt
    - method: pip
      path: .
      extra_requirements:
        - docs