build.commands shall support custom package installation (build.apt_packages)

[danwos]

Details

This is based on a comment from @humitos : #3885 (comment)

build.commands does not allow to install extra packages via build.apt_packages.

Even if build.commands gives me more flexibility in the build process, it takes also a lot of "features" away from me, because I'm not able load/install additional, non-python programs/libs anymore.

Expected Result

Take build.apt_packages into account when using build.commands.
Or allow using sudo apt-get install XY.

Use case

I want to run PlantUML, which normally works on RTD.
But I also need a custom build, as I must build 2 PDFs before building my HTML docs.
Used Config: RTD-file of Sphinx-SimplePDF

Without PlantUML, the Sphinx-Needs extension can not be used to build any kind of diagram.

So in the end, for this use case, all I need is Java in machines when using build.commands :)

[humitos]

I think this is a good feature and I'd like to make it possible somehow. We are currently not allowing sudo or using root user because of security. We've discussed to migrate "the build process" completely outside the application itself and communicate each other via a well defined contract (see #9088); in that case, we can give people full-control of the Docker instance without problems. However, this is more a medium/long term goal, I guess.

Or allow using sudo apt-get install XY.

Is it secure to allow people to execute only one command via sudo by using /etc/sudoers with something similar to:

docs NOPASSWD: /usr/bin/apt-get install, /usr/bin/apt-get update

My concern here is people running things like sudo apt-get install `bash -c "malicious-command shutdown"` . In that case, the malicious-command will be executed as root as part of the apt-get command? My example may not be the best one, but I suppose that trying a little harder it may be possible to break the commands that are allowed to be executed by sudo. cc @agjohnson

[agjohnson]

Yeah, sudoers policies are hard to get right and are a common exploit target. There are similar exploits to yours that would allow root access.

build.commands does not allow to install extra packages via build.apt_packages

So, this is where I've always seen build.jobs being a more powerful implementation. We already support the installation of apt packages with our normal build process, so re-implementation of this feature via sudo doesn't seem very valuable.

The linked configuration file above is using build.commands, but seems it could just use build.jobs instead. The config is duplicating what our build process already does, but does a second build pass in addition. To use build.jobs instead, it could still build a secondary Sphinx project and would get all of the benefits of our existing build features (package installation, apt package installation, etc).

[humitos]

@agjohnson

The linked configuration file above is using build.commands, but seems it could just use build.jobs instead. The config is duplicating what our build process already does, but does a second build pass in addition. To use build.jobs instead, it could still build a secondary Sphinx project

This is a good point and I agree that it should work for this particular case 💯 . @danwos would you like to try this approach using build.jobs and reporting back here? 🙏🏼

That said, even if it works for this case, I still think it's a good feature to develop eventually. I don't think it's a priority tho, in particular because we don't have/know many cases where this situation will be a limitation that cannot be achieved by build.jobs. This feature is pretty useful for the use case: "non-standard build process + require installing system packages"

I've already faced that use cases when trying to build https://sphinx-themes.org/ on Read the Docs. It uses nox to build the documentation and requires installing playwright. This is the YAML file I used as a test: https://github.com/humitos/sphinx-themes.org/blob/9fe52d4ddcea0755a9863e93f1af0273a2feb8e4/.readthedocs.yaml -- cc'ing @pradyunsg since he is maintainer of that project and I assume he hit this problem as well and that's probably why this project is not on Read the Docs.

[agjohnson]

That said, even if it works for this case, I still think it's a good feature to develop eventually.

Yeah, this does feel like a disconnect, but I think we're probably in agreement on priority too.

I would probably still want to discuss a way to reference existing features from build.commands before reimplementing this with another build process pattern. I imagine there is a configuration file pattern that we might like around referencing build job steps from build.commands, like:

build:
  commands:
    - readthedocs: install-apt-packages
    - readthedocs: install-python-packages
    # Or    
    - /usr/bin/readthedocs-install-apt-packages
    # Etc

Or perhaps the long term plan of providing this customization using a buildpack like structure would be another conversation.

[humitos]

I would probably still want to discuss a way to reference existing features from build.commands before reimplementing this with another build process pattern

I'd prefer if we don't build anything inside the core application for this and we push forward the build contract instead (#9088). Once we have that implemented, we can execute the build process in a completely isolated instance outside our application and we can give people full control of it (even with root permissions). We only need to know where the artifacts generated by that process are and copy them back to our application instance before post-processing and uploading them to S3.

[henryiii]

FYI, at scientific-python dev summit, we've been trying to implement a Jekyll build in readthedocs, and apt commands are needed to install ruby (Ruby as a tool would be exactly what we want, actually!). Just wanted to show another use case that doesn't work with build.jobs.

[agjohnson]

@henryiii just to clarify, are you using build.commands or build.jobs configuration? We discussed a path forward to making build.jobs easier to use for arbitrary documentation generation, and this would make APT package installation possible at the same time.

[henryiii]

Our first attempt was with build.commands. I would highly recommend a note in the documentation that apt does not work with build.commands - that would have saved us a lot of debugging. Now we are trying build.jobs; I'm assuming, since we can't control the actual build step, that we'll have to put in a dummy sphinx or mkdocs config to make it pass the build step (though not sure what it does for other languages like nodejs).

And setting up the Python virtual env takes a little useless time. :)

[agjohnson]

Okay yeah, this is exactly the case we discussed then. You have pinpointed the current issue, and you will need to get creative with build.jobs to replace the build command step for building. With the ability to override this step too, build.jobs would be usable by projects looking to replace this step entirely.

[humitos]

We should probably add Ruby to "build.tools" options. There is no reason to not have it there

[humitos]

@henryiii BTW, you could solve your immediate issue by doing something like:

build:
  commands:
    # Install Ruby using "asdf"
    - asdf plugin add ruby https://github.com/asdf-vm/asdf-ruby.git
    - asdf install ruby 2.6.4
    - asdf global ruby 2.6.4
    # ... your other commands here

[henryiii]

This is what we ended up with:

https://github.com/scientific-python/cookie/blob/f84144ac83f080412b3516e2cdb9cd1d8806f19c/.readthedocs.yml

(Also 1-3 of the Zarr webpages are setup like this now too)

It takes about 7 mins, 5 of which are building Ruby, and 1.5 or so are building four of the plugins that Jekyll needs (Ruby doesn't have Python's redistributable "wheels", just an "SDists" equivalent). If we could avoid building Ruby, that would be a nice savings, and if we could cache the binaries, we could save most of the rest of the time. Ruby was already installed on the runners, we were just missing the headers package, otherwise we probably could have used the built-in Ruby (though I'd still have to come up with the right environment variable incantation to use it without sudo, but should be possible).

[humitos]

Glad the yaml snippet helped here. #10346 will improve the download/compiling time spent on Ruby. We cannot cache package/gem installation, tho, but at least the compiling time will be saved. We can continue the conversation on that issue, since we are deviating the topic a little from this original issue 👍

[agjohnson]

I'd like to prioritize adding support for build.jobs.build step override so that we can point users towards build.jobs when they want to use custom build pipelines. I still do think build.jobs is a best fit for advanced usage patterns, and build.commands is more for expert users. But without a build.jobs.build override, we'll have to force these users into expert usage right away.

I think it will be hard for us to feel good about implementing a sudo layer for build.commands usage, so there is a more obvious path in making build.jobs a nicer experience.

Do we want to treat this as a separate issue (perhaps there is one already, but I wasn't finding it)?

[humitos]

I think it will be hard for us to feel good about implementing a sudo layer for build.commands usage, so there is a more obvious path in making build.jobs a nicer experience.
Do we want to treat this as a separate issue (perhaps there is one already, but I wasn't finding it)?

Yes. Overriding pre-defined jobs (e.g. build.jobs.build) is something different from supporting the installation of OS packages. I'd treat them differently 👍🏼 . Also, their requirements differ too much from each other.

[agjohnson]

Yeah, the two are different, technically speaking, but what I'm saying is that if we add support for overriding build.jobs.build, I think we would point most users towards customization through build.jobs instead of build.commands. This issue might then not be a strong concern if users have a way to both use most of our configuration file and replace the build job commands.

I'm also not suggesting to do anything like remove build.commands, but I think build.jobs would be a better implementation and much easier for us to communicate to users.

[humitos]

Gotcha! I think for a lot of cases build.jobs will be enough and I will be happy recommending that to most of our users. However, installing a OS package with build.command, and giving people sudo superpower will still be a good and valid feature to support eventually. I'm not worried about giving people sudo superpowers once we have our build system more isolated. However, I don't think we will be able to implement it before that.

[ssbarnea]

I ended up here while trying to enable git-lfs,... just to discover that it does not work if we use build commands...

[humitos]

@ssbarnea we have a git-lfs example in our documentation that does not require installing packages with APT: https://docs.readthedocs.io/en/latest/build-customization.html#support-git-lfs-large-file-storage. Can you try that and let me know if it works for you?

[ktbarrett]

Why is there this discussion and decision to wait for future infrastructure updates when simply supporting build.apt_packages and build.commands together would solve the original problem? Not supporting it doesn't seem "correct" anyways.

I'm running into the same problem and such a fix would do everything I need it to do.

[ktbarrett]

It seems that build.commands was introduced to allow user full customization of the build process and override the standard process. That's cool and all, but it was introduced the infrastructure to support it was available.

Instead what should have been introduced is a way to override the build step of the standard process. That's something that could be immediately supported and would offer a lot of flexibility for the user. I'm just trying to change what build command to run, which is the extent of what the example in the docs shows, and I think is sufficient for 95% of people who ever look into this feature.

[humitos]

@ktbarrett the thoughts/patterns behind this decision are:

• build.jobs allows users to quickly setup their project and assist them with the default commands that have to be executed for Sphinx and MkDocs projects ¹, creating an abstraction layer
• build.commands allows users to override the default build process completely, giving full control to the user. Read the Docs won't run any command on behalf of the user in this scenario

That said, we weren't able to add support for sudo on build.commands (and build.jobs) yet due to security reasons. On the other hand, we don't want to add support for build.apt_packages when using build.commands because it re-introduces "running command on behalf the users" in the build process we have created specifically to avoid that particular scenario.

Supporting sudo, which I consider is the correct solution here, is in our roadmap but we haven't been able to prioritize this work yet.

I hope that helps to clarify the situation.

Footnotes

1. we plan to expand these doctools in the future. ↩

[ktbarrett]

Yes, I understand what's intended here. What I'm saying is that a solution achievable now would be a separate feature that would allow the user to override the build stage in build.jobs (currently you can only override pre and post actions), so we can customize the build call without overriding the whole build flow. I have 0 interest in overriding the whole build flow and I'm gonna guess that 99+% of other users also have 0 interest in that. IMO It's overkill to make users redo the whole build flow just to override the build call.

[humitos]

What I'm saying is that a solution achievable now would be a separate feature that would allow the user to override the build stage in build.jobs (currently you can only override pre and post actions), so we can customize the build call without overriding the whole build flow.

Yes. We've discussing this internally and we already have a plan to move forward. However, we weren't able to prioritize this work yet.

I'm gonna guess that 99+% of other users also have 0 interest in that. IMO It's overkill to make users redo the whole build flow just to override the build call.

Right, if you want to build Sphinx or MkDocs projects you probably don't need to override it. However, if you want to build a project that uses Pelican, Docusaurus or any other documentation tool, you definitely will need build.commands to override the whole process.

[agjohnson]

I have felt that even Pelican/Docusaurus/etc should all be supported with build.jobs too. Would there be anything stopping the user from doing something like this if we supported it?

build:
  jobs:
    build:
      html:
        - pelican --settings docs/pelicanconf.py --output $READTHEDOCS_OUTPUT/html/ docs/

To me, the build.commands split feels closer to what @ktbarrett is describing -- something that only the expert, or 1-5%, users should use. The majority of our users should use build.jobs so that all of our features/fixes are usable by the project, and so most users don't have to deeply understand RTD to configure their project.

[humitos]

I think this issue is deviating a little from its original purpose at this point.

@agjohnson that workflow probably works for most of the Python doctools. However for any non-Python doctool it doesn't make too much sense to create a virtualenv, install pre defined dependencies, etc.

We will need to un-tie our current build process from Sphinx/MkDocs and Python more to support other doctools with build.jobs.

We've touched on "other pre-defined build packages" that's related to this, but that's for a different conversation that is outside the scope of adding sudo support for build.commads

[agjohnson]

Yeah, I'm not talking that deep on package reuse yet.

This conversation is centered around finishing the overrides to build.jobs.build because we get more value from less effort with that feature. I agree these two features are not necessarily in conflict, but we also haven't been able to prioritize finishing the build.job.build overrides. They accomplish the same goal of getting APT to users with custom build commands though.

There is work in build job overrides, but duplicating apt_packages is going to require the processing the minefield of sudo hardening, which is full of edge cases.

It's good to be mindful of what we're designing for, but I also feel like designing primarily for non-Python tools is a preoptimization. Python projects are the vast majority of projects and gain more from an intermediate configuration of build.jobs.build.html etc.

[ssbarnea]

I ended up on this issue because I discovered that rsync is missing and that I am unable to use apt_packages due to custom build commands.

Not only that rsync is missing, but any attempt to download and install the binary will fail because it will not have access to required system libraries, rsync is not really a portable executable.

rsync: error while loading shared libraries: libpopt.so.0: cannot open shared object file: No such file or directory

Reference: RsyncProject/rsync#72

[agjohnson]

@ssbarnea As a data point, what are you doing with your build that requires build.commands? Would it be possible/easier/harder for you to use a build job override like build.jobs.build.html instead?

To clarify/summarize the above, if projects can use the override build.jobs.build.html, they also have access to build.apt_packages already and do not need another implementation of APT package installation that works with build.commands.

[ssbarnea]

@agjohnson I was wondering about the same. Long story short, I am using tox -e docs as the command for building docs as I want to avoid risking for the docs build process to diverge (be as close as possible when run locally, on GHA or on RTD.

Still, considering that the build.commands are highly advertised with the note that they will become default, I am really worried about the lack of apt_packages. The reality is that is is quite likely for someone to need system packages in order to build, sadly asdf and pip are often not enough. I can understand the concern about sudo but TBH the solution for this is to just use containers and allow users to do whatever they want/need inside.

Update: Apparently nope, we cannot hack the old process because the hardcoded call of python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html is not really compatible with our codebase. We generate some of the RST files, store them in other locations, even the conf.py file is generated.

[agjohnson]

Apparently nope, we cannot hack the old process because the hardcoded call of python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

That makes sense for where we are currently. But with build.jobs.build.html overrides, instead of executing the hard-coded command, or writing all of your own build commands, you'd instead use a configuration based on this:

build:
  apt_packages: rsync
  jobs:
    build:
      html:
        - tox -e docs
        - mv .tox/docs/tmp/html $READTHEDOCS_OUTPUT/html

This is just an example of course. The hard-coded sphinx-build command would be replaced with your tox -e docs, but otherwise all of our normal build process (like support for build.apt_packages) would be usable.

The intention behind build.commands was to support expert, custom builds that need to replace the build process, where build.jobs allowed for extending the build process. Your example may be in the middle, but it doesn't quite seem like you need to replace all of the build process commands. What you are describing seems like it might be a fit for build.jobs.build.html, but that's without looking at anything extra you're doing in around tox and the build process.

[miroi]

Hello,

I have problem - I need apt_packages working with commands .

I have to override the default commands with own commands, but I also need Ubuntu apt_packages.

This is the my .readthedocs.yaml file:

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  commands:
    - echo "in commands:"
    - echo "READTHEDOCS_OUTPUT=$READTHEDOCS_OUTPUT "
    - pwd; ls -lt
    - cmake --version
    - git --version
    - python -V
    - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
    - sphinx-build  --version
    - git submodule update --init --recursive
    - ./setup $READTHEDOCS_OUTPUT
    - pwd
    - ls -lt
    - cd $READTHEDOCS_OUTPUT; pwd; ls -lt; make html; pwd; ls -lt; cd $READTHEDOCS_OUTPUT/html; pwd; ls -lt

[agjohnson]

@miroi Your example matches what we are talking about above. Users using build.commands cannot use any other features of our configuration file, such as APT package installation.

However, your use case seems like you should be able to use our default build steps with additional build.jobs overrides to run the extra commands (instead of defining everything manually with build.commands). You'd want to look at overriding a pre build job with your additional setup (./setup).

To clarify what our plan is here, we will not support APT package installation in combination with build.commands, but instead will support overriding our default commands in build.jobs configuration.

We are working on implementing overrides for build.jobs.build.html -- the ability to control the exact command being executed. This would be the most ideal fit, if you really need to execute make html instead of using our standard Sphinx support.

[miroi]

Hello @agjohnson ,

is it fixed here please, #11710 ?

[agjohnson]

Yes! This PR is merged but not released yet. It will be included in our release tomorrow

[miroi]

rify what our plan is here, we will not support APT package installation in combination with build.commands, but instead will support overriding our default commands in build.jobs configuration.

Could you please help me how to arrange my .readthedocs.yaml file after this feature is merged in ? I need APT packages and run the setup command with make html...And avoid the default commands, which are causing crash...

# .readthedocs.yaml
#
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
#
# For the DIRAC documentation buildup  adapted by Miro Ilias, Matej Bel University, Slovakia
#

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  jobs:     
    pre_create_environment:
      - echo "Command run at 'pre_create_environment' step"
      - echo "READTHEDOCS_OUTPUT=$READTHEDOCS_OUTPUT "
      - pwd; ls -lt 
      - cmake --version
      - git --version
      - python -V
      - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
      - sphinx-build  --version
      - git submodule update --init --recursive
    
      - ./setup $READTHEDOCS_OUTPUT
      - pwd
      - ls -lt
      - cd $READTHEDOCS_OUTPUT; pwd; make html; ls -lt; cd $READTHEDOCS_OUTPUT/html; pwd; ls -lt 

still the default commands are showing up and are hurting the final html....

cat doc/conf.py
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

[miroi]

rify what our plan is here, we will not support APT package installation in combination with build.commands, but instead will support overriding our default commands in build.jobs configuration.

Could you please help me how to arrange my .readthedocs.yaml file after this feature is merged in ? I need APT packages and run the setup command with make html...And avoid the default commands, which are causing crash...

# .readthedocs.yaml
#
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
#
# For the DIRAC documentation buildup  adapted by Miro Ilias, Matej Bel University, Slovakia
#

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  jobs:     
    pre_create_environment:
      - echo "Command run at 'pre_create_environment' step"
      - echo "READTHEDOCS_OUTPUT=$READTHEDOCS_OUTPUT "
      - pwd; ls -lt 
      - cmake --version
      - git --version
      - python -V
      - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
      - sphinx-build  --version
      - git submodule update --init --recursive
    
      - ./setup $READTHEDOCS_OUTPUT
      - pwd
      - ls -lt
      - cd $READTHEDOCS_OUTPUT; pwd; make html; ls -lt; cd $READTHEDOCS_OUTPUT/html; pwd; ls -lt 

still the default commands are showing up and are hurting the final html....

cat doc/conf.py
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

OK, I found the solution - mere, ugly delete of wrong "html" directory, created by default buildup:

# .readthedocs.yaml
#
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
#
# For the DIRAC documentation buildup  adapted by Miro Ilias, Matej Bel University, Slovakia
#

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  jobs:     
    pre_build:
      - echo "in pre_build...."
      - echo "READTHEDOCS_OUTPUT=$READTHEDOCS_OUTPUT "
      - pwd; ls -lt 
      - cmake --version
      - git --version
      - python -V
      - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
      - sphinx-build  --version
      - git submodule update --init --recursive
      - ./setup $READTHEDOCS_OUTPUT
      - pwd
      - ls -lt
    post_build:
      - echo "in post_build..."
      - cd $READTHEDOCS_OUTPUT; pwd; ls -lt; /bin/rm -rf html; cmake ..; make html; ls -lt; cd $READTHEDOCS_OUTPUT/html; pwd; ls -lt 

I still do not know how to avoid the default commands :

cat doc/conf.py
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

[humitos]

@miroi you should use something like the following (probably, with some adjustments):

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  jobs:    
    post_checkout: 
      - git submodule update --init --recursive
    install:
      - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
    pre_build:
        - ./setup $READTHEDOCS_OUTPUT
    build:
      html:
        - make html

I haven't tested it, but it seems this is the YAML structure you are looking for.

[miroi]

@miroi you should use something like the following (probably, with some adjustments):

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  jobs:    
    post_checkout: 
      - git submodule update --init --recursive
    install:
      - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
    pre_build:
        - ./setup $READTHEDOCS_OUTPUT
    build:
      html:
        - make html

I haven't tested it, but it seems this is the YAML structure you are looking for.

Thanks @humitos ! After some playing got a working yaml file, based on your template:

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
    - gfortran
    - gcc
    - g++
    - git
  jobs:    
    post_checkout: 
      - git submodule update --init --recursive
    install:
      - pip install "sphinx<7.0.0" sphinx_rtd_theme sphinxcontrib-bibtex
    pre_build:
        - ./setup $READTHEDOCS_OUTPUT
    build:
      html:
        - cd $READTHEDOCS_OUTPUT; echo "I am in the directory:";pwd; make html; ls -lt

[humitos]

We've deployed some changes related to our builders. Now it's possible to override build.jobs.* and avoid executing the default jobs (e.g. build.jobs.create_environment) if you don't specify sphinx nor mkdocs keys.

That means that you can write a YAML file like the following:

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  apt_packages:
    - cmake
  jobs:    
    build:
      html:
        - make html
      pdf:
        - make pdf

Note that this build:

• won't create a Python virtual environment
• won't install any default Sphinx/MkDocs Python dependencies
• will install cmake via apt-get
• will build the HTML documentation using make html and the PDF file using make pdf

This approach should be enough for those users that require having full control of the Read the Docs building process and also need to use apt-get to install OS level dependencies. Let us know if this helps you with your use case. We really appreciate your feedback.

We plan to keep expanding the build.jobs.* pattern to add more features. We recommend using build.jobs over build.commands.