Read the Docs Addons enabled by default and injected Sphinx context removal

[humitos]

Hi all 👋🏼. I want to communicate we are deprecating auto-injected Sphinx context and enabling Read the Docs Addons by default on October 7th, 2024. Read the public announcement at our blog post Read the Docs Addons enabled by default.

I'm opening this issue here as a way to connect with other Sphinx theme authors and coordinate the required changes on those themes before reaching the deprecation date and make the transition as smooth as possible:

• @jbms & @2bndy5 (jbms/sphinx-immaterial)
• @agoose77 & @choldgraf (executablebooks/sphinx-book-theme)
• @pradyunsg (pradyunsg/furo)
• @drammock (pydata/pydata-sphinx-theme)
• @mgeier (mgeier/insipid-sphinx-theme)

Feel free to ping other theme authors you may know here so they are aware of the upcoming changes as well.

Summary of the changes

Read the Docs will stop installing readthedocs-sphinx-ext Python package by default and stop appending extra configurations to the Sphinx's conf.py when building projects.

We've created projects on Read the Docs to show the Sphinx configuration for the old and new behavior. This allow us to understand what are the differences (search for html_context on each page, as an example):

• Page with current and deprecated behavior: https://test-builds.readthedocs.io/en/latest/sphinx-context.html
• Page with new default behavior: https://test-builds-disable-sphinx-manipulation.readthedocs.io/en/latest/sphinx-context.html

Details of the changes

When building on Read the Docs, there are some changes performed automatically via a Sphinx configuration file by conf.py.tmpl:

• There are some workarounds to support old versions of Sphinx
• Injects plenty of variables in the html_context
• Adds _static directory to html_static_path if it exists
• Sets alabaster theme if there is no theme defined
• Set html_baseurl as canonical_url if Sphinx >=1.8
• Install readthedocs_ext.readthedocs as a Sphinx extension
• Install readthedocs_ext.external_version_warning if the Read the Docs version is external
• Define some LaTeX configurations based on project's language

Template variables available via html_context

If you are depending on any of this variables injected into the HTML context, you will need to perform some changes on your theme. Note there are some environment variables that Read the Docs passes to the build process that you may be able to use for this purpose. As an example, note that html_context["commit"] won't injected anymore, but you can use the environment variable READTHEDOCS_GIT_COMMIT_HASH instead to get this value.

Read the Docs embeded flyout replaced by new addons

We are moving away from the HTML blob injection we are currently doing to generate the flyout into a #readthedocs-embed-flyout selector. We already implemented Read the Docs Addons as a replacement to this approach which is modular and customizable.

The removal of the HTML blob injection means that if the theme were using an explicit selector to decide where to inject the flyout menu, it has to be migrated to listen to the readthedocs-addons-data-ready JavaScript event to get all the Read the Docs data in a JSON-like¹ object and generate the flyout as you prefer. Note that you can style and render the flyout as you prefer --even splitting it into different selectors, placement or integrate it more with your theme.

Example integrations using the JavaScript custom event:

• Read the Docs Sphinx theme kept the exact look&feel than before.
• CPython documentation doesn't render the flyout as-is but split it into version and language selectors at the top-left of the page.

You are free to integrate this data as you prefer, the only requirement here is to mention the documentation is hosted by Read the Docs somewhere in the page 😄

EthicalAds

If your theme is using a specific placement for the EthicalAd as described in our documentation via #ethical-ad-placement, it should keep working properly.

If that's not the case, Read the Docs will try to place the ad in the best placement possible based on some heuristic.

---

Hopefully, I was able to communicate this clearly enough 😅 . I'm happy to keep this issue open or open one issue per theme, depending on your needs, and I'm happy to collaborate with you all with this migration to the new Read the Docs Addons. Let me know if you have any doubt or if you want me to help you with anything related to this work 👍🏼

Footnotes

1. Unfortunately, not yet documented 😓, but you can find an example of it at https://docs.readthedocs.io/_/addons/?client-version=0.17.0&api-version=1&project-slug=docs&version-slug=stable ↩

[2bndy5]

I'm glad to see this getting finalized. I've been periodically checking on this progress, but I was hesitant to implement anything without a definitive date. 👍🏼

Is there a plan to document the "JSON-like" data? (possibly with a schema?)

[humitos]

Is there a plan to document the "JSON-like" data? (possibly with a schema?)

I have not started documenting this yet. However, we've implemented a way for theme author to pin to a particular version of the JSON-like data, so it won't change over time for that theme even if we keep updating this object for any reason. To pin the version we require the theme to define it via:

<meta name="readthedocs-addons-api-version" content="1">

I don't have too much experience with schemas, but I know that @stsewd have done something similar for our readthedocs.yaml file. So, I think we would follow something similar at some point --but we will definitely start with the documentation first.

[humitos]

@2bndy5 oh, by the way, there is a Material for MkDocs example I built that uses the new addons integration to build the version selector at the top (as Mike plugin does) that you can find at https://test-builds.readthedocs.io/en/mkdocs-material/ to check out how it looks in case you want to follow something similar.

[2bndy5]

I noticed that. I'm more inclined to replace/augment the repo info both in header bar and fly-out hamburger menu (on mobile).

My last attempt didn't go so well (using old approach).

[daltzctr]

What exactly is the purpose behind readthedocs_ext.readthedocs and should users start adding this dependency to their requirements and conf.py?

What exactly does the flyout injection? The extension or readthedocs addons?

Does this back-break older builds of documentation?

[humitos]

What exactly is the purpose behind readthedocs_ext.readthedocs and should users start adding this dependency to their requirements and conf.py?

The purpose of that extension was "to give Read the Docs a way to execute code at build time inside the Sphinx process to integrate better with the platform". This is not required anymore and users should not install this extension.

What exactly does the flyout injection? The extension or readthedocs addons?

Since we moved away from installing that extension and we don't have control on the Sphinx build process anymore, all the integration was migrated to JavaScript and happens on the client side now. The JavaScript file that contains all the addons is injected via a Cloudflare Worker when serving the documentation.

BTW, the repository for the addons is at https://github.com/readthedocs/addons, in case you want to chime in.

One of the goals for this was to make the platform being documentation agnostic. Now, Read the Docs supports any documentation tool and all of them have the same integrations in place 🎉

Does this back-break older builds of documentation?

It shouldn't, but if it does, please let us know and we will try to fix it.

You can test this integration by enabling the addons in your project and then hitting an old documentation version.

[humitos]

Hello everybody 👋🏼 -- I wanted to give an update regarding the work we've done in our Sphinx theme. We've integrated the version and language selectors into the navigation bar itself: below the documentation's title and above the search input.

You can see this live example at https://test-builds.readthedocs.io/en/latest/

We've included this feature in sphinx-rtd-theme=3.0.0rc2. Please, give it a try and let us know any feedback you may have. We plan to do the final 3.0.0 release on October 7th together with addons and Sphinx context changes.

If you are a theme developer and want to follow this approach, you can take a look at this PR readthedocs/sphinx_rtd_theme#1601 to use it as inspiration. Note that it uses Read the Docs Addons' CustomEvent, so your project has to have addons enabled to work.

[2bndy5]

Will that become part of stable channel? I assume this question was discussed in a separate thread. The nice thing about having the flyout menu incorporated into the nav menu was that the floating menu didn't obstruct the page's main body, but in 3.0.0-rc2 there's now a floating menu:

[humitos]

Let me clarify. There are different concepts here;

1. Read the Docs Addons flyout: it's shown floating at the bottom right. It's injected in all the pages served by Read the Docs and it's independent from the documentation tool used. It can be disabled from the project's settings page. It's not a Sphinx-specific feature.
2. Sphinx Read the Docs theme's re-implementation of the flyout: it's shown integrated in the navbar at the bottom left. It requires defining flyout_display="attached" in the Sphinx's conf.py. It's independent from the "Read the Docs Addons flyout" mentioned before as they could live together. It's disabled by default.
3. Sphinx Read the Docs theme's version/language selectors: they are integrated in the navbar and shown at the top left. It's independent from Read the Docs Addons flyout and also from the Sphinx theme's re-implementation. They are enabled by default and can be disabled by declaring version_selector=False and language_selector=False

With all of that, you can pick the combination that meets your needs best. If the Read the Docs Addons flyout covers some important content from your documentation project, you can disable it.

@2bndy5 Let me know if this clarifies what is what and how to interact with each of them.

[2bndy5]

It requires defining flyout_display="attached" in the Sphinx's conf.py

Is this supposed to go in the html_theme_options? I haven't been keeping up with python stuff lately (currently enthralled in rust). Is there a docs link (about accepted values and the default if not set)?

I wasn't aware of that complexity this feature introduces.

[humitos]

Is this supposed to go in the html_theme_options? Is there a docs link (about accepted values and the default if not set)?

Yes, it goes under html_theme_options. You can find the docs at https://sphinx-rtd-theme.readthedocs.io/en/latest/configuring.html

[2bndy5]

Thanks, that clarifies it for me indeed.

[mgeier]

Thanks for the heads-up!

1. Read the Docs Addons flyout: it's shown floating at the bottom right. It's injected in all the pages served by Read the Docs and it's independent from the documentation tool used. It can be disabled from the project's settings page. It's not a Sphinx-specific feature.

I'm confused ...

So there is no way for a Sphinx theme to move the flyout from the bottom right to somewhere else?

The only thing we can do is to create an additional place in our theme to show that information and to tell the users of our theme to please disable the bottom-right one in their project settings?

This seems very cumbersome ...

[r0qs]

Hi, I’ve read through this issue and the blog post about enabling RTD addons by default, but I’m still a bit unclear on one point. Will it still be possible to disable the addons via the RTD dashboard?

From my understanding, making the addons enabled by default, as mentioned in the blog post and this issue, does not necessarily imply that we won’t have the option to disable them. However, I came across this recent change (#11628), which completely removes the ability to disable addons. I’m curious about the reasoning behind this change, as the announcement only stated that addons would be enabled by default, without indicating that opting out would not be possible. Did I perhaps miss something?

Read the Docs embeded flyout replaced by new addons

We are moving away from the HTML blob injection we are currently doing to generate the flyout into a #readthedocs-embed-flyout selector. We already implemented Read the Docs Addons as a replacement to this approach which is modular and customizable.

The removal of the HTML blob injection means that if the theme were using an explicit selector to decide where to inject the flyout menu, it has to be migrated to listen to the readthedocs-addons-data-ready JavaScript event to get all the Read the Docs data in a JSON-like1 object and generate the flyout as you prefer. Note that you can style and render the flyout as you prefer --even splitting it into different selectors, placement or integrate it more with your theme.

Example integrations using the JavaScript custom event:

* [Read the Docs Sphinx theme](https://github.com/readthedocs/sphinx_rtd_theme/pull/1526) kept the exact look&feel than before.

* [CPython documentation](https://github.com/python/cpython/pull/116966) doesn't render the flyout as-is but split it into version and language selectors at the top-left of the page.

The reason I’m concerned is that our project has some custom DOM manipulations to adjust the placement of the flyout menu using CSS selectors, which I recently - and unfortunately - discovered in our code: https://github.com/ethereum/solidity/blob/develop/docs/_static/js/initialize.js#L25.

For newer versions of our documentation, I’ve implemented a workaround based on readthedocs/sphinx_rtd_theme#1526, but it doesn’t solve the issue for our older documentation versions, which depended on the .rst-versions element that was removed along with the embedded flyout. Thus, enabling addons breaks the layout of all released documentation.

So, my question is: what would be the recommended approach to address this issue for old documentation versions that relied on such HTML blob injection? For now, the only solution I see is to avoid enabling the addons, but it seems that this will also not be possible. Do you have any alternative suggestions?

[2bndy5]

It would be nice if certain add-ons could be disabled per RTD build. Currently, it only seems to have a global setting.

[humitos]

So there is no way for a Sphinx theme to move the flyout from the bottom right to somewhere else?
The only thing we can do is to create an additional place in our theme to show that information and to tell the users of our theme to please disable the bottom-right one in their project settings?

@mgeier Currently, there is no an "intermediate way to customize the flyout". We've been having this conversation for a long time but we haven't arrived at a place that works for everybody and that we can maintain over time. You can read more about this at readthedocs/addons#51. Feel free to provide any feedback you may have there.

[humitos]

Will it still be possible to disable the addons via the RTD dashboard?

@r0qs It's possible to disable each of the addons independently: flyout, search, etc. However, there is

what would be the recommended approach to address this issue for old documentation versions that relied on such HTML blob injection?

This is not a recommendation 😄 . However, I think you could pin those old versions to an old Read the Docs theme that injects the old flyout (eg. 2.0.0). I'm not 100% sure, but you may need to install https://github.com/readthedocs/sphinx-build-compatibility/ as well for that case. IIRC, the old theme makes usage of some of those variables this Sphinx extension adds -- but you can try without installing sphinx-build-compatibility first.

[humitos]

We've enabled Read the Docs Addons and we've removed the injected Sphinx context in all the projects already 🚀 . We've been testing it and answering questions during the whole day and we haven't noticed anything completely broken. We will stay taking a close look at your messages, so please, open new issues if you have doubts about this and/or if you find something is not working as you expected. You can use the following issue trackers depending on the specific issue:

• Read the Docs Addons: https://github.com/readthedocs/addons/issues
• Read the Docs Sphinx theme: https://github.com/readthedocs/sphinx_rtd_theme/issues
• Read the Docs builds and/or API: https://github.com/readthedocs/readthedocs.org/issues

We are happy to receive any feedback you may have. Feel free to contact us privately at https://app.readthedocs.org/support/ or open an issue if you want to provide public feedback. Thanks for all your support 🙏🏼

This is a big step for us and we've been trying hard to make it as smooth as possible for everybody.

Long live to Read the Docs Addons 👏🏼

[r0qs]

Will it still be possible to disable the addons via the RTD dashboard?

@r0qs It's possible to disable each of the addons independently: flyout, search, etc. However, there is

I think your comment got cut off. But I suppose your were about to say there is no way to disable addons, right?

what would be the recommended approach to address this issue for old documentation versions that relied on such HTML blob injection?

This is not a recommendation 😄 . However, I think you could pin those old versions to an old Read the Docs theme that injects the old flyout (eg. 2.0.0). I'm not 100% sure, but you may need to install https://github.com/readthedocs/sphinx-build-compatibility/ as well for that case. IIRC, the old theme makes usage of some of those variables this Sphinx extension adds -- but you can try without installing sphinx-build-compatibility first.

Thank you for your suggestion. I had a look on the compatibility package, but unfortunately, it doesn't seem to address our issue. Our scenario might be similar to other projects where modifying old release tags is not possible. In our case, each documentation version is directly tied to a specific code release, and all RTD configurations are stored within the repository itself. This means any updates to the repository will only apply to future releases, leaving the previously published documentation versions unchanged.

The issue we’re facing is due to the change that removed the embedded flyout - now replaced by addons - which has impacted not only new releases but also older ones. Our code was relying on certain embedded CSS selectors and did not handle cases where these selectors were not present. This led to type errors that break the layout; for example, if .rst-versions is not present, the code will attempt to call remove() on a null element. Although we’ve addressed this bug in our current code, this fix cannot be retroactively applied to already-released versions, which is the main concern.

If we had access to the static files served by RTD, we could potentially retroactively apply a patch to fix the affected files, but unfortunately, this is not an option either.

As an example, you can see the following links to our documentation:

• The documentation for our develop branch, which already includes our fix: https://docs.soliditylang.org/en/develop
• The broken documentation tied to our previous released version, v0.8.27: https://docs.soliditylang.org/en/v0.8.27 (you can see the error in the browser console: Uncaught TypeError: rstVersions is null).

Is there a way to configure RTD to map an existing version tag (e.g., v0.8.27) to a different branch that could includes the necessary patch (e.g., v0.8.27-rtd-fix)? Ideally, we’d like to continue displaying older tag versions in the flyout menu but have them render using a branch that contains the corrected layout. Does RTD support such a feature? Or do you know if there is an alternative approach to achieve this?

[humitos]

Is there a way to configure RTD to map an existing version tag (e.g., v0.8.27) to a different branch that could includes the necessary patch (e.g., v0.8.27-rtd-fix)? Ideally, we’d like to continue displaying older tag versions in the flyout menu but have them render using a branch that contains the corrected layout. Does RTD support such a feature? Or do you know if there is an alternative approach to achieve this?

@r0qs can you contact us via https://app.readthedocs.org/support/ so we can discuss and coordinate a potential solution? Thanks.

[victorlin]

I am facing a similar problem to @r0qs where our old builds were relying on RTD to add the flyout to the sidebar. Per recommendations in the blog post and sphinx-rtd-theme configuration page, I've added 'flyout_display': 'attached' and disabled the RTD addon flyout. However, this only allows newly built versions to render properly, while old versions will no longer have any flyout.

Example: Go to new build and use the flyout to switch to an older version. The older version no longer has a flyout.

[humitos]

@victorlin I understand. If you disable the flyout addons, it will be disabled in all the versions -- as you noticed already. We currently don't support enabling/disabling addons per version. It's a project's configuration.

I think if you want to use the flyout_display: attached in all your versions, you will need to re-build the old ones using the new version of the theme with this configuration enabled.

The other possible solution is to enable the addons flyout that works in all the versions, old and new ones.

[tsibley]

Hi @humitos, I work with @victorlin, so I'm continuing some conversation here.

We currently don't support enabling/disabling addons per version. It's a project's configuration.

We'd like old versions to look and act as they have for years, while also allowing future new versions to use the new addons framework. But, AFAICT, the limitation of project-global config instead of per-version config really hamstrings any solutions there.

… you will need to re-build the old ones using the new version of the theme with this configuration enabled.

Rebuilding dozens of old versions is likely not to go smoothly due to other changes that have happened in the meantime. It doesn't feel like a viable option.

The other possible solution is to enable the addons flyout that works in all the versions, old and new ones.

We don't want to do this because it's intrusive (e.g. routinely obstructs page text/images) and there's no way to customize it (see also).

[humitos]

@tsibley hi, 👋. Please, open a new issue so we can talk about your particular case. In this issue there are many people mentioned that will be notified otherwise.

The original goal of the issue was already addressed and that's why I closed it a few days ago. I'm happy to talk more about issues derived from this change/decision, but let's do it in separate threads.

[mgeier]

@mgeier Currently, there is no an "intermediate way to customize the flyout". We've been having this conversation for a long time but we haven't arrived at a place that works for everybody and that we can maintain over time.

So if there is no proper way to handle the flyout yet, why did you force it on us already?

make the transition as smooth as possible

So far it was not smooth at all.

To not overly extend this already closed issue, I've created a new one here: #11688

Don't get me wrong, I'm very thankful for the service you provided for many years and the help I got from all team members over the years, but this whole Addons thing made the platform less desirable for me. I'll still leave my projects there (if you let me), but for future projects I'll think twice.