Build: "automagic" builder is broken for some Python versions

[humitos]

Read the Docs executes some code in the build process on behalf the user to facilitate the on boarding process. This means that you can just put some .rst or .md files in your repository and Read the Docs will create a nice documentation site for you without requiring any type of configuration.

This is a nice feature for new users without too much knowledge on Sphinx or MkDocs. However, it has bitten us and also users due that sometimes there are unexpected errors that are hard to debug due to a disparity between Read the Docs and a local build for the same exact commit.

That said, this "automagic" builder is currently broken 😞 . If you follow the example that I mentioned (just put some .rst files in your repository without a conf.py file), Read the Docs will create one for you. Well, that conf.py generated by Read the Docs contains a problematic line:

from recommonmark.parser import CommonMarkParser

that fails with the following error:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/envs/no-conf-py/lib/python3.9/site-packages/sphinx/config.py", line 347, in eval_config_file
    exec(code, namespace)
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/checkouts/no-conf-py/docs/conf.py", line 7, in <module>
    from recommonmark.parser import CommonMarkParser
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/envs/no-conf-py/lib/python3.9/site-packages/recommonmark/parser.py", line 9, in <module>
    from commonmark import Parser
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/envs/no-conf-py/lib/python3.9/site-packages/commonmark/__init__.py", line 4, in <module>
    from commonmark.main import commonmark
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/envs/no-conf-py/lib/python3.9/site-packages/commonmark/main.py", line 14, in <module>
    from commonmark.blocks import Parser
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/envs/no-conf-py/lib/python3.9/site-packages/commonmark/blocks.py", line 5, in <module>
    from commonmark import common
  File "/home/docs/checkouts/readthedocs.org/user_builds/test-builds/envs/no-conf-py/lib/python3.9/site-packages/commonmark/common.py", line 14, in <module>
    HTMLunescape = html.parser.HTMLParser().unescape
AttributeError: 'HTMLParser' object has no attribute 'unescape'

• Build URL with a test case: https://readthedocs.org/projects/test-builds/builds/17932265/
• conf.py.tmpl used to auto-generate the file: https://github.com/readthedocs/readthedocs.org/blob/1fd6a668ef8b9a9638473b4f6813484065b120bd/readthedocs/templates/sphinx/conf.py.conf
• Failing line on commonmark package: https://github.com/readthedocs/commonmark.py/blob/0.8.1/commonmark/common.py#L14

Note that recommonmark has been deprecated since May 2021 (readthedocs/commonmark.py#308) and its repository got archived. So, I don't think this issue will be fixed.

Moving forward

There are different alternatives that we can follow here to workaround and/or fix the problem:

0. Force installing commonmark==0.9.1

• currently, 0.8.1 is being installed by default on Read the Docs
• 0.9.1 does not have this problem (see https://readthedocs.org/projects/test-builds/builds/17932517/)
• pip install on Read the Docs:

  readthedocs.org/readthedocs/doc_builder/python_environments.py

  Line 185 in 1fd6a66

  "commonmark==0.8.1",

• this is the quickest and easiest solution that will allow us to keep this functionality around

1. Replace recommonmark by MyST in the conf.py.tmpl

• this solution would allow us to keep the functionality around but using better and supported tools
• we should check compatibility with old versions of Sphinx (does MyST work with Sphinx 1.8?)
• remove recommonmark from Python packages installed and install myst-parser instead
• this will have an impact on all the builds, even if they don't currently use recommonmark and/or myst-parser

2. Remove the auto-generated conf.py functionality

This is the solution I'd like to discuss a little more. IMO, it seems that not many users are using this feature currently. I took a quick look at the logs (https://onenr.io/0LwG6A5yvQ6) and I found some usage there, but not a ton. It seems the currently is based on the Python version used: it breaks with 3.9 but it works with 3.7.

I parsed the logs manually with grep on util01 and I found that 2543 projects used this feature in the last 90 days. However, I don't know how many of them had successful builds 😄 or if they are spam or not. We could get better data if we spend some more time on this, tho. On Read the Docs for Business, we have 135 projects using this feature in the last 90 days --which is probably where this feature is more useful, if any.

We have been talking about reducing the magic that Read the Docs does on behalf the user, and I think this is a good opportunity to remove this dark magic that produces unexpected errors for users on builds. Besides, we also talked about educating our users suggesting them to use a config file, a requirements file and pin their dependencies properly following our https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html

I think it's good to have a nice and quick on boarding experience. However, I think we should work on making the "the proper and recommended way" (reproducible builds) as part of the on boarding experience, making it simple, nice and quick. Instead of on boarding users in a dirty way (magic things) and then tell them that's not a good way to use our platform.

Note that yesterday I received a support request about this and I thought the user was using recommonmark and I told them to use something newer like MyST. Then the user replied that they are not using recommonmark and they don't know how that got installed. Turned out that we are installing it and configuring it on behalf the user 🙃 . I was so confused.

[humitos]

I parsed the logs manually with grep on util01 and I found that 2543 projects used this feature in the last 90 days. However, I don't know how many of them had successful builds smile or if they are spam or not. We could get better data if we spend some more time on this, tho. On Read the Docs for Business, we have 135 projects using this feature in the last 90 days --which is probably where this feature is more useful, if any.

I ended up using New Relic for this, which I think is more trust-able and accurate than my grepping, 😅

Projects in the last 3 months where Read the Docs auto-created the Sphinx/MkDocs configuration file magically:

• Read the Docs Community
  • Sphinx: 1.5k : https://onenr.io/0VRV6bZz6Ra
  • MkDocs: 152: https://onenr.io/0dQe6v5KZwe
• Read the Docs for Business
  • Sphinx: 206: https://onenr.io/08jqnvyNbQl
  • MkDocs: 15: https://onenr.io/0LREWKZNMQa

Note that this includes spam projects and it does not differentiate failed from successful builds. The real number of project effectively using this feature is lower than these ones.

[humitos]

We implemented "0. Force installing commonmark==0.9.1" as a workaround for now in #9563

[agjohnson]

I agree that option number 2 might be the best overall way to address this one. When I've run into builds that used this feature, the project always falls into one of these categories:

• A project from a new user, who is very lost
• A project squatting on a namespace
• Spam project

Obviously, we shouldn't care about squatting or spam projects. For new users though, we now have much better resources, and this automatic content feature doesn't really serve much purpose.

With better educational resources like example projects, the experience of misconfiguring a project would be much nicer if we were explicit about how to proceed. Masking misconfiguration issues by generating output for the user doesn't help them solve their problem, where giving a build failure would almost certainly help the user more.

If the user really just wants to see how Read the Docs works, by not giving us input, we should point them to using an example project first.

[ericholscher]

Yea, 👍 on removing the conf.py generation. I don't think that's meaningfully helping anyone. I'd love to combine this with a nicer onboarding and error messages in this case though, if possible. It would be nice to at least link these users to our tutorial.

[humitos]

OK. I think we are all on board about moving forward with 2. Remove the auto-generated conf.py functionality. I'm marking this issue as "Accepted" and putting it under the "Roadmap" project so we prioritize in the following weeks/months.

It seems the required work would be:

• use the data from NR combined to our database to know what are the exact projects using this feature (non-spam project with successful builds in last 6 months)
• follow the not-yet-defined process to deprecate this feature
• remove the auto-generated conf.py functionality from our codebase
• make the build error pretty clear about what happened
• point those users to the tutorial

Feel free to edit this list to add anything that I may be missing here.

[humitos]

I hope our situation regarding this issue will improve a lot due to the enforcing of a YAML config file we are doing at https://blog.readthedocs.com/migrate-configuration-v2/. I'm adding this issue to Q4, so we can review the situation at that point after the migrations "is done".

[humitos]

remove the auto-generated conf.py functionality from our codebase
make the build error pretty clear about what happened

This is already happening at #2483 and readthedocs/blog#229