Publish documentation for each stable branch + current development (master) #178

Cynical-Optimist · 2020-10-13T16:13:10Z

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Jan 3, 2018, 09:36

Currently we only publish docs to master, and gitlab still lacks proper support to deploy to a separate URL depending on which branch the CI is running from.

See upstream gitlab bug report.

Said upstream bug report suggests a workaround which might be plausible, which involves using the cache. The problem with this is if ever the cache gets deleted (which is supposed to be expected), we would have to manually trigger CI for every stable branch in history to regain the docs.

Cynical-Optimist · 2020-10-13T16:13:15Z

In GitLab by [Gitlab user @samthursfield] on Feb 12, 2018, 17:51

I had a quick look at readthedocs.org, which would in theory allow us to do this.

It's not possible to upload pre-generated documentation to that website, there is an open issue in their bug tracker about that:

The more pressing issue we need to take a stance on is what are we accepting via html upload. RTD doesn't aim to be just a file host -- you could output and host documentation almost anywhere on the internet. Our goal is making the documentation authoring and viewing experience great, with Sphinx being our primary focus. The documentation experience would go south if we just took any old html and hosted it. We don't have the resources or community support to improve the experience of al the various documentation output folks use.

So I had a go at setting up an automated build at readthedocs.org.

After adding the buildstream repo as a project, I had to add a requirements.txt file in a branch of buildstream.git so that readthedocs could install necessary dependencies in its build sandbox.

I then got a semi-working render up at https://buildstream.readthedocs.io/en/latest/ which had two major issues:

the 'invoking BuildStream' section (autogenerated from the frontend module using sphinx_click) was empty
all the plugin documentation (autogenerated using sphinx-apidoc driven by a Makefile in the docs/ directory) was missing

I don't know how to fix either of these things -- readthedocs doesn't let us run custom build commands, but that is how things currently work in the GitLab Pages build.

Cynical-Optimist · 2020-10-13T16:13:18Z

In GitLab by [Gitlab user @tristanvb] on Mar 22, 2018, 12:33

Alternative proposal: Just build every version we care about every time in CI, while still only deploying the pages job for master.

Maintain a list of branches, link names, and subdirectory also in the .gitlab-ci.yml directly
Update the .gitlab-ci.yml instructions to build docs, such that it performs a loop over the said lists:
- Checkout the latest of the specified buildstream branch in the list
- Build the documentation
- Move the built html documentation into the corresponding subdirectory
Finally, after building the docs, generate a very simple index.html in the main directory, based on the same list.

The documentation at buildstream.gitlab.io/buildstream would look like:

Select the version of BuildStream you want to read about:

* 1.0
* 1.1 (unstable)

Then you can browse the 1.0 documentation at: buildstream.gitlab.io/buildstream/1.0

And browse the current development version at: buildstream.gitlab.io/buildstream/unstable

Every 6 months, as a part of the stable release process, we need to update these lists in the .gitlab-ci.yml, this is relatively low effort.

Cynical-Optimist · 2020-10-13T16:13:20Z

In GitLab by [Gitlab user @jjardon] on Mar 27, 2018, 19:11

We could use this sphinx plugin to achieve this:

https://robpol86.github.io/sphinxcontrib-versioning/

Cynical-Optimist · 2020-10-13T16:13:23Z

In GitLab by [Gitlab user @tristanvb] on May 30, 2019, 09:15

mentioned in merge request website!121

Cynical-Optimist · 2020-10-13T16:13:26Z

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 11, 2019, 13:38

Noting an irc chat on this, revolving around the need to separate the install guide for end users and the install guide for hackers. Ideal end goal is to have all docs under one url, with master under something like 'latest' and then specific versions under their own etc

Cynical-Optimist · 2020-10-13T16:13:29Z

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 14, 2019, 15:06

[Gitlab user @tristanvb] I just went to the upstream bug and it's been scheduled into a milestone so I think gitlab will start work on it soon. Someone there comments:

Now would be a great time to review the description and make sure use cases you care about are covered.

Cynical-Optimist · 2020-10-13T16:13:32Z

In GitLab by [Gitlab user @tristanvb] on Jun 26, 2019, 10:52

marked this issue as related to #1056

Cynical-Optimist · 2020-10-13T16:13:34Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 15, 2019, 16:39

Hey everyone, [Gitlab user @LaurenceUrhegyi] said this might be a good starting point for me.
I've read the comments and the IRC discussion but it's not very clear to me what the target of the issue is.

What we want is documentation and installation guide for master and stable-1.2 respectively (not all the tags on the repository, right?), preferably with a nice way of choosing between the two on https://docs.buildstream.build/ ?

I understand that ideally we'd want these to be generated depending on the current branch, although I'm not sure if that's feasible? (or worth working on right now as gitlab is apparently picking it up?).

Cynical-Optimist · 2020-10-13T16:13:37Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 16, 2019, 17:13

After [Gitlab user @tristanvb] explained to me today what this issue is about, I tried to understand how docs-website currently does it and tried to do something similar.

I've created a small project here so I can make my own changes and see them after running the pipeline. I've managed to set up the CI so that it fetches the documentation for master, 1.2.8 and 1.0.0 (this one I got by looking at the tags, cherry picking the commit for the tag I'm looking for, going to the commit and getting the docs download link. Not sure if that's okay, but it seemed it's alright since the documentation for 1.0.0 is different and seems to be the right one.)

For now, the results are here, a small list which leads to different versions' documentations (which looks awful since I only included the index.html, but it'll need a new design anyway, I believe).

I've a couple of things I'd need a second opinion on: the new page, where we'd include the list with all the releases' documentation, needs a new look.I think we might want to include that About section in there as well, since you'd just see it on every version's own page anyway. On the other hand, since I fetch every artifact and do some setup for each (unzipping, moving files etc.), the CI file might end up being unnecessarily too large, using something like a dictionary for the versions we're listing might be useful, if that's a possibility in the gitlab CI? I believe adding new entries for subsequent releases should also be easy, since you'd just need to copy 2 lines and change some values.

LE: Or keeping one job per version would make sense? We could have one stage for fetching the documentation and setting everything up and a second one for publishing it, so that the former can run in parallel?

Cynical-Optimist · 2020-10-13T16:13:40Z

In GitLab by [Gitlab user @BenjaminSchubert] on Jul 17, 2019, 10:44

Hey, just one note, on readthedocs (which we do not use, but most python projects host their documentation there), they have a nice little widget to change versions (see https://mypy.readthedocs.io/en/latest/ for example, bottom left). I wonder if that would be doable, and if it is natively supported by sphinx or is a readthedocs-only extension.

What would you think of that? Or is that something that got ruled out already?

Cynical-Optimist · 2020-10-13T16:13:43Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 17, 2019, 10:49

Hey [Gitlab user @BenjaminSchubert] not sure if this has already been mentioned. I think it's doable as long as it's nothing really difficult to do in raw HTML. However, we were planning to have more versions there, similar to how glib does it, from my understanding, so that widget might not look too good when there's a list of 20 versions there, unlike Mypy, which only has 2 right now.

Cynical-Optimist · 2020-10-13T16:13:45Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 18, 2019, 13:26

I managed to automate fetching the documentation and setting up the file hierarchy so all versions work fine (except for 1.1.7 for now, if that's relevant). It should also work with new tags introduced, I believe.

In the meantime, I removed all sections other than "About" on the main page, since it should only be a menu towards the documentation for the releases. I'd like it if the other pages didn't include that "About" sections since it's pretty much the same, but I don't think that's easily doable considering we're just fetching precomputed artifacts.

Results can be seen here with sources being available in this repository.

Cynical-Optimist · 2020-10-13T16:13:49Z

In GitLab by [Gitlab user @tristanvb] on Jul 19, 2019, 09:26

This looks like a great start.

I was thinking before that it would make sense to only display the latest micro point release for every major point (e.g. only display 1.2.8, but not 1.2.0 and all of the releases in between which would anyway have exactly the same API), but perhaps this doesnt really matter, and maybe there is a value to displaying them all which I am missing.

That said, I think that this page is missing some indication of "What is the latest stable and what is the latest development snapshot", or it can be difficult for people figure out which documentation they want to read.

For consistency, I think we should display the badges in this page, this can be done with the following HTML:

<object style="vertical-align: middle" data="https://buildstream.gitlab.io/buildstream/_static/release.svg" type="image/svg+xml">
(your browser does not support SVG, please find releases at [https://download.gnome.org/sources/BuildStream/](https://download.gnome.org/sources/BuildStream/).
</object>
<object style="vertical-align: middle" data="https://buildstream.gitlab.io/buildstream/_static/snapshot.svg" type="image/svg+xml">
(your browser does not support SVG, please find releases at [https://download.gnome.org/sources/BuildStream/](https://download.gnome.org/sources/BuildStream/).
</object>

Maybe in this case instead of relying on the svgs to provide the links (the badges link to their respective release tarballs), we could make the badges link directly to their respective documentation (by surrounding the badges in a <a href="link to docs"> ... </a> tag)

Some other enhancements can of course be made, I would like for instance to separate "stable releases" and "development snapshots + master" in separate columns, to make things still more clear, but I don't think that is important enough to block landing this initially.

I think at this point we should move your private repo into a merge request against the https://gitlab.com/BuildStream/docs-website repo, continue this conversation there and hopefully merge very soon.

Cynical-Optimist · 2020-10-13T16:13:52Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 19, 2019, 14:47

mentioned in commit docs-website[Gitlab user @dc49814e6babacc3f6b6e15e9cd26dcf4eb462d5]

Cynical-Optimist · 2020-10-13T16:13:55Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 19, 2019, 14:47

mentioned in merge request docs-website!2

Cynical-Optimist · 2020-10-13T16:13:58Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 19, 2019, 14:52

mentioned in commit docs-website[Gitlab user @296fc5861876fd1957091a3b6558d436770adbde]

Cynical-Optimist · 2020-10-13T16:14:01Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 22, 2019, 09:17

mentioned in commit docs-website[Gitlab user @d99c106e4f1601aae4d2e3b492c8f20991347493]

Cynical-Optimist · 2020-10-13T16:14:04Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 22, 2019, 09:36

mentioned in commit docs-website[Gitlab user @33ffc30353cdb8ae56fffe561c977d3fe613e5de]

Cynical-Optimist · 2020-10-13T16:14:07Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 22, 2019, 09:39

mentioned in commit docs-website[Gitlab user @292e15e236ab75120008ce5723cdb36855bc5ff8]

Cynical-Optimist · 2020-10-13T16:14:10Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 22, 2019, 09:42

mentioned in commit docs-website[Gitlab user @f05f93d75e715dfca17f78564ca6dc87c580d83a]

Cynical-Optimist · 2020-10-13T16:14:12Z

In GitLab by [Gitlab user @AlexFazakas] on Jul 22, 2019, 10:12

[Gitlab user @tristanvb] So I've implemented most of the things you mentioned (I hope) and opened a MR with everything I've done here.

One thing I didn't manage to do was changing what the badges point to. I tried surrounding them with <a class="reference internal" href="link/to/doc"> ... </a> but the badges still pointed to the same resource found in the data field.

Other than that, I think it's ready for a quick review.

LE: Managed to make the badges work by no longer using the <object ...> structure. I replicated the way we do those in main_about.html using internal href's to the documentation.