Monodocs sphinx build #4347

cosmicBboy · 2023-11-01T17:43:35Z

Tracking issue

Describe your changes

This PR adds a custom sphinx build system that imports flytekit, flytesnacks, flytectl, and flyteidl documentation and builds the flyte docs as a single docs site. It introduces the following:

An auto_examples.py sphinx extension to convert flytesnacks examples into myst markdown. This extension also adds convenience directives to render toctree and table of contents in the flytesnacks examples.
An import_projects.py sphinx extension that handles cloning, generating, and rendering docs from imported projects, either from git or from local (e.g. flyteidl)

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Signed-off-by: Niels Bantilan <[email protected]>

thomasjpfan

I can see this done in two stages, which can help with reviewing.

Move flyteidl's docs into this repo. I think this is a majority of this PR.
Move the rest, which I suspect will be a smaller follow up PR.

For step 2, moving flytectl and flytekit looks okay to me. I'm concerned over flytesnacks, because of how many dependencies it adds to building the docs.

doc-requirements.in

Signed-off-by: Niels Bantilan <[email protected]>

pingsutw · 2023-11-22T04:36:19Z

Shoutout to @cosmicBboy. I really like the new doc. Thanks for your tremendous work.

Signed-off-by: Niels Bantilan <[email protected]>

codecov · 2023-11-22T21:37:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c049865) 58.89% compared to head (44e6728) 58.90%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4347      +/-   ##
==========================================
+ Coverage   58.89%   58.90%   +0.01%     
==========================================
  Files         620      620              
  Lines       52440    52458      +18     
==========================================
+ Hits        30883    30900      +17     
- Misses      19090    19091       +1     
  Partials     2467     2467

Flag	Coverage Δ
unittests	`58.90% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: cosmicBboy <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: cosmicBboy <[email protected]>

thomasjpfan

Overall, comparing master's build with this PR's build, the build time went from 100 seconds on master to 448 seconds on this PR. I think that is okay.

docs/.readthedocs.yaml

common-requirements.in

thomasjpfan · 2023-12-04T16:02:35Z

.readthedocs.yml

@@ -16,4 +16,4 @@ sphinx:
 # Optionally set the version of Python and requirements required to build your docs
 python:
  install:
-    - requirements: doc-requirements.txt
+    - requirements: doc-requirements.in


I think using the pinned version .txt is safer for doc builds. Otherwise, we can have PRs that break from unrelated issues.

currently this does not work, as pip-compile will add tensorflow-macos to the doc-requirements.txt file when I run it locally, which will not work on readthedocs build.

planning on using conda here

Currently, I do not see tensorflow-macos in doc-requirements.txt. Is this change still required?

planning on using conda here

Yes, Conda-lock will resolve this because it can lock a linux environment from a macos environment.

In any case, I am okay with merging this as is and fixing it later if doc builds become flaky.

docs/.readthedocs.yaml

- add graphviz as conda dependency - remove monodocs-requirements.{in,txt} Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>

thomasjpfan

Looks like the merge commits are not signed.

Otherwise, I am okay with the current PR. We can always adjust with follow up PRs.

thomasjpfan · 2023-12-05T23:54:39Z

.github/workflows/tests.yml

+          channels: conda-forge
+          channel-priority: true
+          activate-environment: monodocs-env
+          environment-file: monodocs-environment.yaml


For a follow up, I prefer to lock this environment as well. conda-incubator/setup-miniconda supports lock files: https://github.com/conda-incubator/setup-miniconda#example-7-lockfiles

gonna followup on this with a PR

thomasjpfan · 2023-12-05T23:56:52Z

.github/workflows/tests.yml

        with:
-          python-version: "3.8"
-      - name: Install dependencies
+          python-version: 3.9


Nit: The python version is already in monodocs-environment.yaml. Including it here will "override" the version in the environment yaml, so I think it's better to leave it out here:

Suggested change

python-version: 3.9

thomasjpfan · 2023-12-06T00:02:22Z

.readthedocs.yml

@@ -16,4 +16,4 @@ sphinx:
 # Optionally set the version of Python and requirements required to build your docs
 python:
  install:
-    - requirements: doc-requirements.txt
+    - requirements: doc-requirements.in


Currently, I do not see tensorflow-macos in doc-requirements.txt. Is this change still required?

planning on using conda here

Yes, Conda-lock will resolve this because it can lock a linux environment from a macos environment.

In any case, I am okay with merging this as is and fixing it later if doc builds become flaky.

samhita-alla

lgtm! let's merge the PR. 🚀

* rename rsts to docs Signed-off-by: Niels Bantilan <[email protected]> * update docs reqs Signed-off-by: Niels Bantilan <[email protected]> * update reqs, styles Signed-off-by: Niels Bantilan <[email protected]> * update doc requirements Signed-off-by: Niels Bantilan <[email protected]> * edit gitignore Signed-off-by: Niels Bantilan <[email protected]> * update doc requirement Signed-off-by: Niels Bantilan <[email protected]> * update doc requirements Signed-off-by: Niels Bantilan <[email protected]> * update doc requirements Signed-off-by: Niels Bantilan <[email protected]> * minor clean up Signed-off-by: Niels Bantilan <[email protected]> * update deps Signed-off-by: Niels Bantilan <[email protected]> * use doc-requirements.in in .readthedocs.yml Signed-off-by: Niels Bantilan <[email protected]> * updates Signed-off-by: Niels Bantilan <[email protected]> * check in new docs Signed-off-by: Niels Bantilan <[email protected]> * clean up most reference warnings Signed-off-by: Niels Bantilan <[email protected]> * fix ref links Signed-off-by: Niels Bantilan <[email protected]> * update auto examples Signed-off-by: Niels Bantilan <[email protected]> * add logic to find/replace invalid links Signed-off-by: Niels Bantilan <[email protected]> * update docs with changes Signed-off-by: Niels Bantilan <[email protected]> * clean up toc list for deployment docs Signed-off-by: Niels Bantilan <[email protected]> * add back rsts Signed-off-by: Niels Bantilan <[email protected]> * add readthedocs config for monodocs Signed-off-by: Niels Bantilan <[email protected]> * incorporate changes from master Signed-off-by: Niels Bantilan <[email protected]> * reformat concepts section Signed-off-by: Niels Bantilan <[email protected]> * move monodocs readthedocs config Signed-off-by: Niels Bantilan <[email protected]> * debug readthedocs config Signed-off-by: Niels Bantilan <[email protected]> * add monodocs requirements Signed-off-by: Niels Bantilan <[email protected]> * update reqs Signed-off-by: Niels Bantilan <[email protected]> * fix furo dep Signed-off-by: Niels Bantilan <[email protected]> * update logo Signed-off-by: Niels Bantilan <[email protected]> * update logo Signed-off-by: Niels Bantilan <[email protected]> * update css Signed-off-by: Niels Bantilan <[email protected]> * update css Signed-off-by: Niels Bantilan <[email protected]> * update sidebar caption styling Signed-off-by: Niels Bantilan <[email protected]> * update sidebar css * use conda to install monodocs deps Signed-off-by: cosmicBboy <[email protected]> * update conda deps Signed-off-by: cosmicBboy <[email protected]> * update makefile docs target Signed-off-by: cosmicBboy <[email protected]> * add furo dep to environment.yaml Signed-off-by: cosmicBboy <[email protected]> * add retry dep Signed-off-by: cosmicBboy <[email protected]> * update vaex deps Signed-off-by: cosmicBboy <[email protected]> * add graphviz to ci Signed-off-by: cosmicBboy <[email protected]> * add graphviz to readthedocs config Signed-off-by: cosmicBboy <[email protected]> * support pointing to local flytekit, flytesnacks, flytectl - add graphviz as conda dependency - remove monodocs-requirements.{in,txt} Signed-off-by: Niels Bantilan <[email protected]> * delete doc-requirements.txt - defer cleaning this up to followup PR Signed-off-by: Niels Bantilan <[email protected]> * revert doc-requirements.txt Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: cosmicBboy <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Signed-off-by: Paul Dittamo <[email protected]>

cosmicBboy added 9 commits October 26, 2023 15:37

rename rsts to docs

f29663b

Signed-off-by: Niels Bantilan <[email protected]>

update docs reqs

972ed62

Signed-off-by: Niels Bantilan <[email protected]>

update reqs, styles

fe45f3d

Signed-off-by: Niels Bantilan <[email protected]>

update doc requirements

b160f75

Signed-off-by: Niels Bantilan <[email protected]>

edit gitignore

9a281b4

Signed-off-by: Niels Bantilan <[email protected]>

update doc requirement

83db5ba

Signed-off-by: Niels Bantilan <[email protected]>

update doc requirements

3c0af54

Signed-off-by: Niels Bantilan <[email protected]>

update doc requirements

98b4191

Signed-off-by: Niels Bantilan <[email protected]>

minor clean up

9dccd8f

Signed-off-by: Niels Bantilan <[email protected]>

cosmicBboy changed the title ~~Monodocs~~ Monodocs sphinx build Nov 1, 2023

cosmicBboy added 5 commits November 1, 2023 13:48

update deps

027797c

Signed-off-by: Niels Bantilan <[email protected]>

use doc-requirements.in in .readthedocs.yml

a29d3c6

Signed-off-by: Niels Bantilan <[email protected]>

updates

b641337

Signed-off-by: Niels Bantilan <[email protected]>

check in new docs

88b734b

Signed-off-by: Niels Bantilan <[email protected]>

clean up most reference warnings

08b0f10

Signed-off-by: Niels Bantilan <[email protected]>

thomasjpfan mentioned this pull request Nov 6, 2023

[Docs] Monodocs build - a build system to consolidate Flyte docs #4329

Closed

2 tasks

thomasjpfan reviewed Nov 6, 2023

View reviewed changes

doc-requirements.in Outdated Show resolved Hide resolved

cosmicBboy added 2 commits November 13, 2023 17:36

fix ref links

39e3fbf

Signed-off-by: Niels Bantilan <[email protected]>

update auto examples

f299f64

Signed-off-by: Niels Bantilan <[email protected]>

neverett mentioned this pull request Nov 17, 2023

Revamp onboarding docs flyteorg/flytesnacks#1266

Merged

9 tasks

cosmicBboy added 2 commits November 21, 2023 16:16

add logic to find/replace invalid links

013f314

Signed-off-by: Niels Bantilan <[email protected]>

update docs with changes

35e231f

Signed-off-by: Niels Bantilan <[email protected]>

cosmicBboy added 5 commits November 22, 2023 15:47

clean up toc list for deployment docs

21afc4c

Signed-off-by: Niels Bantilan <[email protected]>

add back rsts

a010129

Signed-off-by: Niels Bantilan <[email protected]>

add readthedocs config for monodocs

7da4937

Signed-off-by: Niels Bantilan <[email protected]>

incorporate changes from master

96110f2

Signed-off-by: Niels Bantilan <[email protected]>

reformat concepts section

6ea6738

Signed-off-by: Niels Bantilan <[email protected]>

move monodocs readthedocs config

cd1475e

Signed-off-by: Niels Bantilan <[email protected]>

cosmicBboy added 3 commits November 22, 2023 21:05

update sidebar caption styling

f62247e

Signed-off-by: Niels Bantilan <[email protected]>

update sidebar css

70e46ea

use conda to install monodocs deps

ebb82b4

Signed-off-by: cosmicBboy <[email protected]>

dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Dec 1, 2023

cosmicBboy added 8 commits December 1, 2023 16:36

Merge branch 'master' into monodocs

81c5e49

Signed-off-by: Niels Bantilan <[email protected]>

update conda deps

7573375

Signed-off-by: cosmicBboy <[email protected]>

update makefile docs target

80d4f5a

Signed-off-by: cosmicBboy <[email protected]>

add furo dep to environment.yaml

9cfcc6d

Signed-off-by: cosmicBboy <[email protected]>

add retry dep

c538b3e

Signed-off-by: cosmicBboy <[email protected]>

update vaex deps

e4db528

Signed-off-by: cosmicBboy <[email protected]>

add graphviz to ci

28f36a1

Signed-off-by: cosmicBboy <[email protected]>

add graphviz to readthedocs config

cf1ed7c

Signed-off-by: cosmicBboy <[email protected]>

thomasjpfan reviewed Dec 4, 2023

View reviewed changes

cosmicBboy added 6 commits December 4, 2023 12:14

Merge branch 'master' into monodocs

4d98504

support pointing to local flytekit, flytesnacks, flytectl

7207336

- add graphviz as conda dependency - remove monodocs-requirements.{in,txt} Signed-off-by: Niels Bantilan <[email protected]>

delete doc-requirements.txt - defer cleaning this up to followup PR

b3ee93b

Signed-off-by: Niels Bantilan <[email protected]>

revert doc-requirements.txt

0c0780e

Signed-off-by: Niels Bantilan <[email protected]>

Merge branch 'master' into monodocs

c4b9d8b

Merge branch 'master' into monodocs

c9c5d15

thomasjpfan approved these changes Dec 6, 2023

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 6, 2023

Merge branch 'master' into monodocs

142e058

samhita-alla approved these changes Dec 6, 2023

View reviewed changes

cosmicBboy enabled auto-merge (squash) December 6, 2023 15:10

Merge branch 'master' into monodocs

44e6728

samhita-alla approved these changes Dec 6, 2023

View reviewed changes

cosmicBboy merged commit aff1150 into master Dec 6, 2023
40 of 41 checks passed

cosmicBboy deleted the monodocs branch December 6, 2023 15:26

dosubot bot mentioned this pull request Jan 8, 2024

[Docs] Switch Flyte docs to monodocs build #4686

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monodocs sphinx build #4347

Monodocs sphinx build #4347

cosmicBboy commented Nov 1, 2023

thomasjpfan left a comment

pingsutw commented Nov 22, 2023

codecov bot commented Nov 22, 2023 •

edited

Loading

thomasjpfan left a comment

thomasjpfan Dec 4, 2023

cosmicBboy Dec 4, 2023 •

edited

Loading

thomasjpfan Dec 6, 2023

thomasjpfan left a comment

thomasjpfan Dec 5, 2023

cosmicBboy Dec 6, 2023

thomasjpfan Dec 5, 2023

thomasjpfan Dec 6, 2023

samhita-alla left a comment

Monodocs sphinx build #4347

Monodocs sphinx build #4347

Conversation

cosmicBboy commented Nov 1, 2023

Tracking issue

Describe your changes

Check all the applicable boxes

thomasjpfan left a comment

Choose a reason for hiding this comment

pingsutw commented Nov 22, 2023

codecov bot commented Nov 22, 2023 • edited Loading

Codecov Report

thomasjpfan left a comment

Choose a reason for hiding this comment

thomasjpfan Dec 4, 2023

Choose a reason for hiding this comment

cosmicBboy Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

thomasjpfan Dec 6, 2023

Choose a reason for hiding this comment

thomasjpfan left a comment

Choose a reason for hiding this comment

thomasjpfan Dec 5, 2023

Choose a reason for hiding this comment

cosmicBboy Dec 6, 2023

Choose a reason for hiding this comment

thomasjpfan Dec 5, 2023

Choose a reason for hiding this comment

thomasjpfan Dec 6, 2023

Choose a reason for hiding this comment

samhita-alla left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 22, 2023 •

edited

Loading

cosmicBboy Dec 4, 2023 •

edited

Loading