Using RST for documentation generation (flyteorg#169)

* Fix the links in generated html files * Added handler for build finished event to fix doc links * Added contributing guide * Adding wait for the subprocess * Fixed the path issue with finding the script for fixing the links * Added darwin switch for sed -i flag * Fixed non darwin with -i flag for sed * Using find instead of xargs as it fails on readthedocs * Moving the substitution errors to /dev/null * Escaped the Link values Signed-off-by: Prafulla Mahindrakar <[email protected]> * Using protoc-gen-doc plugin directly Signed-off-by: Prafulla Mahindrakar <[email protected]> * Removed the example message Signed-off-by: Prafulla Mahindrakar <[email protected]> * Fixed datacatalog file generation Signed-off-by: Prafulla Mahindrakar <[email protected]> * Using template to generate docs and also removed unused imports in proto files and fixed doc issues Signed-off-by: Prafulla Mahindrakar <[email protected]> * Minor doc fix Signed-off-by: Prafulla Mahindrakar <[email protected]> * Added dependency to protoc-gen in biolerplate module Signed-off-by: Prafulla Mahindrakar <[email protected]>
eapolinario · May 24, 2021 · 08a958c · 08a958c
1 parent 814493e
commit 08a958c
Show file tree

Hide file tree

Showing 85 changed files with 9,721 additions and 7,964 deletions.
diff --git a/Makefile b/Makefile
@@ -41,19 +41,7 @@ install-piptools:
 .PHONY: doc_gen_deps # these dependencies are required by protoc gen doc for the protos which have external library dependencies. 
 # which includes grpc-gateway, googleapis, k8.io/api and apimachinery, protocolbuffers 
 doc_gen_deps:
-	rm -rf tmp/doc_gen_deps tmp/protocolbuffers tmp/grpc-gateway tmp/k8s.io
-	git clone --depth 1 https://github.com/googleapis/googleapis tmp/doc_gen_deps
-	rm -rf tmp/doc_gen_deps/.git
-	git clone --depth 1 https://github.com/protocolbuffers/protobuf tmp/protocolbuffers
-	cp -r tmp/protocolbuffers/src/* tmp/doc_gen_deps/
-	rm -rf tmp/protocolbuffers
-	git -c advice.detachedHead=false clone --depth 1 --branch v1.15.2 https://github.com/grpc-ecosystem/grpc-gateway tmp/grpc-gateway #v1.15.2 is used to keep the grpc-gateway version in sync with generated protos which is using the LYFT image
-	cp -r tmp/grpc-gateway/protoc-gen-swagger  tmp/doc_gen_deps/
-	rm -rf tmp/grpc-gateway
-	git clone --depth 1 https://github.com/kubernetes/api tmp/k8s.io/api
-	git clone --depth 1 https://github.com/kubernetes/apimachinery tmp/k8s.io/apimachinery
-	cp -r tmp/k8s.io tmp/doc_gen_deps/
-	rm -rf tmp/k8s.io
+	./scripts/doc_gen_deps.sh
 
 .PHONY: doc-requirements.txt
 doc-requirements.txt: doc-requirements.in install-piptools

diff --git a/README.rst b/README.rst
diff --git a/boilerplate/lyft/golang_support_tools/go.mod b/boilerplate/lyft/golang_support_tools/go.mod
@@ -6,7 +6,10 @@ require (
 	github.com/alvaroloes/enumer v1.1.2
 	github.com/golangci/golangci-lint v1.22.2
 	github.com/lyft/flytestdlib v0.2.31
+	github.com/pseudomuto/protoc-gen-doc v1.4.1
 	github.com/vektra/mockery v0.0.0-20181123154057-e78b021dcbb5
 )
 
 replace github.com/vektra/mockery => github.com/enghabu/mockery v0.0.0-20191009061720-9d0c8670c2f0
+
+replace github.com/pseudomuto/protoc-gen-doc => github.com/paweld2/protoc-gen-doc v1.4.2-0.20210329170919-e6b3293482d4
diff --git a/boilerplate/lyft/golang_support_tools/go.sum b/boilerplate/lyft/golang_support_tools/go.sum
diff --git a/boilerplate/lyft/golang_support_tools/tools.go b/boilerplate/lyft/golang_support_tools/tools.go
@@ -7,4 +7,5 @@ import (
 	_ "github.com/lyft/flytestdlib/cli/pflags"
 	_ "github.com/vektra/mockery/cmd/mockery"
 	_ "github.com/alvaroloes/enumer"
+	_ "github.com/pseudomuto/protoc-gen-doc"
 )
diff --git a/boilerplate/lyft/golang_test_targets/download_tooling.sh b/boilerplate/lyft/golang_test_targets/download_tooling.sh
@@ -20,6 +20,7 @@ tools=(
   "github.com/lyft/flytestdlib/cli/pflags"
   "github.com/golangci/golangci-lint/cmd/golangci-lint"
   "github.com/alvaroloes/enumer"
+  "github.com/pseudomuto/protoc-gen-doc"
 )
 
 tmp_dir=$(mktemp -d -t gotooling-XXX)

diff --git a/conf.py b/conf.py
@@ -16,7 +16,6 @@
 import re
 import sys
 
-sys.path.insert(0, os.path.abspath("gen/pb-protodoc/"))
 import recommonmark
 from recommonmark.transform import AutoStructify
 
@@ -31,7 +30,6 @@
 release = re.sub('^v', '', os.popen('git describe').read().strip())
 version = release
 
-
 # -- General configuration ---------------------------------------------------
 
 # Add any Sphinx extension module names here, as strings. They can be
@@ -68,7 +66,7 @@
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-source_suffix = ['.rst', '.md']
+source_suffix = ['.rst']
 
 # The master toctree document.
 master_doc = "index"
@@ -83,7 +81,7 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path .
-exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store', 'tmp/doc_gen_deps', 'gen/*/*/*/*/*', 'boilerplate', 'pull_request_template.md', 'README.rst', 'reference/*', 'CODE_OF_CONDUCT.md']
+exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store', 'tmp/doc_gen_deps', 'gen/*/*/*/*/*', 'CODE_OF_CONDUCT.md', 'pull_request_template.md', 'boilerplate']
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = "tango"

diff --git a/developing.rst b/developing.rst
@@ -1,6 +1,14 @@
 Contributing to Flyte IDL
 =========================
 
+This is one of the core repositories of Flyte and contains the Specification of
+the Flyte Lanugage. The Specification is maintained using Googles fantastic
+Protcol buffers library. The code and docs are auto-generated.
+
+* [flyte.org](https://flyte.org)
+* [Flyte Docs](http://flyte.readthedocs.org/)
+* [FlyteIDL Docs](http://flyteidl.readthedocs.org)
+
 Generate Code from protobuf
 ----------------------------
 #. Make sure docker is installed locally.
@@ -18,9 +26,44 @@ Generate Code from protobuf
 
      make download_tooling
 
+Generate Docs from protobuf
+----------------------------
+
 #. To add new dependencies for a doc, modify ``doc_requirements.in`` and then
 
    .. prompt:: bash
 
      make doc-requirements.txt
 
+Docs structure
+**************
+
+The index.rst files for protos are kept in parallel folder structure under the docs folder.
+All the proto definitions are within protos/flyteidl and there corresponding docs are kept in protos/docs
+
+Each module in protos has same named module under the docs also.
+eg : protos/flyteidl/core has same named doc structure placing it index and other documentation files in protos/docs/core
+
+
+Docs Generation
+***************
+
+Relies on protoc plugin pseudomuto/protoc-gen-doc for generating the docs.
+Currently pseudomuto/protoc-gen-doc is being used for generating the RST files which are placed in protos/docs and corresponding folder based on the module.
+Until this code is checkedin https://github.com/pseudomuto/protoc-gen-doc/pull/440, use the branch to build the protoc-gen-doc executable
+Follow the steps to build and use the protoc-gen-doc
+
+#. This will install all the tooling including protoc-gen-doc plugin for protoc used for documentation
+
+   .. prompt:: bash
+
+      make download_tooling
+
+#. The protoc-gen-doc will now be available for protoc.Following is an example from `generate_protos.sh` file which helps in generating the core documentation from its proto files
+
+   .. prompt:: bash
+
+     core_proto_files=`ls protos/flyteidl/core/*.proto |xargs`
+     # Remove any currently generated file
+     ls -d protos/docs/core/* | grep -v index.rst | xargs rm
+     protoc --doc_out=protos/docs/core --doc_opt=restructuredtext,core.rst -I=protos `echo $core_proto_files`
diff --git a/gen/pb-cpp/flyteidl/admin/matchable_resource.pb.cc b/gen/pb-cpp/flyteidl/admin/matchable_resource.pb.cc
diff --git a/gen/pb-cpp/flyteidl/admin/matchable_resource.pb.h b/gen/pb-cpp/flyteidl/admin/matchable_resource.pb.h
diff --git a/gen/pb-cpp/flyteidl/admin/workflow.pb.cc b/gen/pb-cpp/flyteidl/admin/workflow.pb.cc
diff --git a/gen/pb-cpp/flyteidl/admin/workflow.pb.h b/gen/pb-cpp/flyteidl/admin/workflow.pb.h