🔀 MERGE: Improve Notebook Output Rendering (#243)

This merge encapsulates a major refactor of the notebook output rendering process.
The process was rewritten to be more module and pluggable, hopefully allowing for people to customise the way outputs are rendered for there own purposes.
this is includes lots of additional configuration/improvements, such as ANSI and Markdown rendering, stdout/stderr removal, and utilisation of metadata to control image formatting.
Also, error reporting and testing has been improved.

docs/_static/patch.css

@@ -0,0 +1,7 @@
+/* CSS that should eventually go in sphinx-book-theme */
+
+dd {
+    margin-top: 3px;
+    margin-bottom: 10px;
+    margin-left: 30px;
+}

docs/api/index.rst

@@ -0,0 +1,18 @@
+.. _api/main:
+
+Python API
+==========
+
+.. toctree::
+    :maxdepth: 2
+
+    nodes
+    render_outputs
+
+Miscellaneous
+-------------
+
+.. autoclass:: myst_nb.ansi_lexer.AnsiColorLexer
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/nodes.rst

@@ -0,0 +1,26 @@
+.. _api/nodes:
+
+AST Nodes
+---------
+
+.. automodule:: myst_nb.nodes
+
+.. autoclass:: myst_nb.nodes.CellNode
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: myst_nb.nodes.CellInputNode
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: myst_nb.nodes.CellOutputNode
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: myst_nb.nodes.CellOutputBundleNode
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/render_outputs.rst

@@ -0,0 +1,30 @@
+.. _api/output_renderer:
+
+Output Renderer
+---------------
+
+.. automodule:: myst_nb.render_outputs
+
+.. autoclass:: myst_nb.render_outputs.CellOutputsToNodes
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoexception:: myst_nb.render_outputs.MystNbEntryPointError
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autofunction:: myst_nb.render_outputs.load_renderer
+
+
+.. autoclass:: myst_nb.render_outputs.CellOutputRendererBase
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :special-members: __init__
+
+.. autoclass:: myst_nb.render_outputs.CellOutputRenderer
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/conf.py

@@ -34,6 +34,8 @@
     "sphinx_togglebutton",
     "sphinx_copybutton",
     "sphinx.ext.intersphinx",
+    "sphinx.ext.autodoc",
+    # "sphinx.ext.viewcode"
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -63,19 +65,31 @@
 }
 
 intersphinx_mapping = {
+    "python": ("https://docs.python.org/3.8", None),
     "jb": ("https://jupyterbook.org/", None),
     "myst": ("https://myst-parser.readthedocs.io/en/latest/", None),
     "markdown_it": ("https://markdown-it-py.readthedocs.io/en/latest", None),
     "nbclient": ("https://nbclient.readthedocs.io/en/latest", None),
     "nbformat": ("https://nbformat.readthedocs.io/en/latest", None),
+    "sphinx": ("https://www.sphinx-doc.org/en/3.x", None),
 }
 
 intersphinx_cache_limit = 5
 
+nitpick_ignore = [
+    ("py:class", "docutils.nodes.document"),
+    ("py:class", "docutils.nodes.Node"),
+    ("py:class", "docutils.nodes.container"),
+    ("py:class", "docutils.nodes.system_message"),
+    ("py:class", "nbformat.notebooknode.NotebookNode"),
+    ("py:class", "pygments.lexer.RegexLexer"),
+]
+
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+html_css_files = ["patch.css"]
 
 copybutton_selector = "div:not(.output) > div.highlight pre"
 

docs/index.md

@@ -81,7 +81,9 @@ Finally, here is documentation on contributing to the development of MySt-NB
 
 ```{toctree}
 :titlesonly:
+:maxdepth: 1
 develop/contributing
+api/index
 GitHub Repo <https://github.com/executablebooks/myst-nb>
 ```
 

docs/use/formatting_outputs.md

@@ -0,0 +1,218 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: '0.8'
+    jupytext_version: '1.4.1'
+kernelspec:
+  display_name: Python 3
+  language: python
+  name: python3
+---
+
+(use/format)=
+# Formatting code outputs
+
+(use/format/priority)=
+## Render priority
+
+When Jupyter executes a code cell it can produce multiple outputs, and each of these outputs can contain multiple [MIME media types](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types), for use by different output formats (like HTML or LaTeX).
+
+MyST-NB stores a default priority dictionary for most of the common [Sphinx builders](https://www.sphinx-doc.org/en/master/usage/builders/index.html), which you can be also update in your `conf.py`.
+For example, this is the default priority list for HTML:
+
+```python
+nb_render_priority = {
+  "html": (
+            "application/vnd.jupyter.widget-view+json",
+            "application/javascript",
+            "text/html",
+            "image/svg+xml",
+            "image/png",
+            "image/jpeg",
+            "text/markdown",
+            "text/latex",
+            "text/plain",
+        )
+}
+```
+
+:::{seealso}
+[](use/format/cutomise), for a more advanced means of customisation.
+:::
+
+## Removing stdout and stderr
+
+In some cases you may not wish to display stdout/stderr outputs in your final documentation,
+for example, if they are only for debugging purposes.
+You can tell MyST-NB to remove these outputs using the `remove-stdout` and `remove-stderr` [cell tags](https://jupyter-notebook.readthedocs.io/en/stable/changelog.html#cell-tags), like so:
+
+````md
+```{code-cell} ipython3
+:tags: [remove-input,remove-stdout,remove-stderr]
+
+import pandas, sys
+print("this is some stdout")
+print("this is some stderr", file=sys.stderr)
+# but what I really want to show is:
+pandas.DataFrame({"column 1": [1, 2, 3]})
+```
+````
+
+```{code-cell} ipython3
+:tags: [remove-input,remove-stdout,remove-stderr]
+
+import pandas, sys
+print("this is some stdout")
+print("this is some stderr", file=sys.stderr)
+# but what I really want to show is:
+pandas.DataFrame({"column 1": [1, 2, 3]})
+```
+
+(use/format/images)=
+## Images
+
+With the default renderer, for any image types output by the code, we can apply formatting *via* cell metadata.
+The keys should be placed under `myst`, then for the image we can apply all the variables of the standard [image directive](https://docutils.sourceforge.io/docs/ref/rst/directives.html#image):
+
+- **width**: length or percentage (%) of the current line width
+- **height**: length
+- **scale**: integer percentage (the "%" symbol is optional)
+- **align**: "top", "middle", "bottom", "left", "center", or "right"
+- **classes**: space separated strings
+- **alt**: string
+
+Units of length are: 'em', 'ex', 'px', 'in', 'cm', 'mm', 'pt', 'pc'
+
+We can also set a caption (which is rendered as [CommonMark](https://commonmark.org/)) and name, by which to reference the figure:
+
+````md
+```{code-cell} ipython3
+---
+myst:
+  image:
+    width: 200px
+    alt: fun-fish
+    classes: shadow bg-primary
+  figure:
+    caption: |
+      Hey everyone its **party** time!
+    name: fun-fish
+---
+from IPython.display import Image
+Image("images/fun-fish.png")
+```
+````
+
+```{code-cell} ipython3
+---
+myst:
+  image:
+    width: 300px
+    alt: fun-fish
+    classes: shadow bg-primary
+  figure:
+    caption: |
+      Hey everyone its **party** time!
+    name: fun-fish
+---
+from IPython.display import Image
+Image("images/fun-fish.png")
+```
+
+Now we can link to the image from anywhere in our documentation: [swim to the fish](fun-fish)
+
+(use/format/markdown)=
+## Markdown
+
+Markdown output is parsed by MyST-Parser, currently with the configuration set to `myst_commonmark_only=True` (see [MyST configuration options](myst:intro/config-options)).
+
+The parsed Markdown is integrated into the wider documentation, and so it is possible, for example, to include internal references:
+
+```{code-cell} ipython3
+from IPython.display import display, Markdown
+display(Markdown('**_some_ markdown** and an [internal reference](use/format/markdown)!'))
+```
+
+and even internal images can be rendered!
+
+```{code-cell} ipython3
+display(Markdown('![figure](../_static/logo.png)'))
+```
+
+(use/format/ansi)=
+## ANSI Outputs
+
+By default, the standard output/error streams and text/plain MIME outputs may contain ANSI escape sequences to change the text and background colors.
+
+```{code-cell} ipython3
+import sys
+print("BEWARE: \x1b[1;33;41mugly colors\x1b[m!", file=sys.stderr)
+print("AB\x1b[43mCD\x1b[35mEF\x1b[1mGH\x1b[4mIJ\x1b[7m"
+      "KL\x1b[49mMN\x1b[39mOP\x1b[22mQR\x1b[24mST\x1b[27mUV")
+```
+
+This uses the built-in {py:class}`~myst_nb.ansi_lexer.AnsiColorLexer` [pygments lexer](https://pygments.org/).
+You can change the lexer used in the `conf.py`, for example to turn off lexing:
+
+```python
+nb_render_text_lexer = "none"
+```
+
+The following code[^acknowledge] shows the 8 basic ANSI colors it is based on.
+Each of the 8 colors has an “intense” variation, which is used for bold text.
+
+[^acknowledge]: Borrowed from [nbsphinx](https://nbsphinx.readthedocs.io/en/0.7.1/code-cells.html#ANSI-Colors)!
+
+```{code-cell} ipython3
+text = " XYZ "
+formatstring = "\x1b[{}m" + text + "\x1b[m"
+
+print(
+    " " * 6
+    + " " * len(text)
+    + "".join("{:^{}}".format(bg, len(text)) for bg in range(40, 48))
+)
+for fg in range(30, 38):
+    for bold in False, True:
+        fg_code = ("1;" if bold else "") + str(fg)
+        print(
+            " {:>4} ".format(fg_code)
+            + formatstring.format(fg_code)
+            + "".join(
+                formatstring.format(fg_code + ";" + str(bg)) for bg in range(40, 48)
+            )
+        )
+```
+
+:::{note}
+ANSI also supports a set of 256 indexed colors.
+This is currently not supported, but we hope to introduce it at a later date
+(raise an issue on the repository if you require it!).
+:::
+
+(use/format/cutomise)=
+## Customise the render process
+
+The render process is goverened by subclasses of {py:class}`myst_nb.render_outputs.CellOutputRendererBase`, which dictate how to create the `docutils` AST nodes for a particular MIME type. the default implementation is {py:class}`~myst_nb.render_outputs.CellOutputRenderer`.
+
+Implementations are loaded *via* Python [entry points](https://packaging.python.org/guides/distributing-packages-using-setuptools/#entry-points), in the `myst_nb.mime_render` group.
+So it is possible to inject your own subclass to handle rendering.
+
+For example, the renderers loaded in this package are:
+
+```python
+entry_points={
+    "myst_nb.mime_render": [
+        "default = myst_nb.render_outputs:CellOutputRenderer",
+        "inline = myst_nb.render_outputs:CellOutputRendererInline",
+    ],
+}
+```
+
+You can then select the renderer plugin in your `conf.py`:
+
+```python
+nb_render_plugin = "default"
+```

docs/use/hiding.md

@@ -50,7 +50,7 @@ right to show it.
 
 # This cell has a hide-input tag
 fig, ax = plt.subplots()
-ax.scatter(*data, c=data[0], s=data[0])
+points =ax.scatter(*data, c=data[0], s=data[0])
 ```
 
 Here's a cell with a `hide-output` tag:
@@ -60,7 +60,7 @@ Here's a cell with a `hide-output` tag:
 
 # This cell has a hide-output tag
 fig, ax = plt.subplots()
-ax.scatter(*data, c=data[0], s=data[0])
+points =ax.scatter(*data, c=data[0], s=data[0])
 ```
 
 And the following cell has a `hide-cell` tag:
@@ -70,7 +70,7 @@ And the following cell has a `hide-cell` tag:
 
 # This cell has a hide-cell tag
 fig, ax = plt.subplots()
-ax.scatter(*data, c=data[0], s=data[0])
+points =ax.scatter(*data, c=data[0], s=data[0])
 ```
 
 (use/hiding/markdown)=
@@ -152,7 +152,7 @@ the page at all.
 
 # This cell has a remove-input tag
 fig, ax = plt.subplots()
-ax.scatter(*data, c=data[0], s=data[0])
+points =ax.scatter(*data, c=data[0], s=data[0])
 ```
 
 Here's a cell with a `remove-output` tag:

docs/use/index.md

@@ -10,5 +10,6 @@ start
 myst
 execute
 hiding
+formatting_outputs
 glue
 ```