Merge pull request #49 from joshtemple/release/1.1.0

v1.1.0

docs/source/advanced.rst

@@ -101,9 +101,9 @@ To learn more about the parse tree and the different kinds of nodes, read the fu
 Traversing and modifying the parse tree
 ---------------------------------------
 
-The parse tree follows a design pattern called the `visitor pattern <https://en.wikipedia.org/wiki/Visitor_pattern>`_. The visitor pattern allows us to define flexible algorithms that interact with the tree without having to implement those algorithms on the tree itself.
+The parse tree follows a design pattern called the `visitor pattern <https://en.wikipedia.org/wiki/Visitor_pattern>`_. The visitor pattern allows us to define flexible algorithms that interact with the tree without having to implement those algorithms as methods on the tree's node classes.
 
-Each node implements a method called ``accept``, that accepts a :py:class:`lkml.tree.Visitor` instance and passes itself the corresponding ``visit_`` method on the visitor.
+Each node implements a method called ``accept``, that accepts a :py:class:`lkml.tree.Visitor` instance and passes itself to the corresponding ``visit_`` method on the visitor.
 
 Here's the ``accept`` method for a ``ListNode``.
 
@@ -125,13 +125,7 @@ For example, we could write a linting visitor that traverses the parse tree and
     from lkml.visitors import BasicVisitor
 
     class DescriptionVisitor(BasicVisitor):
-        def _visit(self, parent):
-            """For each node, visit its children."""
-            if parent.children:
-                for node in parent.children:
-                    node.accept(self)
-                    
-        def visit_block(self, block):
+        def visit_block(self, block: BlockNode):
             """For each block, check if it's a dimension and if it has a description."""
             if block.type.value == 'dimension':
                 child_types = [node.type.value for node in block.container.items]
@@ -141,19 +135,43 @@ For example, we could write a linting visitor that traverses the parse tree and
    # Assume we already have a parse tree to visit
    tree.accept(DescriptionVisitor())
 
-:py:class:`lkml.visitors.BasicVisitor` implements a handy shortcut where the default visiting behavior for each node type is defined by ``_visit``. We can simply override that default behavior for ``visit_block`` so we can inspect the dimensions, which are ``BlockNodes``.
+:py:class:`lkml.visitors.BasicVisitor`, will traverse the tree but do nothing. We can simply override that default behavior for ``visit_block`` so we can inspect the dimensions, which are ``BlockNodes``.
 
-For each dimension, we iterate through its children and throw an error if there aren't any with the description type.
+For each block that is a dimension, we iterate through its children and throw an error if a child with the description type is not present.
 
-Modifying the parse tree
-^^^^^^^^^^^^^^^^^^^^^^^^
-Because syntax nodes and tokens are immutable, you can't change them once created, you may only replace or remove them.
+Modifying the parse tree with transformers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Because syntax nodes and tokens are immutable, you can't change them once created, you may only replace or remove them. This makes modifying the parse tree challenging, because the entire tree needs to be rebuilt for each change.
 
-This makes modifying the parse tree challenging, because the entire tree needs to be rebuilt for each change.
+lkml includes a basic transformer, :py:class:`lkml.visitors.BasicTransformer`, which like the visitors, traverses the tree. However, the transformer visits and replaces each node's children, allowing the immutable parse tree to be rebuilt with modifications.
 
-Future versions of lkml will provide helpers to make this process easier.
+As an example, let's write a transformer that injects a user attribute into each ``sql_table_name`` field. This is something that could easily be solved with regex, but let's write a transformer as an example instead::
 
-For now, the Python standard library ``dataclasses`` module has a helpful method, ``replace``, that returns a new instance of a dataclass with the desired modification. Nodes and tokens are dataclasses, so this is a convenient way to "modify" them, even though they are immutable.
+    from dataclasses import replace
+    from lkml.visitors import BasicTransformer
+
+    class TableNameTransformer(BasicTransformer):    
+        def visit_pair(self, node: PairNode) -> PairNode:
+            """Visit each pair and replace the SQL table schema with a user attribute."""
+            if node.type.value == 'sql_table_name':
+                try:
+                    schema, table_name = node.value.value.split('.')
+                # Sometimes the table name won't have a schema
+                except ValueError:
+                    table_name = node.value.value
+                    
+                new_value: str = '{{ _user_attributes["dbt_schema"] }}.' + table_name
+                new_node: PairNode = replace(node, value=ExpressionSyntaxToken(new_value))
+                return new_node
+            else:
+                return node
+    
+    # Assume we already have a parse tree to visit
+    tree.accept(TableNameTransformer())
+
+This transformer traverses the parse tree and modifies all ``PairNodes`` that have the ``sql_table_name`` type, injecting a user attribute into the expression.
+
+We rely on the dataclasses function :py:func:`dataclasses.replace`, which allows us to copy an immutable node (all lkml nodes are frozen, immutable dataclasses) with modifications---in this case, to the ``value`` attribute of the ``PairNode``.
 
 Generating LookML from the parse tree
 -------------------------------------

docs/source/conf.py

@@ -27,7 +27,12 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ["sphinx.ext.doctest", "sphinx.ext.napoleon", "sphinx.ext.autodoc"]
+extensions = [
+    "sphinx.ext.doctest",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.intersphinx",
+]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
@@ -38,6 +43,7 @@
 exclude_patterns = []
 
 master_doc = "index"
+intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
 
 # -- Options for HTML output -------------------------------------------------
 

lkml/visitors.py

@@ -1,5 +1,6 @@
 import logging
-from typing import Union
+from dataclasses import replace
+from typing import Union, overload
 
 from lkml.tree import (
     BlockNode,
@@ -16,11 +17,22 @@
 
 
 class BasicVisitor(Visitor):
-    """Visitor class that calls the ``_visit`` method for every node type."""
+    """Visitor class that calls the ``_visit`` method for every node type.
 
-    @staticmethod
-    def _visit(node: Union[SyntaxNode, SyntaxToken]):
-        raise NotImplementedError
+    This class doesn't actually do anything when visiting a tree other than traverse
+    the nodes. It's meant to be used as a base class for building more useful and
+    complex visitors. For example, override any of the ``visit_`` methods for node-type
+    specific behavior.
+
+    """
+
+    def _visit(self, node: Union[SyntaxNode, SyntaxToken]):
+        """For each node, visit its children."""
+        if isinstance(node, SyntaxToken):
+            return
+        elif node.children:
+            for child in node.children:
+                child.accept(self)
 
     def visit(self, document: DocumentNode):
         return self._visit(document)
@@ -47,3 +59,57 @@ class LookMlVisitor(BasicVisitor):
     @staticmethod
     def _visit(node: Union[SyntaxNode, SyntaxToken]) -> str:
         return str(node)
+
+
+class BasicTransformer(Visitor):
+    """Visitor class that returns a new tree, modifying the tree as needed."""
+
+    @overload
+    def _visit_items(self, node: ContainerNode) -> ContainerNode:
+        ...
+
+    @overload
+    def _visit_items(self, node: ListNode) -> ListNode:
+        ...
+
+    def _visit_items(self, node):
+        """Visit a node whose children are held in the ``items`` attribute."""
+        if node.children:
+            new_children = tuple(child.accept(self) for child in node.children)
+            return replace(node, items=new_children)
+        else:
+            return node
+
+    @overload
+    def _visit_container(self, node: BlockNode) -> BlockNode:
+        ...
+
+    @overload
+    def _visit_container(self, node: DocumentNode) -> DocumentNode:
+        ...
+
+    def _visit_container(self, node):
+        """Visit a node whose only child is the ``container`` attribute."""
+        if node.container:
+            new_child = node.container.accept(self)
+            return replace(node, container=new_child)
+        else:
+            return node
+
+    def visit(self, node: DocumentNode) -> DocumentNode:
+        return self._visit_container(node)
+
+    def visit_container(self, node: ContainerNode) -> ContainerNode:
+        return self._visit_items(node)
+
+    def visit_list(self, node: ListNode) -> ListNode:
+        return self._visit_items(node)
+
+    def visit_block(self, node: BlockNode) -> BlockNode:
+        return self._visit_container(node)
+
+    def visit_pair(self, node: PairNode) -> PairNode:
+        return node
+
+    def visit_token(self, token: SyntaxToken) -> SyntaxToken:
+        return token

setup.py

@@ -2,7 +2,7 @@
 
 from setuptools import find_packages, setup
 
-__version__ = "1.0.1"
+__version__ = "1.1.0"
 
 here = Path(__file__).parent.resolve()