ENH: Support almost unlimited summary preprocessing.

Add a configuration option to reload SUMMARY.md, evaluate it as a template, and
re-parse chapters. This allows us to support full arbitrary preprocessing of
summary.md, but it means we discard the results of any preprocessors that ran
before us.

In most scenarios this will probably be an OK tradeoff. The ideal solution
still requires something like rust-lang/mdBook#2466.

README.md

@@ -1,9 +1,11 @@
 # mdbook-minijinja
 
 mdbook-minijinja is an [mdbook][mdbook] [preprocessor][mdbook-preprocessor]
-that evaluates the files in your book as [minijinja][minijinja]
-templates. Template features are fully supported inside book chapters.  Limited
-template features are available in `SUMMARY.md` (see below).
+that evaluates the files in your book as [minijinja][minijinja] templates.
+
+See the [example
+book](https://github.com/ssanderson/mdbook-minijinja/tree/main/example-book)
+for a full example.
 
 [mdbook]: https://rust-lang.github.io/mdBook
 [mdbook-preprocessor]: https://rust-lang.github.io/mdBook/format/configuration/preprocessors.html
@@ -15,6 +17,23 @@ template features are available in `SUMMARY.md` (see below).
 # book.toml
 [preprocessor.minijinja]
 
+# Whether or not mdbook-minijinja should evaluate SUMMARY.md
+# as a template. If this is true, mdbook-minijinja will reload SUMMARY.md,
+# evaluate it as a template, and then reload book chapters from the
+# re-parsed SUMMARY.md. This discards the effects of any preprocessors
+# that ran before mdbook-minijinja, so mdbook-minijinja should be configured
+# as the first preprocessor if summary preprocessing is enabled. Use
+# the `before` key to configure preprocessor order.
+#
+# Default value is false.
+preprocess_summary = true
+
+# Configure mdbook-minijinja to run before other preprocessors.
+#
+# "index" and "links" are built-in preprocessors run by mdbook by default. If you
+# have other preprocessors enabled, you may want to include them here as well.
+before = ["index", "links"]
+
 # Configure behavior of evaluating undefined variables in minijinja.
 #
 # Options are "strict", "lenient", or "chained".
@@ -29,7 +48,7 @@ undefined_behavior = "strict"
 # include directives will look for templates here.
 #
 # If this path is absolute, it is used as-is. If it is relative, it is
-# interpreted relative to the path containing book.toml.
+# interpreted relative to the directory containing book.toml.
 #
 # See https://docs.rs/minijinja/latest/minijinja/fn.path_loader.html for more
 # details.
@@ -49,21 +68,24 @@ list_of_strings = ["foo", "bar", "buzz"]
 partial_chapter_name = "Partial"
 ```
 
-## SUMMARY.md Limitations
+## Preprocessing SUMMARY.md
 
 The structure of an mdbook is defined by the top-level
 [SUMMARY.md](https://rust-lang.github.io/mdBook/format/summary.html) file,
-which contains a list of the book's chapters and titles. mdbook only invokes
-preprocessors after it has already parsed and evaluated SUMMARY.md. This means
-mdbook-minijinja can only support a limited set of jinja template operations in
-SUMMARY.md:
+which contains a list of the book's sections and chapters.
+
+MDBook only invokes preprocessors after SUMMARY.md has already been loaded and
+parsed. This creates a challenge for preprocessors like mdbook-minijinja that
+want to preprocess SUMMARY.md,
 
-- ✅ Simple if/else conditionals to enable or disable chapters based on
-  variables are supported.
-- ✅ Template expressions within chapter and part titles are supported.
-- ❌ `{% include %}` or other template expansions that evaluate to new book
-  chapters are not supported. All book chapters must be present in the
-  SUMMARY.md source.
+To work around the above, if `preprocess_summary` is set to `true`,
+mdbook-minijinja reloads SUMMARY.md and evaluates it as a minijinja
+template. We then reload all chapters referenced by the updated
+SUMMARY.md. This allows SUMMARY.md to be evaluated as a minijinja template, but
+it means that **we discard the results of any preprocessors that ran before
+mdbook-minijinja**.
 
-For an example of supported functionality, see the [example
-book](./example-book/src/SUMMARY.md).
+If you enable summary preprocessing, we recommend configuring mdbook-minijinja
+as your first preprocessor using the [before and
+after](https://rust-lang.github.io/mdBook/format/configuration/preprocessors.html#require-a-certain-order)
+configuration values. See the example configuration above for an example.

example-book/book.toml

@@ -7,22 +7,43 @@ title = "MDBook Minijinja Example"
 
 [preprocessor.minijinja]
 
+# Whether or not mdbook-minijinja should evaluate SUMMARY.md
+# as a template. If this is true, mdbook-minijinja will reload SUMMARY.md,
+# evaluate it as a template, and then reload book chapters from the
+# re-parsed SUMMARY.md. This discards the effects of any preprocessors
+# that ran before mdbook-minijinja, so mdbook-minijinja should be configured
+# as the first preprocessor if summary preprocessing is enabled. Use
+# the `before` key to configure preprocessor order.
+#
+# Default value is false.
+preprocess_summary = true
+
+# Configure mdbook-minijinja to run before other preprocessors.
+#
+# "index" and "links" are built-in preprocessors run by mdbook by default. If you
+# have other preprocessors enabled, you may want to include them here as well.
+before = ["index", "links"]
+
 # Configure behavior of evaluating undefined variables in minijinja.
 #
-# Options are "strict", "lenient", or "chained". Default is "strict".
+# Options are "strict", "lenient", or "chained".
 #
 # See https://docs.rs/minijinja/latest/minijinja/enum.UndefinedBehavior.html
 # for more details.
+#
+# Default value is "strict".
 undefined_behavior = "strict"
 
 # Path to a directory containing minijinja templates. Minijinja import and
 # include directives will look for templates here.
 #
 # If this path is absolute, it is used as-is. If it is relative, it is
-# interpreted relative to the path containing book.toml.
+# interpreted relative to the directory containing book.toml.
 #
 # See https://docs.rs/minijinja/latest/minijinja/fn.path_loader.html for more
-# details
+# details.
+#
+# Default value is "templates".
 templates = "templates"
 
 # Variables defined in this section will be available for use in templates.

example-book/src/SUMMARY.md

@@ -18,3 +18,7 @@
 # Templates
 
 - [Uses Templates](./templates.md)
+
+{% if condition_true %}
+{% include "conditional_summary_section.md" %}
+{% endif %}

example-book/src/conditional_chapter1.md

@@ -0,0 +1,3 @@
+# Conditional Chapter 1
+
+Stuff

example-book/src/conditional_chapter2.md

@@ -0,0 +1,3 @@
+# Conditional Chapter 2
+
+Stuff

example-book/templates/conditional_summary_section.md

@@ -0,0 +1,4 @@
+# Conditional Section
+
+- [Conditional Chapter 1](./conditional_chapter1.md)
+- [Conditional Chapter 2](./conditional_chapter2.md)

src/config.rs

@@ -5,14 +5,24 @@ use serde::Deserialize;
 
 #[derive(Debug, Deserialize)]
 pub struct MiniJinjaConfig {
+    /// Whether we should preprocess SUMMARY.md.
+    #[serde(default)]
+    pub preprocess_summary: bool,
+
+    /// Variables to be passed to the minijinja environment.
     pub variables: toml::Table,
+
+    /// Undefined behavior setting for minijinja.
     #[serde(default)]
     pub undefined_behavior: UndefinedBehavior,
+
+    /// Templates directory for minijinja.
     #[serde(default = "MiniJinjaConfig::default_templates_dir")]
     pub templates_dir: PathBuf,
 }
 
 impl MiniJinjaConfig {
+    /// Create a new minijinja::Environment based on the configuration.
     pub fn create_env<'source>(&self, root: &PathBuf) -> Environment<'source> {
         let mut env = Environment::new();
         env.set_undefined_behavior(self.undefined_behavior.into());
@@ -22,7 +32,8 @@ impl MiniJinjaConfig {
         } else {
             root.join(&self.templates_dir)
         };
-        log::info!("loading templates from {}", templates_dir.display());
+
+        log::debug!("loading templates from {}", templates_dir.display());
 
         env.set_loader(minijinja::path_loader(templates_dir));
         env

src/preprocessor.rs

@@ -1,7 +1,7 @@
 use mdbook::{
-    book::{Book, SummaryItem},
+    book::Book,
     preprocess::{Preprocessor, PreprocessorContext},
-    BookItem,
+    BookItem, MDBook,
 };
 use serde::Serialize;
 
@@ -14,7 +14,7 @@ impl Preprocessor for MiniJinjaPreprocessor {
         "minijinja"
     }
 
-    fn run(&self, ctx: &PreprocessorContext, mut book: Book) -> anyhow::Result<Book> {
+    fn run(&self, ctx: &PreprocessorContext, book: Book) -> anyhow::Result<Book> {
         let conf: Option<config::MiniJinjaConfig> = ctx
             .config
             .get_deserialized_opt(format!("preprocessor.{}", self.name()))?;
@@ -23,55 +23,34 @@ impl Preprocessor for MiniJinjaPreprocessor {
             anyhow::bail!("missing config section for {}", self.name())
         };
 
-        log::debug!("{conf:#?}");
+        log::trace!("{conf:#?}");
 
         let env = conf.create_env(&ctx.root);
 
-        // XXX: mdBook has already loaded the summary by the time we get here,
-        // so we need to load it ourselves, evaluate it as a template, and then
-        // try to figure out how that should modify what mdbook loaded.
-        //
-        // This doesn't really fully work: we can support basic templated
-        // values in chapter names, and conditionally included/excluded
-        // chapters, but fully general jinja templates aren't supported.
-        let mut summary_text = std::fs::read_to_string(ctx.config.book.src.join("SUMMARY.md"))?;
-        eval_in_place(&env, &mut summary_text, &conf.variables);
+        let mut book = if conf.preprocess_summary {
+            // mdBook has already loaded the summary by the time we get here, so we
+            // need to reload it ourselves, evaluate it as a template, and then
+            // replace what mdbook loaded with our own evaluated templates.
+            //
+            // This discards the output of any preprocessors that ran before us, so
+            // mdbook-minijinja should be configured as the first preprocessor.
+            let summary_path = ctx.config.book.src.join("SUMMARY.md");
+            log::info!("reloading summary from {}", summary_path.display());
 
-        let summary = mdbook::book::parse_summary(&summary_text)?;
-        let summary_names = summary
-            .prefix_chapters
-            .iter()
-            .chain(summary.numbered_chapters.iter())
-            .chain(summary.suffix_chapters.iter())
-            .filter_map(|c| match c {
-                SummaryItem::Link(l) => Some(l.name.clone()),
-                _ => None,
-            })
-            .collect::<std::collections::HashSet<_>>();
+            let mut summary_text = std::fs::read_to_string(summary_path)?;
+            eval_in_place(&env, &mut summary_text, &conf.variables);
+            let summary = mdbook::book::parse_summary(&summary_text)?;
 
-        // Filter out sections that should get dropped after evaluating the
-        // summary as a template.
-        book.sections = book
-            .sections
-            .iter()
-            .filter_map(|item| match item {
-                BookItem::Chapter(c) => match env.render_str(&c.name, &conf.variables) {
-                    Ok(name) => {
-                        if summary_names.contains(&name) {
-                            Some(item)
-                        } else {
-                            None
-                        }
-                    }
-                    Err(e) => {
-                        log_jinja_err(&e);
-                        None
-                    }
-                },
-                _ => Some(item),
-            })
-            .cloned()
-            .collect::<Vec<_>>();
+            let MDBook { book, .. } = MDBook::load_with_config_and_summary(
+                ctx.root.clone(),
+                ctx.config.clone(),
+                summary,
+            )?;
+            book
+        } else {
+            log::info!("skipping preprocessing of SUMMARY.md because preprocess_summary is false");
+            book
+        };
 
         book.for_each_mut(|item| match item {
             BookItem::Chapter(c) => {