Support $XDG_DATA_DIRS or equivalent for templates and lua filters

[pbsds]

Describe your proposed improvement and the problem it solves.

To avoid tailoring defaults.yaml to my specific environment using absolute paths, I'd rather configure my pandoc setup to search various paths for lua filters. defaults.yaml does not support using environment variables to my knowledge, overriding $XDG_DATA_HOME is a very heavy-handed (i run other things as well) and it only supports configuring one path making such a solution not scale.

For this reason I request support for $XDG_DATA_DIRS, which is the natural conclusion of #3582. Optionally an environment variable such as $PANDOC_DATA_DIRS that supports multiple entries could also do the trick.

It could be the solution to #6635. For me, it would allow me to use the nixpkgs package pandoc-lua-filters in a nix-shell environment, which simply dumps the lua filters in a read-only folder:

$ nix-build '<nixpkgs>' -A pandoc
/nix/store/x58dmn3qj75vnrlhvv347kwb2hb465k2-pandoc-2.14.0.3
$ nix-build '<nixpkgs>' -A pandoc-lua-filters
/nix/store/ac0fx53idm0ywln5zi0g6p11g2s4w7sd-pandoc-lua-filters-2020-11-30
$ ls `nix-build '<nixpkgs>' -A pandoc-lua-filters`/share/pandoc/filters
abstract-to-meta.lua    diagram-generator.lua   lilypond.lua                 pagebreak.lua           scrlttr2.lua        table-short-captions.lua
author-info-blocks.lua  include-code-files.lua  lua-debug-example.lua        pandoc-quotes.lua       section-refs.lua    track-changes.lua
bibexport.lua           include-files.lua       minted.lua                   revealjs-codeblock.lua  short-captions.lua  wordcount.lua
cito.lua                latex-hyphen.lua        multiple-bibliographies.lua  scholarly-metadata.lua  spellcheck.lua

Tangentially related due to the discussion: #6760

[jgm]

See https://pandoc.org/MANUAL.html#defaults-files
You can interpolate environment variables in paths in defaults files.

[pbsds]

Great to know as it solves my immediate problem, but what is your opinion on supporting $XDG_DATA_DIRS?

[jgm]

I don't know. I'd like to understand better what this would allow us to do that we can't do now, and why it's important.

[pbsds]

It would allow one to compose multiple directories with templates and filters using only environ variables. It would allow package managers like nix, and modules (commonly found in multi-user cluster environments) to work its magic.

It would also allow one to wrap pandoc like the following:

#!/bin/sh
# this file is in PATH
export XDG_DATA_DIRS=/some/lua/collection:"$XDG_DATA_DIRS"
exec /not/in/PATH/pandoc "$@"

[alerque]

The argument for enabling nix to be a usable way to scope project development environments is a good point here.

[pbsds]

My shell.nix currently looks like this:

{ pkgs ? import <nixpkgs> {} }:

with pkgs; mkShell {
  buildInputs = [
    graphviz
    (pkgs.writeShellScriptBin "pandoc" ''
      exec env XDG_DATA_HOME=${pandoc-lua-filters}/share ${pkgs.pandoc}/bin/pandoc "$@"
    '')
    pandoc-imagine
    nodePackages.mermaid-cli
    haskellPackages.pandoc-crossref
    pandoc-plantuml-filter
    ( texlive.combine {
      inherit (texlive)
        scheme-small
        titlesec
        fontaxes
        supertabular
        xtab
        atkinson;
    }
  )
  ];
}

It is possible to join the lua filter with other filters using symlinkJoin, but using this solution makes it so any user-defined filters in $HOME will not be found.

[jgm]

Adding link to specification:
https://specifications.freedesktop.org/basedir-spec/0.6/ar01s03.html

[0xdevalias]

Adding link to specification:
specifications.freedesktop.org/basedir-spec/0.6/ar01s03.html

Since that link 404's now, here is the modern link:

• https://specifications.freedesktop.org/basedir-spec/latest/

  • Publication Date: 08th May 2021, Version: Version 0.8

  • There is a set of preference ordered base directories relative to which data files should be searched. This set of directories is defined by the environment variable $XDG_DATA_DIRS.

    There is a set of preference ordered base directories relative to which configuration files should be searched. This set of directories is defined by the environment variable $XDG_CONFIG_DIRS.

  • $XDG_DATA_DIRS defines the preference-ordered set of base directories to search for data files in addition to the $XDG_DATA_HOME base directory. The directories in $XDG_DATA_DIRS should be separated with a colon :.

    If $XDG_DATA_DIRS is either not set or empty, a value equal to /usr/local/share/:/usr/share/ should be used.

    $XDG_CONFIG_DIRS defines the preference-ordered set of base directories to search for configuration files in addition to the $XDG_CONFIG_HOME base directory. The directories in $XDG_CONFIG_DIRS should be separated with a colon :.

    If $XDG_CONFIG_DIRS is either not set or empty, a value equal to /etc/xdg should be used.

  • A specification that refers to $XDG_DATA_DIRS or $XDG_CONFIG_DIRS should define what the behaviour must be when a file is located under multiple base directories. It could, for example, define that only the file under the most important base directory should be used or, as another example, it could define rules for merging the information from the different files.

[0xdevalias]

I believe the suggestions here would also provide a clean solution to:

• How to make Lua filters available system-wide in Pandoc? #10392