Inconsistent UI: --filter searches the path but --lua-filter does not

[niclasmattsson]

This could be a Windows issue, I haven't tested this on Linux or Mac. But on my Windows machine, this is how pandoc filters behave:

C:\Stuff>where testfilter*
C:\Stuff\testfilter2.lua
C:\Stuff\Scripts\testfilter1.exe
C:\Stuff\Scripts\testfilter3.lua

C:\Stuff>pandoc test.md --filter testfilter1 -s -o test.pdf

C:\Stuff>pandoc test.md --lua-filter testfilter2.lua -s -o test.pdf

C:\Stuff>pandoc test.md --lua-filter testfilter3.lua -s -o test.pdf
Error running filter testfilter3.lua:
testfilter3.lua: openBinaryFile: does not exist (No such file or directory)

C:\Stuff>pandoc test.md --lua-filter Scripts/testfilter3.lua -s -o test.pdf

C:\Stuff>

This isn't entirely undocumented. For example, running pandoc --help returns this:

C:\Stuff>pandoc --help
[...]
  -F PROGRAM            --filter=PROGRAM
                        --lua-filter=SCRIPTPATH
[...]

But I didn't find any direct mention of this in the --lua-filter README. In any case, I think the solution is not to improve the docs but to make the behavior more consistent, i.e. have --lua-filter search the path just like --filter does.

Potentially related filter path issues (all on Windows): #3563 , #3458, #3247.

[jgm]

Not clear that it makes sense to make a lua filter executable and put it on the path.

Unlike regular filters, which read from stdin and write to stdout, lua filters don't really serve as standalone programs. If you run one, it will do exactly nothing. They only do their filtering magic when interpreted by pandoc.

[niclasmattsson]

Thanks, I understand that rationale. But out of curiosity, what was the reason for the design decision to not have pandoc lua filters work as standalone filters in the same way as ordinary filters? In systems that don't have a standalone Lua interpreter the filters would still do nothing, but they could potentially do useful double duty on systems that have the interpreter.

[jgm]

niclasmattsson <[email protected]> writes:

Thanks, I understand that rationale. But out of curiosity, what was the reason for the design decision to not have pandoc lua filters work as standalone filters in the same way as ordinary filters? In systems that don't have a standalone Lua interpreter the filters would still do nothing, but they could potentially do useful double duty on systems that have the interpreter.

The architecture is completely different. With regular filters, pandoc translates its AST into JSON, then we shell out to the filter, which reads the JSON, transforms it, and writes new JSON, and then pandoc translates this back into its AST. With lua filters we don't shell out at all. Pandoc embeds a lua interpreter and includes built-in marshalling from the Haskell AST to lua values. For this reason, the lua filters are MUCH more efficient (see the stats at the beginning of the lua filters documentation).

[niclasmattsson]

I see, thanks for explaining!