latexmk -output-directory=fullpath -cd causes makeindex to fail

[phdoerfler]

Preliminary questions

Disable all the other extensions except for LaTeX Workshop, and check that you still see this issue.

As far as I can see this is completely unrelated to other extensions.

Make sure to visit the wiki FAQ before filling an issue.

You visited the wiki? Yes

If your issue is with compiling a document (not having to do with finding the root file of a project), check first that you can compile manually.

You can compile a TeX document manually? Yes

Describe the bug

I can compile a TeX document manually using this command line:

latexmk -synctex=1 -interaction=nonstopmode -file-line-error -pdf -outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation

Clean auxiliary files and add -cd (which is what LaTeX-Workshop does) and it no longer builds:

latexmk -synctex=1 -interaction=nonstopmode -file-line-error -pdf -outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation -cd /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation

In both cases I invoked this from the directory where .tex file resides and which is also the outdir.
Unfortunately LaTeX-Workshop uses the -cd argument of latexmk.

Expected behavior

I expect LaTeX-Workshop to not use the -cd argument, but to invoke latexmk with the correct working directory set.
child_process.spawn of node.js allows to specify the cwd.

Logs

LaTeX Workshop Output

https://gist.github.com/phdoerfler/e38fa3bb2a5fe189e31351317dd15a77

Developer Tools Console

I don't see anything suspicious there, but here you go:

/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3930 Overwriting grammar scope name to file mapping for scope source.scala.
Old grammar file: file:///Users/phi/.vscode/extensions/daltonjorge.scala-0.0.5/syntaxes/Scala.tmLanguage.
New grammar file: file:///Users/phi/.vscode/extensions/scala-lang.scala-0.3.9/syntaxes/Scala.tmLanguage.json
register @ /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:3930
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] activating extension
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] starting language server
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] (node:33399) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
t.log @ /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] Creating server
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] [vscode-icons] v9.7.0 activated!
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] runCommand platformio Array(1) Object
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:254 [Extension Host] Found PIO Core 4.1.0

Desktop

• OS: macOS Catalina 10.15.3 (19D76)
• VS Code version: 1.41.1
• Extension version: 8.7.1
• TeX distribution version: MacTex 2019.0508 (brew cask install mactex)

Additional questions

Are you using LaTeX Workshop with VS Code Remote?

No

[jlelong]

FYI, we also use the cwd of spawn, so the -cd option should not matter.
From your logs, the error is related to makeindex

makeindex: Not writing to /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation.ind (openout_any = p).
Can't create output index file /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation.ind.

Also, your command line

latexmk -synctex=1 -interaction=nonstopmode -file-line-error -pdf -outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation

seems incomplete. The file to compile /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation is missing.

I cannot reproduce the issue, please provide a MWE.

[jlelong]

It is related to makeindex not accepting absolute file paths. From the latexmk documentation

This is because modern versions of these programs [bibtex and makeindex], by default, will refuse to work when they find that they are asked to write to a file in a directory that appears not to be the current working directory or one of its subdirectories. This is part of security measures by the whole TeX system that try to prevent malicious or errant TeX documents from incorrectly messing with a user’s files. If for $out_dir or $aux_dir you really do need to specify an absolute pathname (e.g., "/tmp/foo") or a path (e.g., "../output") that includes a higher-level directory, and you need to use makeindex or bibtex, then you need to disable the security measures (and assume any risks). One way of doing this is to temporarily set an operating system environment variable openout_any to "a" (as in "all"), to override the default "paranoid" setting.

Consider the following main-index.tex (coming from cluttex)

\documentclass{article}
\usepackage{makeidx}
\usepackage{hyperref}
\makeindex
\begin{document}
\section{Polynomial}
Polynomial\index{polynomial}

\section{Resultant}
Resultant\index{resultant}

%GCD\index{GCD}

\printindex
\end{document}

Compiling with latexmk fullpath/to/main-index with or without the -cd option works fine

Compiling with latexmk -output-directory=dirname/to/main-index -cd fullpath/to/main-index fails with

Running 'makeindex  -o "/Users/jl/devel/LaTeX-Workshop/test/1932/main-index.ind" "/Users/jl/devel/LaTeX-Workshop/test/1932/main-index.idx"'
------------

makeindex: Not writing to /Users/jl/devel/LaTeX-Workshop/test/1932/main-index.ind (openout_any = p).
Can't create output index file /Users/jl/devel/LaTeX-Workshop/test/1932/main-index.ind.

[phdoerfler]

FYI, we also use the cwd of spawn, so the -cd option should not matter.

Great! So why not just omit the -cd option?
I can only reiterate that omitting the -cd option does fix the issue on my machine, as unexpected as this may be.

From your logs, the error is related to makeindex

Yes, perhaps the -cd option affects makeindex in some way.

Also, your command line […] seems incomplete. The file to compile […] is missing

Yes! This was my first reaction as well. But it's what LaTeX-Workshop is doing. If you look in the linked gist you will find this line near the top:

[15:43:31] Recipe step 1: latexmk, -synctex=1,-interaction=nonstopmode,-file-line-error,-pdf,-outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation,-cd,/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation

Turns out: it does not matter whether you add documentation.tex to that command line or not.
In order to mimic the extension's behaviour as best as possible I, too, omitted the file name.

Between the two command lines I have cleaned all auxiliary files using the extension. This is crucial, because otherwise latexmk will pick up the previous compilation and succeed even when specifying -cd.

[jlelong]

Your analysis of the latexmk command line is wrong. The -cd option of latexmk does not take any argument. So, in

[15:43:31] Recipe step 1: latexmk, -synctex=1,-interaction=nonstopmode,-file-line-error,-pdf,-outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation,-cd,/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation

the last argument /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation is the file to compile. Note that this is an absolute path.

Here are some more tests

• latexmk -output-directory=/dirname/to/main-index fullpath/to/main-index works
• latexmk -output-directory=/dirname/to/main-index -cd fullpath/to/main-index fails
• latexmk -output-directory=/dirname/to/main-index -cd main-index works
• latexmk -cd fullpath/to/main-index works
• latexmk -cd main-index works
• latexmk -output-directory=/tmp main-index fails.

Maybe, we could get in touch with the latexmk author to discuss the case

latexmk -output-directory=/dirname/to/main-index -cd fullpath/to/main-index

which should work

[jlelong]

Great! So why not just omit the -cd option?

See #1895 (comment)

[phdoerfler]

Your analysis of the latexmk command line is wrong. The -cd option of latexmk does not take any argument. So, in

[15:43:31] Recipe step 1: latexmk, -synctex=1,-interaction=nonstopmode,-file-line-error,-pdf,-outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation,-cd,/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation

the last argument /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation is the file to compile. Note that this is an absolute path.

Indeed, my bad! Seems as if I have mistaken the documentation part of the filename/pathname as the directory and not the tex file, probably because it is missing its .tex extension as it is so common in latex land.
And in my head -cd $someDirectory made an awful lot of sense. Sorry for the confusion. To be fair, invoking latex produces a metric ton of logs and it's very easy to get lost in there. Thanks for your patience!

Maybe, we could get in touch with the latexmk author to discuss the case

latexmk -output-directory=/dirname/to/main-index -cd fullpath/to/main-index

which should work

That is probably a good idea. This contact info appears to be up to date:
http://personal.psu.edu/jcc8//software/latexmk-jcc/

Great! So why not just omit the -cd option?

See #1895 (comment)

I see, thanks! Now knowing that -cd does not take any arguments, this makes a lot of sense.

[phdoerfler]

Also, your command line

latexmk -synctex=1 -interaction=nonstopmode -file-line-error -pdf -outdir=/Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation

seems incomplete. The file to compile /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation is missing.

FWIW: While I was wrong to assume that LaTeX-Workshop was also omitting the filename, I was correct in that the command does work despite missing the name of the tex file to compile. Invoking the command eventually results in this output:

Output written on /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/
documentation.pdf (88 pages, 18270575 bytes).
SyncTeX written on /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentation/documentation.synctex.gz.
Transcript written on /Users/phi/Documents/workspace/FAUGitlab/spiht/Documentat
ion/documentation.log.
=== TeX engine is 'pdfTeX'
Latexmk: Index file 'documentation.idx' was written
Latexmk: Found input bbl file 'documentation.bbl'
Latexmk: Found input bbl file 'documentation.bbl'
Latexmk: Log file says output to 'documentation.pdf'
Latexmk: Found biber source file(s) [bib/spiht.bib documentation.bcf]
Latexmk: All targets () are up-to-date

This is after having cleaned all auxiliary files.
Here's a short recording:
https://asciinema.org/a/4Ad1UmuAveLhz9VkkE4z5bI5y

[jlelong]

I wrote an email to the author of latexmk. I will post his answer here.

[jlelong]

The author of latexmk considers it as a bug and he will fix in the next release.
Meanwhile, you can edit the recipe and remove the -cd option if you do not need it.

[phdoerfler]

Thanks for the follow up!

[PsyKoMari]

@jlelong

Meanwhile, you can edit the recipe and remove the -cd option if you do not need it.

Sorry, I read the docs regarding how to edit the default recipes but I can't understand where I've to remove the -cd option , I search latex-workshop.latex.recipes, then I see edit on settings.json and there's nothing, can you please help me and explain how to do this?

[jlelong]

For some strange reason, the Edit on settings.json does not work. You have to copy the setting as JSON using the wheel on the left to latex-workshop.latex.tools. Then paste it into settings.json and edit the latexmk entry.

I agree that it is a little too complicated and we should think of an other solution.

[jlelong]

Rethinking of a ll this, it is wise to remove -cd option and as it can be more easily added than removed. Adding it just amounts to creating a .latexmkrc file containing $do_cd = 1;.