Package management fails to read files when lock directory is ignored

[mefyl]

Expected Behavior

dune build properly installs ocamlfind and proceeds with building the package.

Actual Behavior

Same error on my host machine (Gentoo) and docker Alpine or Ubuntu images :

$ dune runtest       
 Downloading ocaml-base-compiler.5.2.0
    Building ocaml-base-compiler.5.2.0
    Building ocaml-config.3       
    Building ocaml.5.2.0           
 Downloading ocamlfind.1.9.6+dune  
    Building ocamlfind.1.9.6+dune  
Error:                             
_build/.sandbox/47ba9c2fecf90210515b55576e20e1a6/_private/default/.pkg/ocamlfind/source/0001-Harden-test-for-OCaml-5.patch:
No such file or directory
-> required by _build/_private/default/.pkg/ocamlfind/target/cookie
-> required by Computing closure for package "ptime"
-> required by %{lib-available:js_of_ocaml} at timmy-jsoo/tests/dune:12
-> required by Computing directory contents of
   _build/default/timmy-jsoo/tests
 Downloading ocamlbuild.0.15.0+dune          
    Building ocamlbuild.0.15.0+dune          
$ ls dune.lock/ocamlfind.files/0001-Harden-test-for-OCaml-5.patch # Seems to be there
dune.lock/ocamlfind.files/0001-Harden-test-for-OCaml-5.patch
$ 

Reproduction

Reproduction Dockerfile:

FROM alpine

RUN apk add bash curl git make npm gcc g++ tzdata
RUN curl -fsSL https://get.dune.build/install | sh
ENV PATH="/root/.dune/bin:$PATH"
WORKDIR /root
# Tested on timmy but same behavior on any project I test.
RUN git clone https://github.com/mefyl/timmy
RUN git config --global --add safe.directory /root/.cache/dune/git-repo
WORKDIR /root/timmy
RUN dune pkg lock
RUN dune build

Specifications

• Version of dune (output of dune --version): 3.16.0
• Version of ocaml (output of ocamlc --version): 5.2.0

[gridbugs]

This is the same error message as in #10903, however it looks like a different issue.

[Leonidas-from-XIV]

The error is actually that the project ignores dune.lock, thus when ocamlfind tries to read the patch, it doesn't see it (this was investigated by @gridbugs yesterday). I'll update the issue name.

Repro in #11115.

Two options exist (discussed in todays dune-dev meeting):

1. Ignoring the lock directory should always be an error and bail out.
2. Reading files in package management should not be affected by dirs.

The first option has the issue that the lock directories are defined in the dune-workspace, thus if we error on one project, it might not error on another because another user has a different workspace configuration. Thus I lean towards 2. @rgrinberg, what's your take on this?

[rgrinberg]

I think ignoring the dirs configuration in this specific case would be confusing. Isn't it better to just tell the user that dune.lock doesn't exist or is hidden by dirs?

[Leonidas-from-XIV]

The issue is that if I have a dune-workspace that puts the lock dir into linux.lock and add (dirs <my-folders> linux.lock) into the dune file, it would work for me. But if someone else then clones the project and locks it (thus creating the lock directory in dune.lock) it would suddenly fail.

We can make the error message when it fails, but it seems fragile if the same project works on one setup and fails on another.

[rgrinberg]

If there's a workspace file, I think it should be considered part of the setup. There's a bunch of other settings one can write in the workspace file that might break things if the workspace is omitted.

I'm not strongly against making the source tree special case the lock directory, but it doesn't seem like an elegant solution either. I'd start with a good error message and if it proves to be insufficient, we could always revisit.