What is our Pkg3 Story?

[oxinabox]

Pkg3 introduces the idea of projects,
which are a lot like having a separate julia environment (in terms of ~/julia/vX.y) each.
And you are supposed to be able to just git clone them.
They are like self-contained environments

My first thoughts were: Oh, that is going mean big changes to DataDeps.jl,
in terms of where we should (by default) store stuff, like we casually break any closed environment since the default search path search's everywhere, and the default savepath is ~/.julia/datadeps
But on further thought I don't think it will.
DataDeps.jl exists so that data does not have to be part of a project, or in any particular location.
It will just sort out the location itself.

Currently the load path looks like this

• <Pkg>/deps/data (This one is special that it is never the savepath, and is only visible calls made from files within that package)
• ENV["DATADEPS_LOADPATH] (this expands out it if is :s)
• ~/.julia/datadeps or equiv, (this and everything below can be disabled by setting DATADEPS_NO_STANDARD_LOAD_PATH])
• User areas eg ~/.julia/data
• Local computer ares e.g. C:/ProgramData, /usr/local/datadeps
• network areas e.g. /usr/share/datadep

The loadpath starts from the top and search's to the to the bottom until it finds a data directory with the right name.
If it fails then it downloads it,
and when downloading: it looks for the first location that exists and is writable, starting from the top (skipping the Pkg specific directory),
and going down.

Stuff ends up in the Pkg specific directory if it was a ManualDataDep that is included in the repo.
Or if it the user moved the data directory their to deal with a conflict.
Conversely stuff ends up in the lower locations on the load path, if the user moves them down, so that they can be shared e.g. my other students in the group.

[oxinabox]

I think the go is to add a project specific specific directory like the package specific directory.
It would be visible only when that project is "active".
It would be checked second after the package specific directory,
and like the package specific directory it wouldn't be a possible automatic save location.
But users could move things here to avoid conflicts, and ManualDataDeps could be placed here to be downloaded with the project repo.

The reason for it not to be an automatic save location is primarily that data can be big. I have like 3 projects that all use the same ~10GB chunck of data.

[Evizero]

What may be worth considering is adopting the depot concept of Pkg3 (see https://docs.julialang.org/en/v1/stdlib/Pkg/#Glossary-1). I.e. looking for a datadeps subfolder in the known depots in the appropriate order.

EDIT: oh it seems it does already

[oxinabox]

I need to update this. It still isn't resolved properly.
I don't think. But the state us not the same as it was for julia 0.6 when above was written