DataDeps.jl behavor in CI when a file cannot be read

[ali-ramadhan]

I've been encountering some intermittent hangs on some CI builds such as

[2021/05/19 03:44:30.384] WARN  DataDep cubed_sphere_32_grid found at "/data5/glwagner/.julia-2613/datadeps/cubed_sphere_32_grid". But could not read file at "/data5/glwagner/.julia-2613/datadeps/cubed_sphere_32_grid/cubed_sphere_32_grid.jld2". -@-> /data5/glwagner/.julia-2613/packages/DataDeps/ooWXe/src/resolution.jl:35
Something has gone wrong. What would you like to do?
 
 [A] Abort -- this will error out
 [R] Retry -- do this after fixing the problem outside of this script
 [X] Remove directory and retry  -- will retrigger download if there isn't another copy elsewhere
 
[A/R/X]

But I can't send any input so I'm forced to restart the job or build.

I was wondering whether it made sense to pick a default behavior when ENV["CI"]="true" or provide an additional environment variable like DATADEPS_CANNOT_READ_FILE that can take on the values "abort", "retry", or "remove_and_retry".

Perhaps my real issue is a race condition but I thought it might be worth opening an issue about this too.

[oxinabox]

Ah yeah, sounds like you had a race.
Maybe can fix that by making sure the datadep is loaded before either that uses it starts.
by doing something like assert(isdir(datadep"cubed_sphere_32_grid") at that of the test script.

On the surface an enviroment variable makes sense, but other than just abortinging i think no other options make sense.
Since really what you want to do is something like "Wait 10 seconds, then retry".

[aviks]

Yeah, I think an option that forces an abort rather than asking a question would be very useful here. I find DataDeps very useful in non-interactive contexts as well.

[oxinabox]

We can probably duplicate this logic and introduce a DATADEPS_ABORT_ON_NO_READ env var.
So that in CI it defaults to aborting, and out of CI it defaults to not aborting.