Julia issue with new OneDrive placeholder files in Windows Fall Creator Update

[affans]

I am using Julia 0.6. Recently, I upgraded my computer to Fall Creator's update, which introduced Files on Demand in explorer for OneDrive. My entire julia work is on OneDrive and I have explicitly downloaded all files and enabled "Always keep on this device" for my folder.

However, opening a file in Julia REPL gives a weird error; for example, for example using "tab" to autocomplete.

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0 (2017-06-19 13:05 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> pwd()
"E:\\OneDrive\\Documents\\phd_project\\hia_code"

julia> inclue("maERROR (in the keymap): stat: invalid argument (EINVAL)
Stacktrace:
 [1] stat(::String) at .\stat.jl:69
 [2] isdir at .\stat.jl:279 [inlined]
 [3] completions(::String, ::Int64) at .\REPLCompletions.jl:461
 [4] complete_line(::Base.REPL.REPLCompletionProvider, ::Base.LineEdit.PromptState) at .\REPL.jl:289
 [5] complete_line(::Base.LineEdit.PromptState, ::Int64) at .\LineEdit.jl:144
 [6] (::Base.LineEdit.##91#121)(::Base.LineEdit.MIState, ::Base.REPL.LineEditREPL, ::Vararg{Any,N} where N) at .\LineEdit.jl:1351
 [7] (::Base.LineEdit.##13#14{Base.LineEdit.##91#121,String})(::Base.LineEdit.MIState, ::Base.REPL.LineEditREPL) at .\LineEdit.jl:740
 [8] prompt!(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface, ::Base.LineEdit.MIState) at .\LineEdit.jl:1618
 [9] run_interface(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface) at .\LineEdit.jl:1578
 [10] run_frontend(::Base.REPL.LineEditREPL, ::Base.REPL.REPLBackendRef) at .\REPL.jl:945
 [11] run_repl(::Base.REPL.LineEditREPL, ::Base.##509#510) at .\REPL.jl:180
 [12] _start() at .\client.jl:413
julia>
julia>

As you can see I am trying to autocomplete include("main.jl") but it throws that error.

This does not happen if using a local folder or Dropbox. It also breaks other code, for example, the julia-lang plugin for VS Code.

Here is the similar error for julia-lang plugin:

ERROR: LoadError: stat: invalid argument (EINVAL)
Stacktrace:
 [1] stat(::String) at .\stat.jl:69
 [2] isdir at .\stat.jl:279 [inlined]
 [3] #walkdir#12(::Bool, ::Bool, ::Core.#throw, ::Function, ::String) at .\file.jl:471
 [4] process(::LanguageServer.JSONRPC.Request{Val{:initialized},Dict{String,Any}}, ::LanguageServer.LanguageServerInstance) at C:\Users\affan\.vscode\extensions\julialang.language-julia-0.7.0\scripts\languageserver\packages\LanguageServer\src\provider_misc.jl:41
 [5] run(::LanguageServer.LanguageServerInstance) at C:\Users\affan\.vscode\extensions\julialang.language-julia-0.7.0\scripts\languageserver\packages\LanguageServer\src\languageserverinstance.jl:52
 [6] include_from_node1(::String) at .\loading.jl:569
 [7] include(::String) at .\sysimg.jl:14
 [8] process_options(::Base.JLOptions) at .\client.jl:305
 [9] _start() at .\client.jl:371
while loading C:\Users\affan\.vscode\extensions\julialang.language-julia-0.7.0\scripts\languageserver\main.jl, in expression starting on line 27
[Info  - 13:55:52] Connection to server got closed. Server will restart.

Here is some more info:
I laucnhed Julia from the desktop and then tried including the full path to my OneDrive file.

julia> include("E:\\OneDrive\\Documents\\phd_project\\hia_code\\main.jl")
ERROR: could not open file E:\OneDrive\Documents\phd_project\hia_code\main.jl
Stacktrace:
 [1] include_from_node1(::String) at .\loading.jl:569
 [2] include(::String) at .\sysimg.jl:14

Basically I don't think Julia can see anything inside the new "OneDrive" folder.

[ChrisRackauckas]

This isn't a Julia issue. Files on Demand (like they had before Windows 10... it's not a new feature for them they just removed it for awhile) don't exist on your machine by default, so of course Julia will fail when you try to use them. So will MATLAB, Python, R, etc. Instead, if you want to use a file, you need explicitly open it to cause the download or, if you use this folder a lot, tell it to make that folder local.

[musm]

@ChrisRackauckas he does mention that

My entire julia work is on OneDrive and I have explicitly downloaded all files and enabled "Always keep on this device" for my folder.

So it seems something else is going on

[affans]

Right, the files are locally available. I think its a issue with Julia and Windows together.

[StefanKarpinski]

Obvious question: can you try this in Python or R or something (to see if the same issue exists)?

[affans]

@StefanKarpinski I don't have those installed on my machine (who needs em when you got Julia 💯), but I guess I can try with R. I have used, LaTeX successfully though. I am able to go to the command line, navigate to OneDrive folder, and run pdflatex. And I havn't had any other problems with other software - mostly Office and PDF readers (Sumatra and UWP app Xudo)

I will update my post when I run the R/Python test.

[nalimilan]

A good test could be to read the contents of a text file in that directory from different languages.

[ihnorton]

OneDrive uses non-symlink NTFS "reparse points", which apparently breaks everyone's stuff (including Microsoft's stuff!).

See nodejs/node#12737 (and microsoft/vscode#27285 and microsoft/WSL#1941 (comment))

Fixed by: libuv/libuv#1522
(ref also libuv/libuv#1419 which has incomplete fix).

[ihnorton]

(one suggested work-around I saw somewhere: use a normal local folder, and symlink that from within the OneDrive ... supposedly it will be synced)

[christopher-dG]

I've encountered a similar issue with symlinks on Windows when using WSL. If I create a symlink via symlink in Julia, I can tab-complete the link just fine, but if I create a symlink in WSL i.e. touch foo; ln -s foo bar, then trying to tab-complete bar gives me the same error as above.

e: I just tried opening the link in Python and got the invalid argument error so I suppose this isn't Julia's fault (but I think it's still relevant to the issue being raised).

[ihnorton]

@christopher-dG thanks, that could be useful to create a regression test.

Unfortunately our libuv branch has diverged a bit from upstream, but I see there are some open/recent PRs to update libuv... @ararslan @vtjnash -- is libuv/libuv#1522 included in recent updates?

[ararslan]

The libuv commit that we use was updated recently—does this happen on current master? If this is no longer a problem on current master then we should be okay; the libuv update is being backported to 0.6.2. Otherwise I can backport the upstream libuv change to our fork then incorporate that here.

[ihnorton]

libuv/1522 changes are definitely not in the current libuv commit used on master. So, I wouldn't expect this to work on master either.

It looks like our fork is based off of something like libuv 1.9, but upstream is at 1.16, and there have been some changes.

[ararslan]

Backporting to our fork (https://github.com/JuliaLang/libuv/pull/47), once that's merged I can submit a PR that will update the commit used on master here, then backport that to 0.6.2.

[affans]

@ihnorton sorry but what is libuv and how is Julia related to it?

[ararslan]

https://en.wikipedia.org/wiki/Libuv we use it as part of Julia

[affans]

@StefanKarpinski Do you think its a good idea to fix this for the 0.7/1.0 release?

[ararslan]

It's a bugfix, so it can happen at any time and be backported to affected versions

[ViralBShah]

Please reopen if still an issue.