Tests fail locally on Windows

[pkch]

Locally, I am getting:

FAILURE #17 check stubs

typeshed\stdlib\3\ntpath.pyi:1: error: Name 'posixpath' is not defined
typeshed\stdlib\3\macpath.pyi:1: error: Name 'posixpath' is not defined

The entire contents of each of those files is just posixpath.pyi.

But somehow everything passes on CI.

[gvanrossum]

Check the recent PR that fixes symlinks. There is a git setting.

[gvanrossum]

In particular #3213.

We need to add instructions for developers on Windows to README.md. Once those are there (and you've got it working for yourself) we can close this.

[pkch]

So far my experience is that it works as long as the user has permissions for creating symlinks.

Theoretically, the permissions should be already enabled for anyone with the current version of Windows 10 running in developer mode. In practice, it only worked for me when I ran git clone from the Administrator command prompt. (I followed instructions from here and it didn't help.)

I'll let someone else test this out before closing this issue.

[gvanrossum]

Heh. I had no idea that Windows even supported symlinks at all. I have no idea what I was thinking the fix did (I think I believed it made git interpret the symlinks itself. :-) Neat.

I have no way to test this myself so I'll keep it open.

[pkch]

To whoever plans to test it more:

It seems there's a glitch with Microsoft implementation of permissions which means that if you are a member of an Administrator group, you will not be able to use symlinks from a regular command prompt, even if you've followed all the instructions (such as setting the right group policy or enabling Developer mode in Windows 10).

These workaround have been proposed:

1. remove yourself from the Administrator group
2. run all git commands as a non-Adminstrator user (using runas)
3. use Adminstrator (elevated) instead of a regular command prompt (yes, it's counter-intuitive)

I only tested option (3), and it worked for me.

I have been unable to convert an existing repo to a repo with symlinks; I had to make a new one. Before deleting / overwriting the old repo, make sure you are ok with losing:

• unreachable commits (and reflog history itself)
• unpushed commits in branches
• uncommited changes in the working directory, including untracked or ignored file
• probably some other stuff like local-only tags, etc.

[JelleZijlstra]

Thanks for looking into this. Since the symlinks are only in the typeshed submodule, do you really have to fully delete your old repo? I think you should be able to just rm -rf typeshed/ and then again git submodule init/git submodule update.

[pkch]

I thought we also have one in mypy:

./test-data/.flake8@ -> ../typeshed/.flake8

It's less important than the others. If we remove it, and make sure to never add symlinks to mypy itself, what you said should work.

It's only worth it if we think future contributors will first clone the repo without RTFM, and only then start looking things up when they notice tests don't work :)

[gvanrossum]

Yeah, let's remove the one symbolic link in the mypy repo itself.

[pkch]

In addition, I found that to switch between commits (using git checkout) that have any difference in symlinks (e.g., one is added or deleted), one needs to be using Administrator command prompt on Windows (or else Git gets confused and git checkout deletes the file instead of updating the symlink -- at least on my version of Windows 10).

Not a problem, as long as it's documented.

[gvanrossum]

This worries me a bit.

I wonder if a solution that requires one to do many crucial Git manipulations as Administrator isn't too brittle. Maybe we need to reopen python/typeshed#1175 ?

[JukkaL]

I'm also worried about this. Let's reconsider the use of symlinks on Windows?

…

On Sun, Apr 30, 2017 at 05:03 Guido van Rossum ***@***.***> wrote: This worries me a bit. I wonder if a solution that requires one to do many crucial Git manipulations as Administrator isn't too brittle. Maybe we need to reopen python/typeshed#1175 <python/typeshed#1175> ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3222 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDnx6ZFaRA3H4qosPaw2hqLsoNgDxJHks5r1AgQgaJpZM4NFIgV> .

[JelleZijlstra]

Agree that this seems like it is going to be too brittle.

Is it correct that these issues will only affect people developing on mypy, not people just using it? If the latter, we should definitely get this fixed before the next mypy release.

I can try implementing one of the other solutions, but maybe it's better if somebody who actually develops on Windows does it. @pkch are you interested in working on an alternative solution to typeshed symlinks?

[gvanrossum]

People using mypy typically install the wheel. I just downloaded the 0.501 win32 wheel and it contains no symlinks (even though there were some in typeshed at the time of release) so I think these should be safe.

However we frequently recommend people who report a bug to try out master. So if they're on Windows they may be hurt by this.

[pkch]

@JelleZijlstra Yes, I can - as long as it's the solution 4 (which everyone seems to prefer). Solution 2 has too many caveats both in implementation and in use, I'll never be sure I got it right as either the implementor or the user.

That said, should we perhaps confirm that I'm not the only one experiencing the need for Administrator rights to use the current symlink approach? Maybe something is just wrong with my personal setup. It would be a shame to give up on the official git solution just because of that.

[gvanrossum]

Hm, Administrator rights feature prominently in the StackOverflow "solutions" as well. It seems that on recent versions of Windows the filesystem supports symlinks, but the kernel (or some other layer between the filesystem and the user-mode code) forbids them unless you're Administrator, presumably because they're such a novelty that they would confuse most Windows users. :-)

[toolness]

So it looks like support for creating symlinks as a non-administrator was added as part of the Windows 10 Creators Update, which was just released (I updated my system to it a few hours ago). It requires enabling developer mode, but I've confirmed that running mklink (Windows' analog of unix's ln -s) now works without administrative privileges.

However, Git for Windows still doesn't work properly without non-admin privileges. This is because, as the Windows blog post asserts (and a hacker news thread provides a bit more insight on), code that wants to use this new non-admin functionality needs to opt-in to it via a new SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag when using Windows' CreateSymbolicLink API.

It looks like Git for Windows literally just merged support for this a few hours ago in git-for-windows/git#1188, though! So the good news is that hopefully it will work OK once a new version of Git for Windows has been released... and developers have enabled Windows 10 Developer Mode... and maybe also made sure they've run git config core.symlinks true... then I think everything might work properly.

[JukkaL]

Requiring a very recent Windows and git releases for mypy development would be problematic. Some potential contributors use company laptops where they may not be able to update things freely. Some internal tools might require a specific version of Windows and/or git, for example.

[toolness]

Ah, good point! Yeah, I guess in that case it might be good to avoid requiring symlinks...

[emmatyping]

This was fixed in python/typeshed#2132