Pip install Error on windows("The directory is not empty")

[nandak522]

I was trying to install lot many packages with pip(on Windows) and sometimes(only "sometimes") I get the below error.

File "my_hudson_workspace_path\lib\site-packages\pip-1.0-py2.6.egg\pip\basecommand.py", line 126, in main
self.run(options, args)
File "my_hudson_workspace_path\lib\site-packages\pip-1.0-py2.6.egg\pip\commands\install.py", line 228, in run
requirement_set.install(install_options, global_options)
File "my_hudson_workspace_path\lib\site-packages\pip-1.0-py2.6.egg\pip\req.py", line 1102, in install
requirement.remove_temporary_source()
File "my_hudson_workspace_path\lib\site-packages\pip-1.0-py2.6.egg\pip\req.py", line 610, in remove_temporary_source
rmtree(self._temp_build_dir)
File "my_hudson_workspace_path\lib\site-packages\pip-1.0-py2.6.egg\pip\util.py", line 29, in rmtree
onerror=rmtree_errorhandler)
File "c:\Python26\Lib\shutil.py", line 216, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "c:\Python26\Lib\shutil.py", line 216, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "c:\Python26\Lib\shutil.py", line 216, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "c:\Python26\Lib\shutil.py", line 216, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "c:\Python26\Lib\shutil.py", line 216, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "c:\Python26\Lib\shutil.py", line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "c:\Python26\Lib\shutil.py", line 223, in rmtree
os.rmdir(path)
WindowsError: [Error 145] The directory is not empty: 'e:\temp\pip-sr1bjf-build\build\Pygments\tests\examplefiles\output'

Same error is coming with a few different packages also:
WindowsError: [Error 145] The directory is not empty: 'e:\temp\pip-cor26d-build\build\pytz\pytz\zoneinfo\Africa
WindowsError: [Error 145] The directory is not empty: 'e:\temp\pip-tyqd6c-build\build\SQLAlchemy\doc\build\templates'

Is this an issue with Pip or something related to my environment.

[carljm]

Hard to say unless you can track it down to reliable steps to reproduce.

[pnasrat]

This is could be an issue that a file in the directory is locked, I've seen it happen with CI servers before and folks resort to using unlocker, etc.

You can use handle.exe from sysinternals to check which process currently have open handles for the file

http://technet.microsoft.com/en-us/sysinternals/bb896655

If you can give more details on what is keeping the file open we might be able to help. We probably should clean up the exception handling and ensure we report an error not a traceback to the user.

[jfilipe]

Had this happen to me, and it was due to having a the build directory open in a window. Closing it resolved the issue.

[paulftw]

Happens to me all the time when I run buildout in the Dropbox folder.

[ptthiem]

This might be similar an issue I was having with installing pip. I had to:

diff -ru pip-1.2.1.orig/pip-1.2.1/setup.py pip-1.2.1/pip-1.2.1/setup.py
--- pip-1.2.1.orig/pip-1.2.1/setup.py    2012-09-01 15:28:40.000000000 -0500
+++ pip-1.2.1/pip-1.2.1/setup.py    2013-01-10 22:35:44.000000000 -0600
@@ -4,10 +4,10 @@
 import sys
 from setuptools import setup

-
 def read(*parts):
-    return codecs.open(os.path.join(os.path.abspath(os.path.dirname(__file__)), *parts), 'r').read()
-
+    with codecs.open(os.path.join(os.path.abspath(os.path.dirname(__file__)), *parts), 'r') as f:
+        return f.read()

 def find_version(*file_paths):
     version_file = read(*file_paths)

I'll have to unpatch to see that actual error was.

[pfmoore]

I had this happen a lot when installing pip as part of a virtualenv creation - I always put it down to virus checkers temporarily holding some file open (mainly because it was intermittent). But this may be the real reason. I'll try testing on my slow machine with the virus checker tomorrow, and see if I can reproduce the original issue, and if so, whether this patch fixes it. I'll report back tomorrow...

[ptthiem]

if you are dealing with virtualenv, you might have to pactch the tarball that virtualenv installs pip from. site-packages\virtualenv_support\pip...tar.gz

[pfmoore]

Never mind, when I try to reproduce the issue I had, I can't. I suspect it wasn't pip-related anyway (IIRC, distribute's setup process was also an issue).

Having said that, I don't have a problem with the setup.py patch. It's clearly "more correct" than the original code. Although you need to add a __future__ import to use "with", as pip needs to support Python 2.5.

[ptthiem]

I found my original error. It was on installing pip on pypy under virtualenv\win32 (doing test environments). When distribute cleaned up the working directory files were still open. So this isn't exactly related to this bug, sorry. I was in the process of submitting from a list of bugs\patches to a group of projects and was looking for already submitted issues. But if you all want the patch I can fix it and submit, just let me know where to shoot: Here, a new issue, pull request, etc.

[zephraph]

I realize this is an older issue, but this problem is still persisting.

I've isolated it to here. This os.rmdir is called after the files have been downloaded to the temp directory, unpacked, and copied to the build directory. The directory it's trying to remove should in theory always be empty; however, as mentioned previously it's possible for some other software to hold onto a file in the directory just long enough to cause the os.rmdir to fail.

In my case I have some corporate anti-virus that consistently holds files open long enough to make nearly every pip install fail. I tried changing os.rmdir to rmtree and got got an error saying it was trying to remove some file that didn't exist (meaning it was finally released by the anti-virus and removed as scheduled).

I was able to get past this by using rmtree(temp_dir, ignore_errors=True). The other option would be to wait until the directory is empty.

I'll submit a PR for either desired option of fixing the issue. Granted, I realize it's not an issue everyone will have, but this change also wouldn't negatively affect anyone else.

[dstufft]

It should likely use pip.utils.rmtree but it doesn't sound like something that would hurt to do.

[zephraph]

That's what I was using. pip.utils.rmtree has already been imported into that file and is used in a few other places.

[pfmoore]

pip.utils.rmtree isn't actually the right solution here, as it will fail for an undeletable file (read-only files can be deleted if you clear the read-only bit).

What's needed here is something similar that, when it hits an error, waits for a short while (I have no idea how long is reasonable, depends on how the virus checker works) and then retries the delete. Whether we loop if the retry fails (risking an infinite wait for a genuinely undeletable file) or ignore errors on the retry (risking leaving junk lying around) is an open question.

Simplest answer, though, is just to ignore errors as suggested by the OP. This will leave junk around in the face of a virus checker as described, but at least it won't fail.

Why is it that virus checkers seem to cause far more problems than the viruses they are supposed to be protecting us again? :-(

[dstufft]

Perhaps we should do something like https://pypi.python.org/pypi/retrying in pip.utils.rmtree.

[zephraph]

I wouldn't be opposed to ignoring and logging or waiting and retrying. It just kinda needs something.

[pfmoore]

Note that so far there have been three scenarios reported as causing it to happen:

1. Having the build directory open in a window
2. Building in a Dropbox folder
3. Anti-virus

Of these, two (2 and 3) are temporary and waiting will likely resolve the issue. The third (open in a window) I suspect is an instance of the more general "something is using the directory as the cwd" which won't go away until that process completes (and it can happen that a persistent background process ends up with the directory as cwd, meaning it won't go away till a reboot). But really, why are people messing round in the build directory while the build is running? (They do, though, so we need to cater for it).

Also, there are possibly multiple places this matters. @zephraph pointed at where we download a HTTP URL to a temporary directory. The original report was about build directories (which is likely in the cleanup method of pip.utils.build.BuildDirectory now, although that code wasn't there at the time of the original report).

I'm inclined to suggest that we enhance pip.utils.rmtree to be defined to do the best we possibly can to ensure that a directory is removed, and use it everywhere, consistently. On that basis, it should:

1. Remove the "readonly" flag from read-only files, as at present.
2. If removing still fails then, wait a short period and retry a second time [1].

If the deletion fails at this point, and ignore_errors is set, log a visible warning message to the user [2]. Otherwise let the error propagate to the caller.

Notes:
[1] I don't really like the timed wait - it could add nasty delays. I'd like to keep the timeout as short as possible, @zephraph have you any idea what would work for you? I'm inclined to make it 0.1s. But maybe I'm being paranoid - this should only happen when the code is currently aborting after all. I'm definitely against the "wait indefinitely" option, though.
[2] The idea here is that if we're continuing regardless of errors, we should warn the user that we left junk around. If the caller wants to abort on an error, that's not needed. (There's a question, I guess, of whether we want to ignore errors or not at the call sites, and with this new function maybe we revisit some of the existing decisions here).

[dstufft]

The retrying thing can handle the retries, it's a nice little wrapper that, if an exception is raised, will simply retry the operation. You can specify a maximium number of retries, or a maximum amount of time, and you can do a backoff factor so it retries quickly and then retries again waiting longer.

[pfmoore]

Yeah, I get this. It looks quite nice (were you proposing vendoring it, or writing our own? I assume the former). I wasn't 100% sure where you intended to apply it - to pip.utils.rmtree, I guess? So we'd get the current "make the file read only" behaviour first, and only retry if that wasn't enough. That's basically what I was saying, but implemented via the wrapper, which saves us having to roll our own.

[zephraph]

In my case using pip.utils.rmtree with ignore_errors=True allowed me to install. I did see one error so between reading files and trying to remove them everything seemed to clean up and pass. I would think a relatively short delay would be preferable, but perhaps more tries.

I'd vote for the retry decorator using the back-off factor with a total time of no more than 3 seconds. That way the majority of cases would be caught in the first, fast few tries and there's a longer period for any funky happenings.

[dstufft]

Yea I'd just vendor it assuming it works on 2.6+ and 3.2+. No reason to write our own if we can help it. I also think just applying it to rmtree makes sense. I can't think of a situation where we want to delete a directory, but only want to delete it if we can the first time. I'm not sure what happens if you pass a non-existent directory to rmtree, but it's possible that it will need to be made to count that as a "success".

[pfmoore]

@zephraph Yeah, but ignore_errors=True leaves temp directories behind. Sorry, but temp clutter is a bit of a sore point with me... ;-)

[zephraph]

While that technically fixes the problem, it still probably isn't the most graceful way to handle things.

I'm in agreement with the above conversation which follows these steps:

1. Vendor retrying (so long as it supports 2.6+ and 3.2+)
2. Apply @retry to pip.utils.rmtree. Optionally apply the back-off factor with max time of 3 seconds (should never reach this)
3. Replace all instances of os.rmdir with pip.utils.rmtree

[iconberg]

Well, in my opinion, for a small problem use a small solution.
As long it is a temporary directory that may cleaned by os system periodicaly or demanded, i would not spent much work in a solution.
If i would, maybe it is simpler to tell the os to delete the directory at next boot.

I dont know .rmtree, but if something locked, and you ignore errors you get the same result!?

[pluckey]

I'm getting this error every time i try to install a new package via pip. It's never happened on this machine before - I suspect it's due to antivirus (Panda) keeping a hold on the newly created dir.

[zephraph]

That's very likely the case. I've somewhat abandoned python at work until I can get time to fix this or someone else fixes it.

[pfmoore]

OK, I've created PR #2397 for this. It needs tweaking to tune the retry behaviour (at the moment, it just retries a few times with no delays, which isn't much good). I'll work on that in a while.

@dstufft could you check my vendoring of retrying.py, please? I'm on Windows and git goes on about CRLF translation, so I want to be sure what gets committed looks OK on Unix.

I haven't done anything yet about places that don't currently use pip.utils.rmtree. I'll have a look at them - if there aren't too many and they look safe, I'll add them to the PR.

[pfmoore]

Closed via #2397

[pluckey]

@pfmoore Thanks Paul that fixed it.

[eliot1785]

Looks like this was closed, but I'm still experiencing the issue. Two times today I've attempted to install something using pip, and got a message about a directory not being empty.

When retrying pip states "Requirements already satisfied," but I'm not sure whether to trust my installation now. If the error happened during cleanup then I can live with temp files not being cleared, but it's hard to say where in the install it happened. Any tips would be appreciated.

Sorry for the messed up formatting, due to cmd width limits:

pip install django
Collecting django
Downloading Django-1.11.5-py2.py3-none-any.whl (6.9MB)
100% |████████████████████████████████| 7.0MB 169kB/s
Collecting pytz (from django)
Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
100% |████████████████████████████████| 491kB 1.5MB/s
Installing collected packages: pytz, django
Successfully installed django-1.11.5 pytz-2017.2
Exception:
Traceback (most recent call last):
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\shutil.py", line 391, in _rmtree_unsafe
os.rmdir(path)
OSError: [WinError 145] The directory is not empty: 'C:\Users\sdewey\AppData\Local\Temp\pip-build-lao4_e6l\django
\django\contrib\gis\db\backends\mysql'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip\basecommand.py", line 215, in
main
status = self.run(options, args)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip\commands\install.py", line 385
, in run
requirement_set.cleanup_files()
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip\utils\build.py", line 38, in _
exit_
self.cleanup()
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip\utils\build.py", line 42, in c
leanup
rmtree(self.name)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip_vendor\retrying.py", line 49,
in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip_vendor\retrying.py", line 212
, in call
raise attempt.get()
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip_vendor\retrying.py", line 247
, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip_vendor\six.py", line 686, in
reraise
raise value
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip_vendor\retrying.py", line 200
, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip\utils_init_.py", line 102,
in rmtree
onerror=rmtree_errorhandler)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
[Previous line repeated 3 more times]
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\shutil.py", line 393, in rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "c:\users\sdewey\docume1\mlteam1\agentl1.io\python1\venv\lib\site-packages\pip\utils_init.py", line 114,
in rmtree_errorhandler
func(path)
OSError: [WinError 145] The directory is not empty: 'C:\Users\sdewey\AppData\Local\Temp\pip-build-lao4_e6l\django
\django\contrib\gis\db\backends\mysql'

[eliot1785]

Sorry, rather than reopening this old thread I thought it would be prudent to just open a new issue: #4734