This repository has been archived by the owner on Apr 14, 2021. It is now read-only.
Concurrent bundle install runs raise errors unlinking ~/.bundle/cache/compact_index/rubygems.org.443.[..]/versions #4519
Labels
Milestone
Comments
domcleal
added a commit
to domcleal/bundler
that referenced
this issue
May 10, 2016
When bundler is run concurrently using the same bundle dir in $HOME, the versions file can be updated from two processes at once. The download has been changed to a temporary file, which is securely moved into place over the original. If retrying the update operation, the original file is no longer immediately deleted and instead a full download is performed, later overwriting the original file if successful. If two processes are updating in parallel, this should ensure the original file isn't corrupted and that both processes succeed. - Fixes rubygems#4519
domcleal
added a commit
to domcleal/bundler
that referenced
this issue
May 10, 2016
When bundler is run concurrently using the same bundle dir in $HOME, the versions file can be updated from two processes at once. The download has been changed to a temporary file, which is securely moved into place over the original. If retrying the update operation, the original file is no longer immediately deleted and instead a full download is performed, later overwriting the original file if successful. If two processes are updating in parallel, this should ensure the original file isn't corrupted and that both processes succeed. - Fixes rubygems#4519
domcleal
added a commit
to domcleal/bundler
that referenced
this issue
May 10, 2016
When bundler is run concurrently using the same bundle dir in $HOME, the versions file can be updated from two processes at once. The download has been changed to a temporary file, which is securely moved into place over the original. If retrying the update operation, the original file is no longer immediately deleted and instead a full download is performed, later overwriting the original file if successful. If two processes are updating in parallel, this should ensure the original file isn't corrupted and that both processes succeed. - Fixes rubygems#4519
domcleal
added a commit
to domcleal/bundler
that referenced
this issue
May 10, 2016
When bundler is run concurrently using the same bundle dir in $HOME, the versions file can be updated from two processes at once. The download has been changed to a temporary file, which is securely moved into place over the original. If retrying the update operation, the original file is no longer immediately deleted and instead a full download is performed, later overwriting the original file if successful. If two processes are updating in parallel, this should ensure the original file isn't corrupted and that both processes succeed. - Fixes rubygems#4519
homu
added a commit
that referenced
this issue
May 15, 2016
Safely store concurrent compact index downloads When bundler is run concurrently using the same bundle dir in $HOME, the versions file can be updated from two processes at once. The download has been changed to a temporary file, which is securely moved into place over the original. If retrying the update operation, the original file is no longer immediately deleted and instead a full download is performed, later overwriting the original file if successful. If two processes are updating in parallel, this should ensure the original file isn't corrupted and that both processes succeed. - Fixes #4519 --- This would be useful on 1.12.x if possible, since the new caching behaviour with a shared home directory is causing the errors described in #4519.
segiddins
pushed a commit
that referenced
this issue
May 16, 2016
Safely store concurrent compact index downloads When bundler is run concurrently using the same bundle dir in $HOME, the versions file can be updated from two processes at once. The download has been changed to a temporary file, which is securely moved into place over the original. If retrying the update operation, the original file is no longer immediately deleted and instead a full download is performed, later overwriting the original file if successful. If two processes are updating in parallel, this should ensure the original file isn't corrupted and that both processes succeed. - Fixes #4519 --- This would be useful on 1.12.x if possible, since the new caching behaviour with a shared home directory is causing the errors described in #4519.
indirect
added a commit
that referenced
this issue
Oct 3, 2016
As we noticed in #4519, we need to use a temporary directory to hold compact index downloads so that multiple processes don't write to the same files at the same time and break everything. The fix for that was #4561, which added temporary directories to hold all files as they download, and then uses the (atomic) `FileUtils.cp` to move the completed downloads into place, so there is never a point where multiple processes are trying to write into the file at once. Unfortunately, using `Dir.mktmpdir` requires that the parent directory be _either_ world writable or sticky, but not both. Based on #4599, it looks like it's common for home directories to be both world writable and sticky. While that's a security problem by itself, it's not a big concern for Bundler and the compact index. So we want to let users continue to use Bundler, even with the compact index, without having to change the permissions on their home directories. This commit changes the `mktmpdir` call to create the temporary directory inside the default OS tempdir, which is typically `/tmp` or `/var/tmp` depending on distro. Since that directory is designed to hold other temporary directories, that change should (theoretically) reduce or eliminate the problem reported in #4599.
Merged
homu
added a commit
that referenced
this issue
Oct 4, 2016
use /tmp for mktmpdir As we noticed in #4519, we need to use a temporary directory to hold compact index downloads so that multiple processes don't write to the same files at the same time and break everything. The fix for that was #4561, which added temporary directories to hold all files as they download, and then uses the (atomic) `FileUtils.cp` to move the completed downloads into place, so there is never a point where multiple processes are trying to write into the file at once. Unfortunately, using `Dir.mktmpdir` requires that the parent directory be _either_ world writable or sticky, but not both. Based on #4599, it looks like it's common for home directories to be both world writable and sticky. While that's a security problem by itself, it's not a big concern for Bundler and the compact index. So we want to let users continue to use Bundler, even with the compact index, without having to change the permissions on their home directories. This commit changes the `mktmpdir` call to create the temporary directory inside the default OS tempdir, which is typically `/tmp` or `/var/tmp` depending on distro. Since that directory is designed to hold other temporary directories, that change should (theoretically) reduce or eliminate the problem reported in #4599.
indirect
pushed a commit
that referenced
this issue
Oct 11, 2016
use /tmp for mktmpdir As we noticed in #4519, we need to use a temporary directory to hold compact index downloads so that multiple processes don't write to the same files at the same time and break everything. The fix for that was #4561, which added temporary directories to hold all files as they download, and then uses the (atomic) `FileUtils.cp` to move the completed downloads into place, so there is never a point where multiple processes are trying to write into the file at once. Unfortunately, using `Dir.mktmpdir` requires that the parent directory be _either_ world writable or sticky, but not both. Based on #4599, it looks like it's common for home directories to be both world writable and sticky. While that's a security problem by itself, it's not a big concern for Bundler and the compact index. So we want to let users continue to use Bundler, even with the compact index, without having to change the permissions on their home directories. This commit changes the `mktmpdir` call to create the temporary directory inside the default OS tempdir, which is typically `/tmp` or `/var/tmp` depending on distro. Since that directory is designed to hold other temporary directories, that change should (theoretically) reduce or eliminate the problem reported in #4599.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What did you do?
I ran the command
bundle installin four parallel processes using different RVM gemsets (therefore different GEM_HOME), but under the same user account with a shared HOME environment variable. No Gemfile.lock was present, it was a clean checkout of a project with only a Gemfile.What did you expect to happen?
I expected Bundler to complete without error by either locking the files it's downloading to be safer with multiple processes, or to use caches that are unique per GEM_HOME.
What happened instead?
Instead, what actually happened was one of the
bundle installcommands failed intermittently (a very small percentage of the time) with an error deleting~/.bundle/cache/compact_index/rubygems.org.443.29b0360b937aa4d161703e6160654e47/versions.It appears that the compact index cache is shared for a whole user account without any locking, so multiple processes can try to download the same file at the same time and cause errors for each other.
Error details
Environment
Bundler 1.12.1
Rubygems 2.4.8
Ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]
GEM_HOME /usr/local/rvm/gems/ruby-2.0.0-p353@test_kafo_parsers_master_pull_request-0
GEM_PATH /usr/local/rvm/gems/ruby-2.0.0-p353@test_kafo_parsers_master_pull_request-0:/usr/local/rvm/gems/ruby-2.0.0-p353@global
RVM 1.26.11 (1.26.11)
Git 1.8.3.1
rubygems-bundler (1.4.4)
orig_path
Set via BUNDLE_ORIG_PATH: "/usr/local/rvm/gems/ruby-2.0.0-p353@test_kafo_parsers_master_pull_request-0/bin:/usr/local/rvm/gems/ruby-2.0.0-p353@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p353/bin:/usr/local/rvm/bin:/usr/local/bin:/usr/bin"
orig_gem_path
Set via BUNDLE_ORIG_GEM_PATH: "/usr/local/rvm/gems/ruby-2.0.0-p353@test_kafo_parsers_master_pull_request-0:/usr/local/rvm/gems/ruby-2.0.0-p353@global"
The text was updated successfully, but these errors were encountered: