bundle package -- all && bundle install --local fails to install

[ismell]

What you're trying to accomplish

I have a directory structure like so

* repo
  * gems
    * my-gem
  * services
    * app
      * Gemfile
      * Gemfile.lock
      * vendor
      * ...

My Gemfile contains a relative path to my-gem

gem "my-gem", :path => "../../gems/my-gem"

I'm trying to package up my gems into the vendor/cache folder so I can easily deploy it to my production servers. My deployment artifact will only contain the contents of app.

The command you ran

On the build box I ran

bundle package --all

This installed all my gems dependencies into vendor/cache including all my git and path dependencies. It also updated the Gemfile.lock to show the path as vendor/cache/my-gem.

I transported the app folder to my production server and did

bundle install --local

What you expected to happen

When running with --local bundle should install all the gems in the vendor/cache.

What actually happened

Bundler installed the gems but then tried to update vendor/cache and failed due to the fact that the actual source of the gems does not exist.

The exception backtrace(s), if any

root@0b94b04659c6:/app# bundle install --local
Resolving dependencies...
Installing rake (10.0.4) 
Installing bson (1.8.6) 
Installing bson_ext (1.8.6) 
Installing timers (1.1.0) 
Installing celluloid (0.14.1) 
Installing connection_pool (1.0.0) 
Installing contractual (0.0.2) 
Installing little-plugger (1.1.3) 
Installing multi_json (1.7.6) 
Installing logging (1.8.1) 
Using cws_log (0.0.222) from source at /app/vendor/cache/cws_log 
Using cws_util (0.0.1) from source at /app/vendor/cache/cws_util 
Installing multipart-post (1.2.0) 
Installing faraday (0.8.7) 
Installing faraday_middleware (0.9.0) 
Installing json (1.8.0) 
Installing mongo (1.8.6) 
Installing net-http-persistent (2.8) 
Installing rack (1.5.2) 
Installing rack-protection (1.5.0) 
Installing tilt (1.4.1) 
Installing sinatra (1.4.2) 
Using cws_api (0.0.250) from source at /app/vendor/cache/cws_api 
Using cws_authz_client (0.0.104) from source at /app/vendor/cache/cws_authz_client 
Installing multi_xml (0.5.4) 
Installing httparty (0.11.0) 
Using cws_globalreg (0.0.187) from source at /app/vendor/cache/cws_globalreg 
Using cws_service_crud_util (0.01) from source at /app/vendor/cache/cws_service_crud_util 
Installing mime-types (1.23) 
Installing rest-client (1.6.7) 
Using cws_tester (0.0.149) from source at /app/vendor/cache/cws_tester 
Installing ffi (1.8.1) 
Installing thor (0.18.1) 
Using foreman (0.62.0) from git://github.wdig.com/boulder/foreman.git (at /app/vendor/cache/foreman-c9f0e9e49f16) 
Installing kgio (2.8.0) 
Installing listen (0.7.3) 
Installing rack-test (0.6.2) 
Installing raindrops (0.11.0) 
Installing rb-inotify (0.9.0) 
Installing redis (3.0.4) 
Installing redis-namespace (1.3.0) 
Using rerun (0.8.0) from git://github.wdig.com/boulder/rerun.git (at /app/vendor/cache/rerun-174bc3ce2ad1) 
Installing sidekiq (2.12.1) 
Installing unicorn (4.6.2) 
Installing xml-simple (1.1.2) 
Using bundler (1.3.5) 
Updating files in vendor/cache
Unfortunately, a fatal error has occurred. Please see the Bundler troubleshooting documentation at
http://bit.ly/bundler-issues. Thanks!
/usr/local/lib/ruby/1.9.1/fileutils.rb:1515:in `stat': No such file or directory - ../../gems/cws_log/. (Errno::ENOENT)
    from /usr/local/lib/ruby/1.9.1/fileutils.rb:1515:in `block in fu_each_src_dest'
    from /usr/local/lib/ruby/1.9.1/fileutils.rb:1531:in `fu_each_src_dest0'
    from /usr/local/lib/ruby/1.9.1/fileutils.rb:1513:in `fu_each_src_dest'
    from /usr/local/lib/ruby/1.9.1/fileutils.rb:436:in `cp_r'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/source/path.rb:81:in `cache'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/runtime.rb:109:in `block in cache'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/runtime.rb:107:in `cache'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/cli.rb:248:in `install'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/vendor/thor/task.rb:27:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/vendor/thor/invocation.rb:120:in `invoke_task'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/vendor/thor.rb:344:in `dispatch'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/vendor/thor/base.rb:434:in `start'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/bin/bundle:20:in `block in <top (required)>'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/friendly_errors.rb:3:in `with_friendly_errors'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/bin/bundle:20:in `<top (required)>'
    from /usr/local/bin/bundle:19:in `load'
    from /usr/local/bin/bundle:19:in `<main>'
    from /usr/local/bin/ruby_noexec_wrapper:14:in `eval'
    from /usr/local/bin/ruby_noexec_wrapper:14:in `<main>'

Everything output by running bundle env

root@0b94b04659c6:/app# bundle env
Bundler 1.3.5
Ruby 1.9.3 (2013-05-15 patchlevel 429) [x86_64-linux]
Rubygems 1.8.23
GEM_HOME 
rubygems-bundler (1.2.0)

Bundler settings
  cache_all
    Set for your local app (/app/.bundle/config): "true"


Gemfile
source "https://rubygems.org"

# Application gems
gem "unicorn"
gem "sidekiq"
gem "net-http-persistent"
gem "xml-simple"

# Local CWS gems
gem "cws_log", :path => "../../gems/cws_log"
gem "cws_util", :path => "../../gems/cws_util"
gem "cws_tester", :path => "../../gems/cws_tester"
gem "cws_api", :path => "../../gems/cws_api"
gem "cws_globalreg", :path => "../../gems/cws_globalreg"
gem "cws_authz_client", :path => "../../gems/cws_authz_client"
gem "cws_service_crud_util", :path => "../../gems/cws_service_crud_util"

group :test do
  # Only used for testing
  gem "rest-client"
end

group :development do
  gem 'rb-inotify', '~> 0.9'
  gem "rerun", :git => "git://github.wdig.com/boulder/rerun.git", :branch => "master"
  gem "foreman", :git => "git://github.wdig.com/boulder/foreman.git", :branch => "master"

  gem "rake"
  gem "rack-test"
end


Gemfile.lock
GIT
  remote: git://github.wdig.com/boulder/foreman.git
  revision: c9f0e9e49f16b71cea760f211dcfb644382f05be
  branch: master
  specs:
    foreman (0.62.0)
      thor (>= 0.13.6)

GIT
  remote: git://github.wdig.com/boulder/rerun.git
  revision: 174bc3ce2ad1ad786b98197d762f7e41fa114398
  branch: master
  specs:
    rerun (0.8.0)
      listen (~> 0.7.3)

PATH
  remote: vendor/cache/cws_api
  specs:
    cws_api (0.0.250)
      bson_ext
      contractual
      cws_log
      cws_util
      faraday (~> 0.8.0)
      faraday_middleware (~> 0.9.0)
      json
      mongo
      net-http-persistent (~> 2.8.0)
      sinatra

PATH
  remote: vendor/cache/cws_authz_client
  specs:
    cws_authz_client (0.0.104)
      bson_ext
      cws_api
      cws_log
      cws_util
      faraday (~> 0.8)
      faraday_middleware
      json
      mongo
      sinatra

PATH
  remote: vendor/cache/cws_globalreg
  specs:
    cws_globalreg (0.0.187)
      cws_util
      faraday
      faraday_middleware
      httparty
      mongo
      net-http-persistent

PATH
  remote: vendor/cache/cws_log
  specs:
    cws_log (0.0.222)
      logging

PATH
  remote: vendor/cache/cws_service_crud_util
  specs:
    cws_service_crud_util (0.01)
      cws_util
      json
      mongo
      sinatra

PATH
  remote: vendor/cache/cws_tester
  specs:
    cws_tester (0.0.149)
      cws_util
      rest-client

PATH
  remote: vendor/cache/cws_util
  specs:
    cws_util (0.0.1)

GEM
  remote: https://rubygems.org/
  specs:
    bson (1.8.6)
    bson_ext (1.8.6)
      bson (~> 1.8.6)
    celluloid (0.14.1)
      timers (>= 1.0.0)
    connection_pool (1.0.0)
    contractual (0.0.2)
    faraday (0.8.7)
      multipart-post (~> 1.1)
    faraday_middleware (0.9.0)
      faraday (>= 0.7.4, < 0.9)
    ffi (1.8.1)
    httparty (0.11.0)
      multi_json (~> 1.0)
      multi_xml (>= 0.5.2)
    json (1.8.0)
    kgio (2.8.0)
    listen (0.7.3)
    little-plugger (1.1.3)
    logging (1.8.1)
      little-plugger (>= 1.1.3)
      multi_json (>= 1.3.6)
    mime-types (1.23)
    mongo (1.8.6)
      bson (~> 1.8.6)
    multi_json (1.7.6)
    multi_xml (0.5.4)
    multipart-post (1.2.0)
    net-http-persistent (2.8)
    rack (1.5.2)
    rack-protection (1.5.0)
      rack
    rack-test (0.6.2)
      rack (>= 1.0)
    raindrops (0.11.0)
    rake (10.0.4)
    rb-inotify (0.9.0)
      ffi (>= 0.5.0)
    redis (3.0.4)
    redis-namespace (1.3.0)
      redis (~> 3.0.0)
    rest-client (1.6.7)
      mime-types (>= 1.16)
    sidekiq (2.12.1)
      celluloid (>= 0.14.1)
      connection_pool (>= 1.0.0)
      json
      redis (>= 3.0)
      redis-namespace
    sinatra (1.4.2)
      rack (~> 1.5, >= 1.5.2)
      rack-protection (~> 1.4)
      tilt (~> 1.3, >= 1.3.4)
    thor (0.18.1)
    tilt (1.4.1)
    timers (1.1.0)
    unicorn (4.6.2)
      kgio (~> 2.6)
      rack
      raindrops (~> 0.7)
    xml-simple (1.1.2)

PLATFORMS
  ruby

DEPENDENCIES
  cws_api!
  cws_authz_client!
  cws_globalreg!
  cws_log!
  cws_service_crud_util!
  cws_tester!
  cws_util!
  foreman!
  net-http-persistent
  rack-test
  rake
  rb-inotify (~> 0.9)
  rerun!
  rest-client
  sidekiq
  unicorn
  xml-simple

My thought is that bundle install --local should not try and update the vendor/cache folder.

[TimMoore]

I can reproduce this with a very simple skeleton project: https://github.com/TimMoore/bundler_issues_2494

Clone that and then run ./reproduce.sh to see the bug.

[TimMoore]

It looks like you can work around this by removing the .bundle/config file from the copied directory. I think it's the BUNDLE_CACHE_ALL: 'true' configuration that's causing the problem.

[ismell]

Well what do you know. I guess that's a good work around. I can use that for now. Thanks :)

[indirect]

Running bundle install --local --no-cache will prevent Bundler from trying to update the cache after installing. The way the caching options work right now is kind of crazy-making, and I want to change them to something more straightforward, but that will have to wait for Bundler 2.0 to avoid breaking backwards compatibility.

On Jun 15, 2013, at 6:31 AM, Raul E Rangel [email protected] wrote:

Well what do you know. I guess that's a good work around. I can use that for now. Thanks :)

—
Reply to this email directly or view it on GitHub.

[TimMoore]

So as a short-term solution for 1.x, should --local imply --no-cache? I think that makes sense... I'm not sure why you would ever want to install from the cache and then update the cache afterwards.

[TimMoore]

Another strange thing: bundle package --all doesn't seem to update the path in Gemfile.lock the first time you run it, but it does if you run it a second time.

[indirect]

Yeah, I think patching --local to also set --no-cache makes sense for 1.x.

On Sat, Jun 15, 2013 at 6:46 PM, Tim Moore [email protected]
wrote:

Another strange thing: bundle package --all doesn't seem to update the path in Gemfile.lock the first time you run it, but it does if you run it a second time.

Reply to this email directly or view it on GitHub:
#2494 (comment)

[TimMoore]

Fixed by #2509. The fix should be in 1.3.6 and 1.4.0 when those are released.

[TimMoore]

Setting --local to imply --no-cache was released in 1.5.0, but unfortunately this causes problems. In the case where a gem is installed but not in vendor/cache, running bundle install --local no longer populated the cache with the missing gem. That change was reverted in #2800 so for now the best solution to this issue is to pass --no-cache explicitly.