Bundling sometimes fails on JRuby

[smellsblue]

If I bundle Gemstash from MRI, creating a Gemfile.lock and then switch to JRuby, bundling with missing gems fails with the following error:

Could not find gem 'gemstash (>= 0) ruby' in source at ..
Source contains 'gemstash' at: 0.1.0

Doing the reverse works fine. (see the shell session at the bottom to see a full example of the problem)

The Gemfile simply points to the gemspec:

source "https://rubygems.org"

# Specify your gem's dependencies in gemstash.gemspec
gemspec

I suspect the key to the problem may be how I specified the platform:

  spec.platform      = "java" if RUBY_PLATFORM == "java"

Which produces this in my lock on MRI:

PATH
  remote: .
  specs:
    gemstash (0.1.0)
...

And this in JRuby:

PATH
  remote: .
  specs:
    gemstash (0.1.0-java)
...

I am using MRI 2.2.2 and JRuby 9.0.3.0.

Example shell session illustrating the problem:

$ rvm use 2.2.2
Using ~/.rvm/gems/ruby-2.2.2
$ rm Gemfile.lock
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Using ast 2.1.0
Using parser 2.2.3.0
Using astrolabe 1.3.1
Using bundler 1.10.6
Using dalli 2.7.4
Using diff-lcs 1.2.5
Using multipart-post 2.0.0
Using faraday 0.9.2
Using faraday_middleware 0.10.0
Using lru_redux 1.1.0
Using puma 2.15.3
Using sequel 4.28.0
Using rack 1.6.4
Using rack-protection 1.5.3
Using tilt 2.0.1
Using sinatra 1.4.6
Using sqlite3 1.3.11
Using thor 0.19.1
Using gemstash 0.1.0 from source at .
Using powerpack 0.1.1
Using rack-test 0.6.3
Using rainbow 2.0.0
Using rspec-support 3.3.0
Using rspec-core 3.3.2
Using rspec-expectations 3.3.1
Using rspec-mocks 3.3.2
Using rspec 3.3.0
Using ruby-progressbar 1.7.5
Using tins 1.6.0
Using rubocop 0.35.1
Bundle complete! 6 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
$ cat Gemfile.lock
PATH
  remote: .
  specs:
    gemstash (0.1.0)
      dalli (~> 2.7)
      faraday (~> 0.9)
      faraday_middleware (~> 0.10)
      lru_redux (~> 1.1)
      puma (~> 2.14)
      sequel (~> 4.26)
      sinatra (~> 1.4)
      sqlite3 (~> 1.3)
      thor (~> 0.19)

GEM
  remote: https://rubygems.org/
  specs:
    ast (2.1.0)
    astrolabe (1.3.1)
      parser (~> 2.2)
    dalli (2.7.4)
    diff-lcs (1.2.5)
    faraday (0.9.2)
      multipart-post (>= 1.2, < 3)
    faraday_middleware (0.10.0)
      faraday (>= 0.7.4, < 0.10)
    lru_redux (1.1.0)
    multipart-post (2.0.0)
    parser (2.2.3.0)
      ast (>= 1.1, < 3.0)
    powerpack (0.1.1)
    puma (2.15.3)
    rack (1.6.4)
    rack-protection (1.5.3)
      rack
    rack-test (0.6.3)
      rack (>= 1.0)
    rainbow (2.0.0)
    rake (10.4.2)
    rspec (3.3.0)
      rspec-core (~> 3.3.0)
      rspec-expectations (~> 3.3.0)
      rspec-mocks (~> 3.3.0)
    rspec-core (3.3.2)
      rspec-support (~> 3.3.0)
    rspec-expectations (3.3.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.3.0)
    rspec-mocks (3.3.2)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.3.0)
    rspec-support (3.3.0)
    rubocop (0.35.1)
      astrolabe (~> 1.3)
      parser (>= 2.2.3.0, < 3.0)
      powerpack (~> 0.1)
      rainbow (>= 1.99.1, < 3.0)
      ruby-progressbar (~> 1.7)
      tins (<= 1.6.0)
    ruby-progressbar (1.7.5)
    sequel (4.28.0)
    sinatra (1.4.6)
      rack (~> 1.4)
      rack-protection (~> 1.4)
      tilt (>= 1.3, < 3)
    sqlite3 (1.3.11)
    thor (0.19.1)
    tilt (2.0.1)
    tins (1.6.0)

PLATFORMS
  ruby

DEPENDENCIES
  bundler (~> 1.10)
  gemstash!
  rack-test (~> 0.6)
  rake (~> 10.0)
  rspec (~> 3.3)
  rubocop (~> 0.34)

BUNDLED WITH
   1.10.6
$ rvm use jruby
Using ~/.rvm/gems/jruby-9.0.3.0
$ gem uninstall dalli
Successfully uninstalled dalli-2.7.4
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Could not find gem 'gemstash (>= 0) ruby' in source at ..
Source contains 'gemstash' at: 0.1.0
$ rm Gemfile.lock
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies....................
Using rake 10.4.2
Using ast 2.1.0
Using parser 2.2.3.0
Using astrolabe 1.3.1
Using bundler 1.10.6
Installing dalli 2.7.4
Using diff-lcs 1.2.5
Using multipart-post 2.0.0
Using faraday 0.9.2
Using faraday_middleware 0.10.0
Using jdbc-sqlite3 3.8.11.2
Using lru_redux 1.1.0
Using puma 2.15.3
Using sequel 4.28.0
Using rack 1.6.4
Using rack-protection 1.5.3
Using tilt 2.0.1
Using sinatra 1.4.6
Using thor 0.19.1
Using gemstash 0.1.0 from source at .
Using powerpack 0.1.1
Using rack-test 0.6.3
Using rainbow 2.0.0
Using rspec-support 3.3.0
Using rspec-core 3.3.2
Using rspec-expectations 3.3.1
Using rspec-mocks 3.3.2
Using rspec 3.3.0
Using ruby-progressbar 1.7.5
Using tins 1.6.0
Using rubocop 0.35.1
Bundle complete! 6 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
$ cat Gemfile.lock
PATH
  remote: .
  specs:
    gemstash (0.1.0-java)
      dalli (~> 2.7)
      faraday (~> 0.9)
      faraday_middleware (~> 0.10)
      jdbc-sqlite3 (~> 3.8)
      lru_redux (~> 1.1)
      puma (~> 2.14)
      sequel (~> 4.26)
      sinatra (~> 1.4)
      thor (~> 0.19)

GEM
  remote: https://rubygems.org/
  specs:
    ast (2.1.0)
    astrolabe (1.3.1)
      parser (~> 2.2)
    dalli (2.7.4)
    diff-lcs (1.2.5)
    faraday (0.9.2)
      multipart-post (>= 1.2, < 3)
    faraday_middleware (0.10.0)
      faraday (>= 0.7.4, < 0.10)
    jdbc-sqlite3 (3.8.11.2)
    lru_redux (1.1.0)
    multipart-post (2.0.0)
    parser (2.2.3.0)
      ast (>= 1.1, < 3.0)
    powerpack (0.1.1)
    puma (2.15.3-java)
    rack (1.6.4)
    rack-protection (1.5.3)
      rack
    rack-test (0.6.3)
      rack (>= 1.0)
    rainbow (2.0.0)
    rake (10.4.2)
    rspec (3.3.0)
      rspec-core (~> 3.3.0)
      rspec-expectations (~> 3.3.0)
      rspec-mocks (~> 3.3.0)
    rspec-core (3.3.2)
      rspec-support (~> 3.3.0)
    rspec-expectations (3.3.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.3.0)
    rspec-mocks (3.3.2)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.3.0)
    rspec-support (3.3.0)
    rubocop (0.35.1)
      astrolabe (~> 1.3)
      parser (>= 2.2.3.0, < 3.0)
      powerpack (~> 0.1)
      rainbow (>= 1.99.1, < 3.0)
      ruby-progressbar (~> 1.7)
      tins (<= 1.6.0)
    ruby-progressbar (1.7.5)
    sequel (4.28.0)
    sinatra (1.4.6)
      rack (~> 1.4)
      rack-protection (~> 1.4)
      tilt (>= 1.3, < 3)
    thor (0.19.1)
    tilt (2.0.1)
    tins (1.6.0)

PLATFORMS
  java

DEPENDENCIES
  bundler (~> 1.10)
  gemstash!
  rack-test (~> 0.6)
  rake (~> 10.0)
  rspec (~> 3.3)
  rubocop (~> 0.34)

BUNDLED WITH
   1.10.6
$ gem uninstall dalli
Successfully uninstalled dalli-2.7.4
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Using ast 2.1.0
Using parser 2.2.3.0
Using astrolabe 1.3.1
Using bundler 1.10.6
Installing dalli 2.7.4
Using diff-lcs 1.2.5
Using multipart-post 2.0.0
Using faraday 0.9.2
Using faraday_middleware 0.10.0
Using jdbc-sqlite3 3.8.11.2
Using lru_redux 1.1.0
Using puma 2.15.3
Using sequel 4.28.0
Using rack 1.6.4
Using rack-protection 1.5.3
Using tilt 2.0.1
Using sinatra 1.4.6
Using thor 0.19.1
Using gemstash 0.1.0 from source at .
Using powerpack 0.1.1
Using rack-test 0.6.3
Using rainbow 2.0.0
Using rspec-support 3.3.0
Using rspec-core 3.3.2
Using rspec-expectations 3.3.1
Using rspec-mocks 3.3.2
Using rspec 3.3.0
Using ruby-progressbar 1.7.5
Using tins 1.6.0
Using rubocop 0.35.1
Bundle complete! 6 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
$ rvm use 2.2.2
Using ~/.rvm/gems/ruby-2.2.2
$ gem uninstall dalli
Successfully uninstalled dalli-2.7.4
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Using ast 2.1.0
Using parser 2.2.3.0
Using astrolabe 1.3.1
Using bundler 1.10.6
Installing dalli 2.7.4
Using diff-lcs 1.2.5
Using multipart-post 2.0.0
Using faraday 0.9.2
Using faraday_middleware 0.10.0
Using lru_redux 1.1.0
Using puma 2.15.3
Using sequel 4.28.0
Using rack 1.6.4
Using rack-protection 1.5.3
Using tilt 2.0.1
Using sinatra 1.4.6
Using sqlite3 1.3.11
Using thor 0.19.1
Using gemstash 0.1.0 from source at .
Using powerpack 0.1.1
Using rack-test 0.6.3
Using rainbow 2.0.0
Using rspec-support 3.3.0
Using rspec-core 3.3.2
Using rspec-expectations 3.3.1
Using rspec-mocks 3.3.2
Using rspec 3.3.0
Using ruby-progressbar 1.7.5
Using tins 1.6.0
Using rubocop 0.35.1
Bundle complete! 6 Gemfile dependencies, 31 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

[smellsblue]

The same failure happens for JRuby 1.7.22 as well

[smellsblue]

I created a simple gem called example to replicate this issue with just a single dependency. Then I started digging into the issue, and I found some insights into the problem, but I don't really know Bundler code, so I'm not sure what direction to take the fix.

lib/bundler/definition.rb seems to be where the Gemfile.lock and Gemfile are loaded and checked for updates and whatnot.

This code then goes on to construct the Bundler::Resolver which ultimately checks the dependencies and attempts to resolve them:

    def resolve
      @resolve ||= begin
        last_resolve = converge_locked_specs
        if Bundler.settings[:frozen] || (!@unlocking && nothing_changed?)
          last_resolve
        else
          # Run a resolve against the locally available gems
          requested_ruby_version = ruby_version.version if ruby_version
          last_resolve.merge Resolver.resolve(expanded_dependencies, index, source_requirements, last_resolve, requested_ruby_version)
        end
      end
    end

Now, I was able to dig in and hack a fix, but I don't really understand what all the pieces are supposed to do yet. So, I don't really understand what the purpose of expanded_dependencies is, but in my example gem it starts with the following:

[#<Bundler::DepProxy:0x4612b856 @__platform="ruby", @dep=Gem::Dependency.new("example", Gem::Requirement.new([">= 0"]), :runtime)>,
 #<Bundler::DepProxy:0x22875539 @__platform=#<Gem::Platform:0x97a @cpu=nil, @os="java", @version=nil>, @dep=Gem::Dependency.new("example", Gem::Requirement.new([">= 0"]), :runtime)>,
...

The key thing to notice is that it starts with the ruby platform. I patched `expanded_dependencies with the following which caused my bundle to start succeeding:

    def expanded_dependencies
      @expanded_dependencies ||= expand_dependencies(dependencies, @remote)
    end

    def expand_dependencies(dependencies, remote = false)
      deps = []
      dependencies.each do |dep|
        dep = Dependency.new(dep, ">= 0") unless dep.respond_to?(:name)
        next unless remote || dep.current_platform?
        dep.gem_platforms(@platforms).each do |p|
          # ========> NOTE: The following line is my patch <========
          next if p == "ruby" && dep.name == "example"
          deps << DepProxy.new(dep, p) if remote || p == generic(Gem::Platform.local)
        end
      end
      deps
    end

So this drops the ruby platform that ultimately causes the failure, but clearly isn't a real fix.

Let's dig deeper into what's going on by looking at lib/bundler/resolver.rb.

verify_gemfile_dependencies_are_found!(requirements) is the method where the error is coming from, and it is because search_for(requirement).empty? returns true for that very first requirement that I patched out previously (the example for ruby platform one).

The last line of search_for(dependency) is significant here:

search.select {|sg| sg.for?(platform, @ruby_version) }.each {|sg| sg.activate_platform(platform) }

The search only has 1 result: [#<Gem::Specification:0x3520 example-0.1.0-java>] (which is a SpecGroup from the same file, which extends Array, hence the surrounding brackets). The select block is then passing platform which is ruby (because it is coming from that first dependency from before, again). I'm not really sure what @ruby_version is, but it's not relevant here. The for? method on the SpecGroup then looks at @specs which explicitly has nil in the hash, causing the select to reduce to nothing, which ultimately comes back to verify_gemfile_dependencies_are_found!(requirements) and results in an exception.

Let's look at where @specs comes from. It is in the initializer for SpecGroup, which is looking at the values in reverse order, and matching against all the platforms. Thus, that spec in the SpecGroup remember is #<Gem::Specification:0x3520 example-0.1.0-java>, which clearly has the java platform. If we look at the match_platform implementation in lib/bundler/match_platform.rb, we will see that our platform (java) ends up being false for match_platform for ruby, causing @specs to get nil for ruby.

In the reverse case (going from jruby to ruby), things end up being reversed. However, the SpecGroup has example-0.1.0, which is for the ruby platform, which ends up matching every platform, and so it makes sense then that this same problem doesn't happen in reverse.

Now.... what needs to change...? Bundler::Resolver#search_for? Bundler::Definition#expand_dependencies? Something else entirely? Throw in the towel and just let the answer to this bug be: separate gemspec files? This is the part where I'm stumped.

[segiddins]

Wow, great job investigating! I think the platform in the dependency is the problem here, but I can't be sure.

[smellsblue]

@segiddins thanks! After more investigating, I think I figured out a solution that works, and yeah, the dependency platforms was the problem. My solution restricts the platforms to what the gemspec indicates the platforms are, which causes the expanded_dependencies method to only include those platforms, and then everything seems to work ok.