Document source priority in 2.0

[indirect]

In Bundler 2, all gems have to have an unambiguous source. Since all sources after the first must have blocks with gems in them, this is pretty straightforward:

source "rubygems"
source "private" do
  gem "depends_on_rack"
end

If a usable version of rack is present in the private source, it will be installed. If there is no usable version of rack in the private source, a usable version of rack from the "rubygems" source will be installed. We should document this in the Gemfile manpage and in the website guide to sources.

[segiddins]

Also what if there are two gems that depend on rack in different source blocks, each which have rack?

[indirect]

💥 (undefined, and we should figure that out)

[segiddins]

IMHO that should be an error, and you should have to specify rack in one of them

[RochesterinNYC]

I'm in support of requiring more explicitness in the Gemfile in such cases (or in general when it comes to sources).

[ntl]

When more than one source hosts a gem of the same name, and that gem is a dependency of a gem listed in the Gemfile (but is not itself listed in the Gemfile), then I think bundler should implicitly prefer the source of the gem that depends upon it. Once ambiguity is introduced by multiple gems from different sources depending on that gem, an explicit source ought to be mandated by bundler. This seems like a reasonable tradeoff between developer convenience and maintaining deterministic behavior.

Currently, the state of working with multiple sources in ruby is a bit frustrating, because gems aren't canonically referenced by fully qualified URLs. Instead, there is a global namespace ala DNS. I, for one, am tired of getting stumped to come up with a name every time I publish an open source library in ruby. As it stands, bundler & rubygems works great if you want to throw up a private server with a few gems to augment what's on http://rubygems.org, but I think this change would move more in the direction of being able to support more decentralized setups, which is a big win, in my view. Going along with the example, a gem called rack hosted at one URL could actually be a totally different library than the rack hosted at rubygems.org.

So, I think I'm in agreement with @RochesterinNYC . Just wanted to voice my preference as well.

[ntl]

Another note: I think the behavior of dependencies-of-dependencies ought to be the same as the behavior of dependencies. So, going along @indirect 's example above, suppose I have:

source 'https://source-a.com'

source 'https://source-b.com' do
  gem 'depends_on_something_that_depends_on_rack'
end

In that case, I believe that bundler should prefer any gem called rack from source-b over source-a. Obviously, if there's any gem not hosted on source-b that depends on rack, bundler should fail unless an explicit source for rack is provided.

[segiddins]

This still needs docs, I think, @indirect ?

[indirect]

@segiddins do we have an executable example that we can use to demonstrate source priority? should I build one? should we document it solely inside the manpage for Gemfile, or elsewhere as well?

[segiddins]

I can dig up a spec we can use for an example. Gemfile seems fine for now, unless it starts confusing people

[indirect]

@segiddins looks like we still need to implement the error, if you still think we should have that:

$ cat gems.rb
source "https://rubygems.org" # this server has rake 12.0.0

source "http://localhost:9292/private" do # this server has rake 12.0.0
  gem "depends_on_rake"
end

source "http://localhost:4242/private" do # this server has rake 12.0.0
  gem "depends_on_rake2"
end

$ bundle install
Fetching gem metadata from http://localhost:4242/private/...
Fetching gem metadata from http://localhost:9292/private/...
Fetching gem metadata from https://rubygems.org/.
Fetching gem metadata from http://localhost:4242/private/..
Fetching gem metadata from http://localhost:9292/private/..
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Bundle complete! 2 Gemfile dependencies, 4 gems now installed.
Bundled gems are installed into `./.bundle`

$ cat gems.locked
GEM
  remote: https://rubygems.org/
  specs:

GEM
  remote: http://localhost:4242/private/
  specs:
    depends_on_rake2 (0.1.0)
      rake (~> 12.0)
    rake (12.0.0)

GEM
  remote: http://localhost:9292/private/
  specs:
    depends_on_rake (0.1.1)
      rake (~> 12.0)

PLATFORMS
  ruby
  x86_64-darwin-16

DEPENDENCIES
  depends_on_rake!
  depends_on_rake2!

BUNDLED WITH
   2.0.0.dev

[segiddins]

Why would that error?

[indirect]

@segiddins earlier in this ticket, you said:

what if there are two gems that depend on rack in different source blocks, each which have rack? IMHO that should be an error, and you should have to specify rack in one of them

The example above is my attempt to deliberately create that situation. Did I do it wrong? Or have you changed your mind about what should error?

[segiddins]

ah I didn't look that far up... :/ That might be hard to do, but I can look into it

[segiddins]

I can't really think of how to implement this sanely in the resolver at the moment -- the problem is that it's valid for the gem to come from multiple sources, so long as the source it ends up coming from is "'nearer" to the vertex in the dependency graph than any other source that contains a gem with that name

[segiddins]

@greysteil not intending to really rope you into this, but can you think of anything off of the top of your head?

[indirect]

@segiddins I'm struggling to see how that description ("the source is nearer to the vertex") applies to the second gem, which has a "rack" gem in its own source, and still ends up using the "rack" gem from the other source block instead. Can you elaborate a little bit?

[segiddins]

In this case, they're equally far away. But in this scenario

a
  b
    c
aa
  c

c is closer to aa, so in my mind we can unambiguously take it from aa's source.

Unrelated, but a further wrinkle is that the default source exists, but ideally would only be treated as a fallback for those gems we can find in any source of its transitive predecessors?

[indirect]

@segiddins the rake from localhost:9292 is used, and the rake from localhost:4242 is ignored, even though depends_on_rake2 is inside the source localhost:4242. So Bundler is choosing the second-closest rake source for depends_on_rake2 without printing a warning or error.

[indirect]

Here are the available gems:

source1
  rake
source2
  needs-rake
  rake
source3
  needs-rake2
  rake

Here are the gems chosen by Bundler:

source2
  needs-rake
  rake
source3
  needs-rake2

The rake is close to one gem, but farther from another.

[indirect]

Based on previous discussion, I think we should actively raise if rake is present in two or more sources in the gemfile without an explicit source.

[segiddins]

Even if one of those sources is the default source? Even if one of those sources is not considered a potential source for the gem in question?

[segiddins]

The rake is close to one gem, but farther from another.

rake is equally far from need-rake and needs-rake2 in your example

[greysteil]

No top-of-my-head idea how to implement a distance-based source error in the resolver. If you go down that route, I guess we could do it in Index#requirement_satisfied_by? in a similar way to how we ensure CocoaPods specs have the same root name?

[indirect]

@segiddins I think what I'm trying to get at is that rake from source2 is 0 sources away from needs_rake, which is fine, but: rake from source3 is 0 sources away from needs_rake2, and it was not chosen. Instead, rake from source2 was chosen, which is 1 source away from needs_rake2. We rejected source3's rake, even though it was closer to needs_rake2.

Moving wayyyy back to look at the big picture, isn't this situation supposed to raise an error if any gem needs rake, there is no Gemfile source pinning for rake, and there is more than one source that provides rake?

[segiddins]

Moving wayyyy back to look at the big picture, isn't this situation supposed to raise an error if any gem needs rake, there is no Gemfile source pinning for rake, and there is more than one source that provides rake?

I don't know. I think there's an argument to be made that

source "https://rubygems.org"

source other_source do
  gem "depends_on_rack"
end

Should succeed and install rack from other_source. But I'm honestly not sure what the best answer is in this scenario.

I guess, from a conservative standpoint, it might be best to error in that scenario for now because that gives us the most latitude to make "improvements" should we think up some brilliant algorithm?

[segiddins]

Another question -- should we raise in this scenario?

source source1 do
  gem "depend_on_rack"
end
source source2 do
  gem "does_not_depend_on_rack"
end

where both sources contain rack?

[segiddins]

it turns out we actually have a test for the scenario I described in #4629 (comment), and it asserts that you should be able to install in that scenario, without warning, with the gem coming from the source block source

[segiddins]

@indirect #5985 attempts to implement this

[indirect]

Closed by #5985.