Source blocks should use their nesting to determine resolution precedence

[nerdrew]

The following Gemfile should work:

source 'https://example.com/private/' do
  gem "my-gem"

  source 'https://example.com/really-private/' do
    gem "secret-gem" # secret-gem depends on my-gem, it should find it in /private

    path ".." do
      gem "secret-local-gem" # secret-local-gem depends on secret-gem, it should find it in /really-private
      gem "local-gem" # local-gem depends on my-gem, it should find it in /private
    end
  end
end

Thoughts? Would a PR for this be acceptable?

[hsbt]

I don't understand why you put my-gem to private instead of really-private. Is it political reason of your organization?

[nerdrew]

It is just an illustration.

This is copied from my actual setup:

source 'https://rubygems.org'

source 'https://example.com/private' do
  git 'github.com:example/my-gem.git', branch: 'master' do
    # my-gem also exists in /private, but I want this overriden version
    # my-gem depends on shared-gem in /private
    gem 'my-gem'
  end
end

[hsbt]

Your first example already works today if you pin each gem to its source explicitly, without nesting:

source 'https://example.com/private/' do
  gem "my-gem"
end

source 'https://example.com/really-private/' do
  gem "secret-gem"
end

path ".." do
  gem "secret-local-gem"
  gem "local-gem"
end

Every gem that appears as a dependency here (my-gem, secret-gem) is itself declared and pinned to its source, so resolution already finds my-gem in /private and secret-gem in /really-private. Nesting changes nothing in this case.

So the request really comes down to convenience: you'd rather not spell out my-gem and the other shared dependencies explicitly, and instead have nesting auto-pin them to the enclosing source. The only thing nesting would actually change is auto-pinning undeclared transitive dependencies to an enclosing source, and that explicit pin is intentional.

It's what keeps each gem's source unambiguous, and dropping it in favor of nesting-based precedence would weaken the per-source pinning that protects against source confusion. Saving a few lines in the Gemfile isn't worth giving that up, so I'd rather keep requiring the explicit declaration.

[nerdrew]

I oversimplified my actual example. It actually has a bunch of gems in the repo, and whenever we add another one, every app that consumes them needs to update their Gemfile to list every single gem. And they need to manually add all the transitive dependencies to the /private source block. It feels like it defeats the purpose of transitive dependencies. And the process of finding the transitive dependencies is a game of whack-a-mole. Add gems to git block. Try to bundle install. Fails. Add missing transitive dependencies to /private. Retry.

ource 'https://rubygems.org'

source 'https://example.com/private' do
  git 'github.com:example/gem-monorepo.git', branch: 'master' do
    # my-gem1 also exists in /private, but I want this overriden version
    # my-gem1 depends on shared-gem-1 in /private
    gem 'my-gem1'
    # my-gem2 also exists in /private, but I want this overriden version
    # my-gem2 depends on shared-gem-2 in /private
    gem 'my-gem2'
  end
end

[nerdrew]

It is really annoying having a bunch of gems listed in the Gemfile that you don't actively use (or worse, maybe they were transitive dependencies that are no longer necessary).

[hsbt]

The two parts of this are worth separating.

Listing each gem from the git monorepo is inherent to git sources, not something nesting would change. A git repo has no enumerable index, so you always declare each gem you want inside the git block, with or without this feature.

The transitive-dependency part should already work without manual listing. Since Bundler 2.3.0 (the dependency-confusion fix in #4609), when every scoped rubygems source implements the dependency API and there's a single global source, Bundler precomputes source requirements for indirect dependencies too: any gem that lives only in /private is auto-pinned there and resolved from there, even as a transitive dependency of a git-sourced gem. No gem line required.

So if you're still playing whack-a-mole, it points to one of: a Bundler older than 2.3, a /private source that doesn't serve the compact index / dependency API, or multiple global sources.