Binary order is incorrect

[schneems]

Report

Expected: If I put a version of rake in a Gemfile.lock then I should get that same version when I run my app.

Actual: Using this app https://github.com/heroku/ruby-getting-started/blob/5e7ce01610a21cf9e5381daea66f79178e2b3c06/Gemfile.lock#L188

Using the Ruby getting started guide running rake fails because it loads the version that ships with Ruby rather than the version installed via bundler

$ heroku run:inside web-7665b497c-hzpk9 -a desolate-waters-58318 "bash"
Running launcher bash on ⬢ desolate-waters-58318... up, web-7665b497c-hzpk9
heroku@web-7665b497c-hzpk9:/workspace$ which rake
/layers/heroku_ruby/ruby/bin/rake
heroku@web-7665b497c-hzpk9:/workspace$ rake -v
rake aborted!
Gem::LoadError: You have already activated rake 13.0.6, but your Gemfile requires rake 13.2.1. Prepending `bundle exec` to your command may solve this.
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/runtime.rb:304:in `check_for_activated_spec!'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/runtime.rb:25:in `block in setup'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/spec_set.rb:191:in `each'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/spec_set.rb:191:in `each'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/runtime.rb:24:in `map'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/runtime.rb:24:in `setup'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler.rb:162:in `setup'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/setup.rb:10:in `block in <top (required)>'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/ui/shell.rb:159:in `with_level'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/ui/shell.rb:111:in `silence'
/layers/heroku_ruby/bundler/gems/bundler-2.5.9/lib/bundler/setup.rb:10:in `<top (required)>'
<internal:/layers/heroku_ruby/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
<internal:/layers/heroku_ruby/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
/workspace/config/boot.rb:3:in `<top (required)>'
/workspace/config/application.rb:1:in `require_relative'
/workspace/config/application.rb:1:in `<top (required)>'
<internal:/layers/heroku_ruby/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
<internal:/layers/heroku_ruby/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
/workspace/Rakefile:4:in `<top (required)>'
(See full trace by running task with --trace)
heroku@web-7665b497c-hzpk9:/workspace$ which -a rake
/layers/heroku_ruby/ruby/bin/rake
/layers/heroku_ruby/gems/bin/rake
heroku@web-7665b497c-hzpk9:/workspace$ echo $PATH
/layers/heroku_ruby/bundler/bin:/layers/heroku_ruby/ruby/bin:/layers/heroku_ruby/gems/bin:/layers/heroku_ruby/bundler/bin:/layers/heroku_nodejs-engine/dist/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

While technically bundle exec will fix it, it's not the root cause and is different between the two:

heroku@web-7665b497c-hzpk9:/workspace$ bundle exec rake -V
rake, version 13.2.1
heroku@web-7665b497c-hzpk9:/workspace$ rake -V
rake, version 13.0.6
heroku@web-7665b497c-hzpk9:/workspace$ bundle exec which rake
/layers/heroku_ruby/gems/ruby/3.2.0/bin/rake
heroku@web-7665b497c-hzpk9:/workspace$ which rake
/layers/heroku_ruby/ruby/bin/rake

Classic

The binstubs take ultimate precedence

~ $ echo $PATH
/app/.heroku/yarn/bin:/app/bin:/app/vendor/bundle/bin:/app/vendor/bundle/ruby/3.1.0/bin:/app/.heroku/node/bin:/app/.heroku/yarn/bin:bin:vendor/bundle/ruby/1.9.1/bin:/usr/local/bin:/usr/bin:/bin:/app/vendor/jemalloc/bin:/app/bin:/app/node_modules/.bin
~ $ which rake
/app/bin/rake

In addition, it doesn't put the original location of the binaries on the path, instead it symlinks the binstub directory to these location:

~ $ ls -la ./bin | grep "vendor"
lrwxrwxrwx  1 u33431 dyno   28 Oct 28  2023 erb -> ../vendor/ruby-3.1.4/bin/erb
lrwxrwxrwx  1 u33431 dyno   28 Oct 28  2023 gem -> ../vendor/ruby-3.1.4/bin/gem
lrwxrwxrwx  1 u33431 dyno   28 Oct 28  2023 irb -> ../vendor/ruby-3.1.4/bin/irb
lrwxrwxrwx  1 u33431 dyno   29 Oct 28  2023 racc -> ../vendor/ruby-3.1.4/bin/racc
lrwxrwxrwx  1 u33431 dyno   28 Oct 28  2023 rbs -> ../vendor/ruby-3.1.4/bin/rbs
lrwxrwxrwx  1 u33431 dyno   29 Oct 28  2023 rdbg -> ../vendor/ruby-3.1.4/bin/rdbg
lrwxrwxrwx  1 u33431 dyno   29 Oct 28  2023 rdoc -> ../vendor/ruby-3.1.4/bin/rdoc
lrwxrwxrwx  1 u33431 dyno   27 Oct 28  2023 ri -> ../vendor/ruby-3.1.4/bin/ri
lrwxrwxrwx  1 u33431 dyno   29 Oct 28  2023 ruby -> ../vendor/ruby-3.1.4/bin/ruby
lrwxrwxrwx  1 u33431 dyno   33 Oct 28  2023 ruby.exe -> ../vendor/ruby-3.1.4/bin/ruby.exe
lrwxrwxrwx  1 u33431 dyno   33 Oct 28  2023 typeprof -> ../vendor/ruby-3.1.4/bin/typeprof

Notes

In CNBs the spec says that naming is the only mechanism (at this time) to control ordering https://github.com/buildpacks/spec/blob/f3fd7555c8320a9da8101f52e6e952e9679fa150/buildpack.md#L773

The lifecycle MUST order all paths provided by a given buildpack alphabetically ascending.

We can rename layers to adjust their order, however, layer names are visible to end users so it is awkward to have a layer like zgems. It's also worth noting that "alphabetically ascending" is not a rigorous specification layer names are not limited to ASCII characters https://github.com/buildpacks/spec/blob/f3fd7555c8320a9da8101f52e6e952e9679fa150/buildpack.md?plain=1#L592-L593 so it's possibly an implementation dependent concern. (though ASCII character ordering a-z is likely very consistent).

Moving forward

• Strong: Need to add ./bin/ binstubs to the path, this should be loaded last (takes precedence). We could call this something like user_binstubs
• Possibly: Rename gems or ruby layers so that gems is loaded after ruby (takes precedence). Possibly changing to ruby to binary_ruby.
• Possibly: Symlink binaries to the binstub directory so they take precedence in the order of creation/addition.

GUS-W-17544084

[edmorley]

@schneems For Python I ended up changing the layer name to get the ordering I needed (and have integration test coverage of the PATH value):
https://github.com/heroku/buildpacks-python/blob/871a13266d2b31cbde4e9570439dd2ad9a88040a/src/layers/pip_dependencies.rs#L36-L39
https://github.com/heroku/buildpacks-python/blob/871a13266d2b31cbde4e9570439dd2ad9a88040a/tests/pip_test.rs#L86

(Also, prior to an upstream fix the lifecycle ordering actually differed between build time and run time which made this layer naming ordering quirk even more insidious: buildpacks/lifecycle#1393)