Use RUBY_ENGINE_VERSION to decide the GEM_HOME

[eregon]

• Such that each installed Ruby has its own GEM_HOME, even if RUBY_VERSION remains the same for multiple releases of a non-MRI Ruby implementation.
• This is particularly important for TruffleRuby, where different releases with the same RUBY_VERSION might compile C extensions differently, and each release should have a different GEM_HOME.

cc @postmodern @havenwood

[havenwood]

I think defined?(RUBY_ENGINE_VERSION) here should be Object.const_defined?(:RUBY_ENGINE_VERSION), since mruby doesn't implement the defined? keyword.

[eregon]

Good point, fixed.
I wonder if we even need to check that RUBY_ENGINE_VERSION is defined, it's been there for a long time, but probably better safe than sorry.

I tried with MRuby 1.4.0 and 2.0.0, and it works except chruby mruby shows:

trace (most recent call last):
	[0] -:4
-:4: uninitialized constant LoadError (NameError)

(which is the same on master)

[havenwood]

I was glad you checked, since RUBY_ENGINE_VERSION was added to CRuby in 2.3.0.

[havenwood]

JRuby sets --env-shebang by default and it rarely makes API changes without a RUBY_VERSION bump, so it currently effectively shares gems between RUBY_ENGINE_VERSIONs of the same RUBY_VERSION. This change would mean that most JRuby versions would no longer share gems. On the other hand, the new behavior is consistent between engines, in that none would share gems between versions.

In the future, we could consider more broadly supporting sharing gems between versions of the same ABI if RubyGems adopts a default --env-shebang.

[eregon]

I think this is also safer for JRuby. JRuby doesn't have C extensions but it has JRuby extensions which access JRuby internals that could change between JRuby releases with the same RUBY_VERSION and cause different javac compilation of JRuby extensions, leading to confusing require-time errors. While probably not common, it's certainly a risk and I think it's better for every Ruby installed to have its own GEM_HOME, rather than having this safety only for MRI.

Pure-ruby gems could be shared arbitrarily between all installed Rubies, I think that would be a way to share many gems safely if desired.

[postmodern]

I like this feature as it returns chruby to it's original behavior back when the patch-level was included in the version string. However, changing GEM_HOME to use RUBY_ENGINE_VERSION will change the current behavior, so we'll need to release this in 0.4.0 or make a 1.0.0 release, and notify users of the change.

[eregon]

@postmodern Good to hear 😃

I think 0.4.0 would be fine, as it's not really a hard breaking change (in fact it's safer behavior, so arguably some sort of bug fix), but it will require users to reinstall gems on alternative Ruby implementations (no changes on MRI). Since for most cases reinstalling gems is just a few bundle install away, I think it's a fairly easy process.

notify users of the change.

Do you mean an entry in the changelog, and/or something else?

[postmodern]

Definitely a ChangeLog entry and maybe a note in the README.

[eregon]

@postmodern I added a ChangeLog entry and updated the document paths for GEM_HOME in the README. What do you think?

[eregon]

@postmodern Does the PR look good now? Could you merge it?

[eregon]

@postmodern @havenwood This is becoming a real problem for TruffleRuby users, so I would like to merge this or use RbConfig::CONFIG['ruby_version'].
Related: oracle/truffleruby#1715

I think chruby (and Ruby managers in general, and Bundler, and RubyGems) should use either:

• RUBY_ENGINE_VERSION: So gems for MyRuby 1.2.3 are not reused for for MyRuby 1.2.4 (which can lead to nasty bugs due to C extension ABI changing between MyRuby versions).
• RbConfig::CONFIG['ruby_version'] which seems considered the ABI version, and is used by RubyGems and Bundler.

For chruby, I would think RUBY_ENGINE_VERSION is a nicer transition as it wouldn't change anything for MRI users, only for JRuby/TruffleRuby/Rubinius users which would need to reinstall their gems when upgrading chruby.

A third way would be to just use RubyGems' Gem.default_path, which is basically [Gem.default_dir, Gem.user_dir], and swap the order:

# With no environment, the defaults are:
  - GEM PATHS:
     - /home/eregon/.rubies/ruby-2.6.2/lib/ruby/gems/2.6.0
     - /home/eregon/.gem/ruby/2.6.0
# With chruby:
  - GEM PATHS:
     - /home/eregon/.gem/ruby/2.6.2
     - /home/eregon/.rubies/ruby-2.6.2/lib/ruby/gems/2.6.0

Or maybe we should really solve this in RubyGems by defaulting to the user directory first, but I think that is going to be a long battle.

[havenwood]

I'd love to use RbConfig::CONFIG['ruby_version'] and go by pseudo-ABI, but it seems that's blocked until RubyGems supports --env-shebang by default.

I think this PR is the best path for now.

[havenwood]

We could also try to go the JRuby route with CRuby and TruffleRuby and enable --env-shebang upstream. It makes sense to me to make the change in RubyGems, but it unfortunately didn't make the RubyGem 3.0 release so it'll likely have to wait until 4.0 on the RubyGems side.

[eregon]

I gave this more thought over the weekend, and actually I think the best way would be use the directory name used by chruby for the user gem home. For instance, ruby-2.6.3 would store gems in ~/.gem/ruby-2.6.3.

That way, even if I have ruby-2.6.1-enabled-shared (configured with --enabled-shared) and ruby-2.6.1, they won't share the gem home, which is actually necessary as extensions compile differently with that setting and probably other settings too.
Doing that would fix both using MRI with different configure options and not mix gems from multiple ABI versions in the same directory.

@havenwood @postmodern Would you be OK with that? I'll make a PR soon with that idea.

[postmodern]

@eregon using the ruby install directory name for the gem home directory name is an interesting idea. This might also require compiling rubies with different installation directories (ex: ~/.rubies/ruby-2.6.3-shared vs ~/.rubies/ruby2.6.3) into differently named src directories.

I would also like to point out that RubyGems now stores the C extension library files in sub-directories based on the host architecture, Ruby ABI version, and static vs --enable-shared.

$ ruby-install ruby
...
$ ruby-install --install-dir ~/.rubies/ruby-enable-shared ruby -- --enable-shared
...
$ # in a new terminal
$ chruby ruby-2.6.3
$ gem install nokogiri
...
$ chruby ruby-enable-shared
$ gem install nokogiri
...
$ tree .gem/ruby/2.6.3/extensions/
.gem/ruby/2.6.3/extensions/
└── x86_64-linux
    ├── 2.6.0
    │   └── nokogiri-1.10.3
    │       ├── gem.build_complete
    │       ├── gem_make.out
    │       ├── mkmf.log
    │       └── nokogiri
    │           └── nokogiri.so
    └── 2.6.0-static
        └── nokogiri-1.10.3
            ├── gem.build_complete
            ├── gem_make.out
            ├── mkmf.log
            └── nokogiri
                └── nokogiri.so

This does now allow rubygems to keep C extensions for different Ruby ABI versions or configurations separate. The only remaining issue are the absolute env-shebang which will explicitly load the ruby which was used to install the gem last.

[eregon]

@postmodern Very interesting, I did not know RubyGems uses different directories per ABI version.
That should help to solve the problem of TruffleRuby now that we set RbConfig::CONFIG['ruby_version'] properly.

FWIW, here is the current bug:

$ chruby truffleruby-19.0.0
$ gem install msgpack
$ ruby -rmsgpack -e0
OK
$ chruby truffleruby-19.1.0
$ ruby -rmsgpack -e0       
~/.rubies/truffleruby-19.1.0/lib/mri/rubygems/core_ext/kernel_require.rb:54:in `require': Global variable rb_tr_true is declared but not defined. (RuntimeError)
	from ~/.rubies/truffleruby-19.1.0/lib/mri/rubygems/core_ext/kernel_require.rb:54:in `require'
	from ~/.gem/truffleruby/2.6.2/gems/msgpack-1.3.0/lib/msgpack.rb:11:in `<top (required)>'
	from ~/.rubies/truffleruby-19.1.0/lib/mri/rubygems/core_ext/kernel_require.rb:130:in `require'
	from ~/.rubies/truffleruby-19.1.0/lib/mri/rubygems/core_ext/kernel_require.rb:130:in `require'
	from -e:1:in `require'
	from -e:1:in `<main>'

Because both use the same compiled extension:

~/.gem/truffleruby/2.6.2/extensions/x86_64-linux/2.6.0/msgpack-1.3.0/msgpack/msgpack.su

Here is what happens when using latest master, which sets RbConfig::CONFIG['ruby_version'] properly:

$ chruby truffleruby-jvm

$ gem list | grep msgpack
Ignoring msgpack-1.3.0 because its extensions are not built. Try: gem pristine msgpack --version 1.3.0
msgpack (1.3.0)

$ ruby -rmsgpack -e0    
Ignoring msgpack-1.3.0 because its extensions are not built. Try: gem pristine msgpack --version 1.3.0
~/code/truffleruby-ws/truffleruby/mxbuild/truffleruby-jvm/jre/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- msgpack (LoadError)
	from ~/code/truffleruby-ws/truffleruby/mxbuild/truffleruby-jvm/jre/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb:54:in `require'
	from -e:1:in `require'
	from -e:1:in `<main>'

So it does list msgpack in gem list, but yet it warns the extension is not compiled.
gem pristine msgpack --version 1.3.0 does fix it, but it's very cumbersome to do that for many gems.
So that part is not very nice behavior. I think it's probably still nicer for a user experience to separate gem homes for different Ruby installations or versions. And as you said, the absolute shebang is still an issue.
But at least, the good news is the error is clearer than expected.

[eregon]

@postmodern @havenwood I've been playing with a radical idea: not setting any GEM_ variable. I've been using that for over a week now. This actually works great when rubies are installed under the $HOME (e.g., in ~/.rubies), as one can just install gems without sudo, and it just works: gems are installed inside the Ruby's prefix, so are guaranteed to be separate for each installed Ruby, and gem binaries get installed next to ruby in $RUBY_ROOT/bin. FWIW, rbenv also doesn't set GEM_* variables.

It's really awesome for developing TruffleRuby, because in that case I like to have MRI chosen with chruby, but I also want to be able to install gems on MRI and on TruffleRuby (via tool/jt.rb ruby in the truffleruby repo), without needing to switch the current ruby with chruby. This is super convenient to compare MRI and TruffleRuby behavior in the same shell.
This doesn't work if GEM_HOME is set as gems would get mixed in the same home, and so it requires a lot of workaround like removing GEM_HOME in the wrapper script to run TruffleRuby in development. That issue is partially documented in https://github.com/oracle/truffleruby/blob/master/doc/user/ruby-managers.md#using-truffleruby-without-a-ruby-manager

And of course, the diff is pretty nice and makes chruby reach another level of simplicity and minimalism, by letting RubyGems use its defaults: eregon@a9555c9
Notably, chruby $MYRUBY doesn't even need to startup the given ruby, which makes chruby faster, especially for Rubies with a slower startup.

Now, the open issues with that approach:

• What if Rubies are system-installed (not under $HOME)? Then gems probably require sudo to be installed or gem install --user, which is less convenient. Although, an easy workaround could be to add install: --user in .gemrc. RubyGems already has the concept of user gems installed under $HOME/.gem/$RUBY_ENGINE/$ABI_VERSION, which I think works fine (chruby would need to add $HOME/.gem/$RUBY_ENGINE/$ABI_VERSION/bin to PATH).
• Rubinius has its RubyGems EXECUTABLE DIRECTORY in gems/bin instead of bin (Rubinius does not install gems binary in $PREFIX/bin and so bundle and pry are not available rubinius/rubinius#3821). From the chruby ChangeLog, it seems GEM_HOME was initially set to handle Rubinius. This is unfortunate, but it's probably easy to workaround that, e.g., by adding $RUBY_ROOT/gems/bin to PATH if it exists.
• So-called "bundled gems" (pre-installed gems like did_you_mean) get mixed with user-installed gems. I think this is not much an issue in practice, but there is some value to be able to just rm -rf MESSED_UP_GEM_HOME and start from scratch, which is not that easy in that case.

What do you think?
Do you think this could be a good approach in chruby?
Or is it too incompatible? What specifically would be too incompatible?

[postmodern]

@eregon just so I'm totally crystal clear, what exactly is causing that rb_tr_true error in your first example? If both TruffleRuby 19.0 and 19.1 have the same RUBY_VERSION (thus sharing the same msgpack extension file), shouldn't their CRuby ABI's be roughly the same? Are you absolutely certain this isn't an issue with TruffleRuby's CRuby ABI layer?

---

Not setting GEM_HOME and letting RubyGems install into the ruby's gem dir will clearly not work. Rubies not installed by the user (aka /opt/rubies/ and sudo ruby-install ...) are non-writable. Furthermore, there is value in keeping user-installed gems separate from the ruby, in case you need to delete/re-install one or the other.

We could derive GEM_HOME from RUBY_ROOT, so that each Ruby would get it's own gem user-install directory. Although, it would prevent Rubies of the same RUBY_ENGINE and RUBY_VERSION from sharing gems; provided the ruby's ABI doesn't break and gem pristine --extensions was ran and --env-shebang was enabled... This is kind of an exotic feature though, so we could axe it in favor of stricter gem dir separation. Personally, I would prefer using RbConfig::CONFIG['ruby_version'] in the GEM_HOME to enable gem sharing/recycling between versions or security upgrades; having to reinstall your gems because you upgraded from x.y.4 to x.y.5 is annoying.

I also feel the need to point out chruby explicitly forbids Ruby-specific workarounds. Any solution we come up with has to work with all (current/stable) Rubies out-of-the-box and all possible setups (such as non-user-writable rubies in /opt/rubies/).

[eregon]

@eregon just so I'm totally crystal clear, what exactly is causing that rb_tr_true error in your first example?

They have the same RUBY_VERSION, which is 2.6.2, aka the version of the Ruby language they are compatible with. RUBY_VERSION is not the ABI version though. RbConfig::CONFIG['ruby_version'] is the closest to an ABI version, and it's what is used by RubyGems and Bundler. For practical purposes, the TruffleRuby's ABI changes for every release (e.g., macros are defined differently).

RbConfig::CONFIG['ruby_version'] was 2.6.0 in both 19.0 and 19.1 though, which indeed is a bug (now fixed): oracle/truffleruby#1715
With that fixed, I get Ignoring msgpack-1.3.0 because its extensions are not built, showing chruby should have different gem homes for different ABI versions or for each installed Ruby.

---

Not setting GEM_HOME and letting RubyGems install into the ruby's gem dir will clearly not work. Rubies not installed by the user (aka /opt/rubies/ and sudo ruby-install ...) are non-writable.

Right, that's issue 1 above, and I think the most problematic. It works fine for me, because I never install a ruby version globally, but there are other use cases that ruby-install and chruby support so I understand we cannot do it. It's OK, I'll just use my branch for now.

Passing --user to gem install is a workaround supported directly in RubyGems though, but I agree it's not as practical. What if RubyGems automatically installed in a user directory when it detects it cannot write to the Ruby's prefix? It could be an idea, but obviously that wouldn't help for older Rubies and RubyGems. If we fixed this in RubyGems, I'd be tempted to simply install to the user dir by default, but changing defaults is likely not going to be easy.

In summary, it's sad that RubyGems defaults are not convenient and even tend to force using sudo for gem install which seems very risky and is inconvenient.

Furthermore, there is value in keeping user-installed gems separate from the ruby, in case you need to delete/re-install one or the other.

Which is issue 3 above, indeed.

Although, it would prevent Rubies of the same RUBY_ENGINE and RUBY_VERSION from sharing gems; provided the ruby's ABI doesn't break and gem pristine --extensions was ran and --env-shebang was enabled... This is kind of an exotic feature though, so we could axe it in favor of stricter gem dir separation.

That sounds very exotic, already just by having 2 MRIs of the same RUBY_VERSION (and for non-MRI RUBY_VERSION does not relate to ABI). It would also not work well if one is --enable-shared and the other not. I believe anyone who had to use gem pristine found it inconvenient and confusing.

This stricter gem separation is what both RVM and rbenv do (RVM by setting GEM_HOME based on the Ruby name, rbenv by not setting GEM_HOME and not supporting system-wide Rubies).
IMHO, it's the safe and easy to understand way.

Personally, I would prefer using RbConfig::CONFIG['ruby_version'] in the GEM_HOME to enable gem sharing/recycling between versions or security upgrades; having to reinstall your gems because you upgraded from x.y.4 to x.y.5 is annoying.

That would be fine too, except that one would also needs to care about --enable-shared. And I would think other ./configure flags like --enable-shared can influence C-extension compilation (e.g., some debug flag that adds code in a macro), and make them ABI-incompatible.

And of course, env-shebang still blocks this (and forever will by the the rule below).
Unless maybe we'd write install: --env-shebang to ~/.gemrc?

I also feel the need to point out chruby explicitly forbids Ruby-specific workarounds. Any solution we come up with has to work with all (current/stable) Rubies out-of-the-box and all possible setups (such as non-user-writable rubies in /opt/rubies/).

Right. The line is a bit blurry to me though because, e.g., Object.const_defined?(:RUBY_ENGINE) is basically MRuby-specific and not so different from if Gem executable dir != prefix/bin.

---

I am sorry for this very long thread. I think I got a pretty full understanding of the situation now.
The current gem dir based on RUBY_VERSION is suboptimal, because it leads to gem pristine warnings and non-usable executables for non-MRI Ruby implementations which use the same RUBY_VERSION for multiple releases.
The only safe solution forward I see, to still support existing Rubies and system-wide Rubies, is to have one gem home per installed Ruby, i.e., base the gem home on the ruby's name (the name from chruby's list).
Incidentally, it would also solve the gem home issue I had 6 years ago with chruby #186.

[postmodern]

RbConfig::CONFIG['ruby_version'] was 2.6.0 in both 19.0 and 19.1 though, which indeed is a bug (now fixed): oracle/truffleruby#1715

If the problem is fixed upstream, I always advocate for releasing a new version, encouraging users to upgrade, and moving forward.

With that fixed, I get Ignoring msgpack-1.3.0 because its extensions are not built, showing chruby should have different gem homes for different ABI versions or for each installed Ruby.

This definitely can seem confusing since the user may not know the gem dir is reused, and may require gem pristine --extensions --executables. We should shift the discussion to what's more predictable and less annoying to the users.

What if RubyGems automatically installed in a user directory when it detects it cannot write to the Ruby's prefix?

I have advocated for this in the past, but to no avail. Imo, it makes sense that if someone runs gem install as a regular user, it should install into the user's gem home. If they run sudo gem install it should install into the "system gem home"; not that sudo runs in a separate shell environment thus bypassing the user's PATH, GEM_HOME, etc.

If we fixed this in RubyGems, I'd be tempted to simply install to the user dir by default, but changing defaults is likely not going to be easy.

While I am pro-upgrading and moving forward, unfortunately it would require a lot of work (PR and proving it would not break backwards compatibility) for RubyGems to consider accepting the feature.

It would also not work well if one is --enable-shared and the other not. I believe anyone who had to use gem pristine found it inconvenient and confusing.

I already pointed out that RubyGems uses different directories for shared vs static built extensions. Extensions built against Rubies without --enable-shared are installed into GEM_HOME/extensions/ARCH-OS/RUBY_ABI_VERSION-static directory, where as extensions built against --enabled-shared Rubies lack the -static suffix on their extensions directory. While gem pristine is well documented, it isn't frequently used, unless you have to rebuild or fix your gems.

That would be fine too, except that one would also needs to care about --enable-shared.

Again, this doesn't seem to be an issue. Unless you are using a much older version of RubyGems prior to the extensions/ directory compartmentalization?

Right. The line is a bit blurry to me though because, e.g., Object.const_defined?(:RUBY_ENGINE) is basically MRuby-specific and not so different from if Gem executable dir != prefix/bin.

Technically true, although all Rubies supported Object.defined? and it didn't change the existing behavior or introduce new functionality. The original logic (262a9f1) used defined?(:RUBY_ENGINE) because RUBY_ENGINE was added in CRuby 1.9.1, and I wanted to support 1.8.7 at the time. I fail to see how pointing out this minor detail adds to the discussion, which should be focused on gem installation directories.

---

The only safe solution forward I see, to still support existing Rubies and system-wide Rubies, is to have one gem home per installed Ruby, i.e., base the gem home on the ruby's name (the name from chruby's list).

It is looking like this is the consensus. However, there are two downsides that we must recognize. If each Ruby gets it's own separate gem home...

1. users will have to re-install their gems after upgrading to a new patch-release version (i.e. security update). This would require knowing the list of gems you wish to re-install or manually mving the gem dir and running gem pristine.
2. users will likely complain to chruby about used up disk space due to orphaned gem dirs. This would require remembering to rm -rf old gem dirs after manually uninstalling rubies.

I understand your desire to fix this for TruffleRuby users, but we must consider the negative impact on other users. How do we mitigate these negative impacts? Possibly a separate script for managing gem installation directories? Thoughts?

[eregon]

If the problem is fixed upstream, I always advocate for releasing a new version, encouraging users to upgrade, and moving forward.

Of course, we should have 19.2.0 soon with that fix.

I have advocated for this in the past, but to no avail.

Did you discuss this on the RubyGems tracker or so? That idea sounds great to me, I'd be curious why some people disagree.

It would also not work well if one is --enable-shared and the other not. I believe anyone who had to use gem pristine found it inconvenient and confusing.

I already pointed out that RubyGems uses different directories for shared vs static built extensions.

Yes I know, I meant even though RubyGems doesn't mix them up, it causes gem pristine warnings, which is a less-than-ideal experience.
And the complete deal killer for sharing a gem home between 2 Rubies is gem executables having an absolute path by default (no --env-shebang), so only working for one of the 2 Rubies.

It is looking like this is the consensus. However, there are two downsides that we must recognize. If each Ruby gets it's own separate gem home...

1. users will have to re-install their gems after upgrading to a new patch-release version (i.e. security update). This would require knowing the list of gems you wish to re-install or manually mving the gem dir and running gem pristine.

Doesn't RubyGems have a command to export a list of installed gems, that can be easily imported with RubyGems again?

2. users will likely complain to chruby about used up disk space due to orphaned gem dirs. This would require remembering to rm -rf old gem dirs after manually uninstalling rubies.

By patch-release version, I assume you mean Z in X.Y.Z (not the old pNNN which is no longer used), like 2.6.1 -> 2.6.2.

This is already the case, every installation of MRI already uses its own gem directory, so at least it's not any worse than before for MRI, and it's correct for other implementations.
In which case is it any worse than currently?

[eregon]

Closing this PR in favor of #419