Native extension can not be loaded for gem built from git

[timon]

Hi! This could be related to #3413 and #3014

My program can not load native extension for eventmachine gem when it is specified with github flag in Gemfile.

The extension is built in gem's /ext/ dir, but is not copied into /lib/

Below are relevant excerpts with description of my environment (Host is CentOS 7)

$ bundle -v
Bundler version 1.10.6

$ ruby -v
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]

$ grep eventmachine Gemfile
  gem 'eventmachine', github: "eventmachine/eventmachine"

$ cat .bundle/config

---
BUNDLE_FROZEN: '1'
BUNDLE_PATH: vendor/bundle
BUNDLE_DISABLE_SHARED_GEMS: '1'

$ rm -rf vendor/bundle && bundle install | grep eventmachine
Fetching git://github.com/eventmachine/eventmachine.git
Using eventmachine 1.0.8 from git://github.com/eventmachine/eventmachine.git (at master)

$ bundle exec irb -reventmachine
Unable to load the EventMachine C extension; To use the pure-ruby reactor, require 'em/pure_ruby'
[PATH]/vendor/bundle/ruby/bundler/gems/eventmachine-8135b8c055b1/lib/eventmachine.rb:8:in `require':LoadError: cannot load such file -- rubyeventmachine

The extension is built, however, and cp could be used as workaround:

$ ls vendor/bundle/ruby/bundler/gems/eventmachine-8135b8c055b1/{ext,lib}/rubyeventmachine.*
ls: cannot access vendor/bundle/ruby/bundler/gems/eventmachine-8135b8c055b1/lib/rubyeventmachine.*: No such file or directory
vendor/bundle/ruby/bundler/gems/eventmachine-8135b8c055b1/ext/rubyeventmachine.so

$ cp vendor/bundle/ruby/bundler/gems/eventmachine-8135b8c055b1/{ext,lib}/rubyeventmachine.so

$ bundle exec irb -reventmachine
irb(main):001:0> $LOADED_FEATURES.grep /rubyeventmachine/
=> ["[PATH]/vendor/bundle/ruby/bundler/gems/eventmachine-8135b8c055b1/lib/rubyeventmachine.so"]

[segiddins]

@timon so sorry this never got a response -- could you check whether 1.12.0.pre.2 improves things for you?

[timon]

@segiddins No, tried with 1.12.0.rc, same problem

[b-ggs]

@timon Can't seem to reproduce this with 1.12.5 on Arch. Is this still an issue with this version?

[coilysiren]

ping @timon

Can't seem to reproduce this with 1.12.5 on Arch. Is this still an issue with this version?

[jcoleman]

I can confirm that prior to 1.12.5 I saw this issue when referencing a gem with a git ref in my Gemfile but upgrading to 1.12.5 fixed it.

[jcoleman]

@segiddins: A new wrinkle to add: vending doesn't work, but if I delete the cached gem it works.

Test case: https://gist.github.com/jcoleman/25c66a32c8338abb027471a0e8efe71a

If I run:

bundle install
bundle exec ruby test_case.rb

everything runs as expected.

But if I run:

bundle package --all
bundle install
bundle exec ruby test_case.rb

then it fails as it is unable to load the native extension.

If after running the vendor/package example above I then run:

rm -rf vendor/cache/pg_query-5ce0265749dd
bundle exec ruby test_case.rb

then it's back to working.

[b-ggs]

Could you check if running 1.13.0.rc.1 fixes this?

boggs@hackbook ~/dev/test-bundler/4106
$ dbundle install
Fetching git@github.com:jcoleman/pg_query.git
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using json 1.8.3
Using bundler 1.13.0.rc.1
Using pg_query 0.10.0 from git@github.com:jcoleman/pg_query.git (at master@8d0a8b6)
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
boggs@hackbook ~/dev/test-bundler/4106
$ dbundle exec ruby test_case.rb
boggs@hackbook ~/dev/test-bundler/4106
$ dbundle package --all
Using json 1.8.3
Using bundler 1.13.0.rc.1
Using pg_query 0.10.0 from git@github.com:jcoleman/pg_query.git (at master@8d0a8b6)
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Updating files in vendor/cache
  * json-1.8.3.gem
boggs@hackbook ~/dev/test-bundler/4106
$ dbundle install
Using json 1.8.3
Using bundler 1.13.0.rc.1
Using pg_query 0.10.0 from git@github.com:jcoleman/pg_query.git (at /Users/boggs/dev/test-bundler/4106/vendor/cache/pg_query-8d0a8b68c381@8d0a8b6)
Updating files in vendor/cache
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
boggs@hackbook ~/dev/test-bundler/4106
$ dbundle exec ruby test_case.rb
boggs@hackbook ~/dev/test-bundler/4106
$ dbundle --version
Bundler version 1.13.0.rc.1

Didn't seem to run into any issues while trying to repro on this version.

[jcoleman]

@b-ggs I was able to reproduce again; it seems my initial repro steps comment was slightly incorrect.

LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$ gem install bundler -v "1.13.0.rc.1"
Fetching: bundler-1.13.0.rc.1.gem (100%)
Successfully installed bundler-1.13.0.rc.1
1 gem installed
LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$ rm -rf vendor/
LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$ bundle package --all
Using json 1.8.3
Using bundler 1.12.5
Using pg_query 0.11.0 from git@github.com:jcoleman/pg_query.git (at f870ce1@f870ce1)
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Updating files in vendor/cache
  * json-1.8.3.gem
LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$ bundle exec ruby test_case.rb
/Users/jamecoleman/Downloads/25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c/vendor/cache/pg_query-f870ce136ea1/lib/pg_query.rb:4:in `require': cannot load such file -- pg_query/pg_query (LoadError)
        from /Users/jamecoleman/Downloads/25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c/vendor/cache/pg_query-f870ce136ea1/lib/pg_query.rb:4:in `<top (required)>'
        from test_case.rb:1:in `require'
        from test_case.rb:1:in `<main>'
LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$ rm -rf vendor/cache/pg_query-f870ce136ea1/
LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$ bundle exec ruby test_case.rb
LM-CHI-21000962:25c66a32c8338abb027471a0e8efe71a-f4743c07b8c9be35c81e09a80eaac52f370f0f7c jamecoleman$

[jcoleman]

I also updated my gist since it seemed to reference a commit that my machine couldn't resolve for some reason; not sure why it wasn't a problem for you.

[b-ggs]

@jcoleman Hmm, is this not the expected behavior when if you haven't run bundle install after running bundle package --all? (cc: @segiddins @RochesterinNYC)

boggs@hackbook.local ~/dev/test-bundler/4106
$ cat Gemfile
gem "pg_query", :git => "git@github.com:jcoleman/pg_query.git", :ref => "f870ce136ea15a33cf9e2aabf7e8b20607061d8a"
boggs@hackbook.local ~/dev/test-bundler/4106
$ cat test_case.rb
require 'pg_query'
PgQuery.parse("SELECT 1")
boggs@hackbook.local ~/dev/test-bundler/4106
$ dbundle package --all
Fetching git@github.com:jcoleman/pg_query.git
Resolving dependencies...
Using json 1.8.3
Using bundler 1.13.0.rc.1
Using pg_query 0.11.0 from git@github.com:jcoleman/pg_query.git (at f870ce1@f870ce1)
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Updating files in vendor/cache
  * json-1.8.3.gem
boggs@hackbook.local ~/dev/test-bundler/4106
$ dbundle exec ruby test_case.rb
/Users/boggs/dev/test-bundler/4106/vendor/cache/pg_query-f870ce136ea1/lib/pg_query.rb:4:in `require': cannot load such file -- pg_query/pg_query (LoadError)
      from /Users/boggs/dev/test-bundler/4106/vendor/cache/pg_query-f870ce136ea1/lib/pg_query.rb:4:in `<top (required)>'
from test_case.rb:1:in `require'
      from test_case.rb:1:in `<main>'
boggs@hackbook.local ~/dev/test-bundler/4106
$ dbundle install
Using json 1.8.3
Using bundler 1.13.0.rc.1
Using pg_query 0.11.0 from git@github.com:jcoleman/pg_query.git (at /Users/boggs/dev/test-bundler/4106/vendor/cache/pg_query-f870ce136ea1@f870ce1)
Updating files in vendor/cache
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
boggs@hackbook.local ~/dev/test-bundler/4106
$ dbundle show pg_query
/Users/boggs/dev/test-bundler/4106/vendor/cache/pg_query-f870ce136ea1
boggs@hackbook.local ~/dev/test-bundler/4106
$ dbundle exec ruby test_case.rb
boggs@hackbook.local ~/dev/test-bundler/4106
$

Seemed to work fine after running bundle install after packaging the gems, and bundle show pg_query seems to point to the packaged gem inside vendor/cache.

(By the way, I modified your gist to remove the commit reference since my machine couldn't resolve that revision either, that's why it worked for me. Used your exact Gemfile to repro this time around though.)

[jcoleman]

@b-ggs I'm not sure if that's expected behavior officially or not. At the very least I'd say it would be confusing behavior, since the package run prints out output exactly like an install, and, in fact, installs the gems into the vendor cache directory.

[segiddins]

I cannot reproduce the bug using bundler 1.14.0.pre.2, using the steps given in #4106 (comment)

[segiddins]

Closing due to lack of ability to reproduce. If the issue persists, please open up a new bug?