Do not rebuild native extensions on every install for git dependencies

[grosser]

given a Gemfile with a native extensions

source "https://rubygems.org"
gem "rinku",                :git => "https://github.com/shajith/rinku.git", :tag => "v1.7.2-unicode"

the native extension should not be rebuild if it already was build the last time around

bundle --local --path vendor/bundle --quiet

basically make lib/bundler/source/path.rb:188 smarter

this should bring extra speed when bundling + allow to re-bundle without having a running app try to load a invalid/missing c extension

[grosser]

Monkey-patch:

# https://github.com/bundler/bundler/issues/3014
# Do not re-build native extensions: speedup + avoiding crashing already running apps
# in Gemfle
Bundler::GemInstaller.class_eval do
  def build_extensions
    marker = "#{gem_dir}/ext-built"
    return if File.exist?(marker)
    result = super
    File.write(marker, "true")
    result
  end
end

before: 1.2s for bundle + running apps crash when re-bundling
after: 0.6s for bundle + happy deploys

[indirect]

This is a good idea, if your monkey-patch makes it impossible to ever update the gem. You’ll keep using that old c-ext for the rest of your life, no matter how many times the git repo gets new commits and you update to use them.

On May 6, 2014, at 12:20 PM, Michael Grosser notifications@github.com wrote:

Monkey-patch:

Bundler::GemInstaller.class_eval do
def build_extensions
marker = "#{gem_dir}/ext-built"
return if File.exist?(marker)
result = super
File.write(marker, "true")
result
end
end
—
Reply to this email directly or view it on GitHub.

[grosser]

already tried that, new version will have a new sha -> different folder, so
it will rebuild the c-ext

On Sun, May 11, 2014 at 3:02 AM, André Arko notifications@github.comwrote:

This is a good idea, if your monkey-patch makes it impossible to ever
update the gem. You’ll keep using that old c-ext for the rest of your life,
no matter how many times the git repo gets new commits and you update to
use them.

On May 6, 2014, at 12:20 PM, Michael Grosser notifications@github.com
wrote:

Monkey-patch:

Bundler::GemInstaller.class_eval do
def build_extensions
marker = "#{gem_dir}/ext-built"
return if File.exist?(marker)
result = super
File.write(marker, "true")
result
end
end
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3014#issuecomment-42766664
.

[andremedeiros]

He's right, @indirect.

I'm assigning this to myself and adding a test / fix for it.

[andremedeiros]

@grosser I've spent some time thinking about this and came up with the following question: wouldn't your patch make it so that if I package, say, a Rails app, it wouldn't recompile even if it was under a different system?

[grosser]

it would not do the same for normal gems, right ?
(if not the file can contain the hostname / ip / xyz identifier)

On Tue, May 13, 2014 at 9:51 AM, André Medeiros notifications@github.comwrote:

@grosser https://github.com/grosser I've spent some time thinking about
this and came up with the following question: wouldn't your patch make it
so that if I package, say, a Rails app, it wouldn't recompile even if it
was under a different system?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3014#issuecomment-42981228
.

[mbishop-fiksu]

+1 for this issue. We've noticed when we deploy to a running instance, bundle install will replace the native-library in a c-extension and that causes Passenger to fail if that gem happens to be accessed at the same time. Until it is restarted, Passenger appears to continue to fail on calls to that gem.

[indirect]

@andremedeiros any progress on this one? I would like to include this fix in the next release. :)

[TimMoore]

@andremedeiros is right: this causes the a git gem cached on one platform not to rebuild when you install on another platform... someone who was unwittingly using this monkey patch reported this on IRC.

We'll need a solution to that before we incorporate this change. Maybe something as simple as dropping the ext-built file in a platform-specific directory? It would probably be a good idea to have a flag to force a rebuild, too.

[andremedeiros]

I'm with @TimMoore. We need to finger-print the machine (and repeatable throughout similar setups.)

[indirect]

I'm unclear on how you can install an already-compiled git extension on a different platform? Bundler explicitly doesn't support copying installed bundles between machines.

[TimMoore]

This is when you bundle package --all. The git repo gets cloned into vendor/cache and used from there. The extensions are also built into that directory, which is a little weird.

[indirect]

Yeah, that is a bit weird. :/ At a minimum, they shouldn't be checked in, but I would expect to clone from that repo as part of install as usual so that install creates a clean copy and a new built extension for the computer being installed on.

On Wed, Jul 2, 2014 at 10:22 PM, Tim Moore notifications@github.com
wrote:

This is when you bundle package --all. The git repo gets cloned into vendor/cache and used from there. The extensions are also built into that directory, which is a little weird.

Reply to this email directly or view it on GitHub:
#3014 (comment)

[TimMoore]

Essentially, it looks like Bundler treats a cached git repo like a path repo, except that it also builds extensions.

[simsicon]

@indirect @TimMoore I met the situation, I need to package my web app and all my gems including git repos to vendor/cache, and deploy the package to an environment without Internet connection in other platform. Maybe bundler can just try to remove old extension building outputs, so that it would not break bundle install --local.

[ndbroadbent]

+1, @grosser's patch just fixed a big issue for us, where one of our C extensions would be recompiled during a deploy, when it ran bundle install. If someone happened to hit a controller action that called the extension, it would take our site down until the webserver processes were restarted at the end of the deploy. Same for any background workers that called out to the extension.

[indirect]

The solution to this problem is probably to build extensions into a platform-specific directory, the way Rubygems now does for installed gems with extensions. We'll still need to include the sha in the platform-specific directory name, though.

[egonSchiele]

+1, this is an issue for us too.

[moeffju]

👍 We run into this or a similar issue with the debase gem. When using it from a git repository and running bundle install, the following artifacts get dropped in our vendor/cache:

?? vendor/cache/debase-264cddf71b48/ext/.RUBYARCHDIR.time
?? vendor/cache/debase-264cddf71b48/lib/debase_internals.bundle

We have .gitignored the vendor/cache/extensions/ directory since it is architecture- and sometimes machine-specific, but debase does not put its C extension there.

[Intrepidd]

👍 bon courage working on this !

[segiddins]

@indirect / @andremedeiros this can be closed as of 1.12.0.pre.2, no?

[indirect]

I think so, yes.