Support requiring Gemfiles inside of Gemfiles (i.e. Gemfileception)

[dtognazzini]

Background

Gems, by themselves, only declare dependencies; they do not inform how to satisfy those dependencies. Bundler declares gem dependencies for package management and also specifies how to satisfy the dependencies (e.g. source, path, gem(git:), etc.)

It would be nice to reuse Gemfile declarations (both dependency specification and satisfaction) in other Gemfiles without copying and pasting.

Proposal

Provide a require_gemfile method in Bundler::DSL that takes a path to another Gemfile that can hold common dependency resolution declarations. For example:

<root>/Gemfile

require_gemfile "common/Gemfile"
gem "some_gem"

<root>/common/Gemfile

source "https://my_fav_gemserver.net"
path "gems" do
  gem "my_fav_gem"
end

The above would be identical to the following single file:

<root>/Gemfile

source "https://my_fav_gemserver.net"
path "common/gems" do
  gem "my_fav_gem"
end
gem "some_gem"

Notice: the resulting Gemfile nests the relative paths of the required Gemfile (common/Gemfile) under the relative path from the requiring Gemfile (common/).

Use Case 1: Sharing common gems

This is very similar to rubygems/bundler#3102, which uses eval_gemfile. In addition to sharing dependency declaration, require_gemfile allows sharing all supported DSL methods.

What require_gemfile offers over eval_gemfile is that it rebases relative paths from the required Gemfile to relative paths from the requiring Gemfile.

Use Case 2: Sharing source code, repository local gem paths

It's become a common pattern amongst Rails projects to decompose the system into an ecosystem of inline gems containing Rails Engines. The main Rails Application pulls in the inline gems by declaring dependencies in the application-level Gemfile. For example:

<system_root>/Gemfile

path "gems" do
  gem "subsystem_one"
  gem "subsystem_two"
  gem "subsystem_three"
end

If subsystem_one grows crazily, it could be further decomposed into many gems, perhaps:

<system_root>/subsystem_one/subsystem_one.gemspec

 s.add_runtime_dependency("subsystem_one-authorization")
 s.add_runtime_dependency("subsystem_one-billing")
 s.add_runtime_dependency("subsystem_one-analytics")

<system_root>/subsystem_one/Gemfile

# path "gems" here would be <system_root>/subsystem_one/gems
path "gems" do
  gem "subsystem_one-authorization"
  gem "subsystem_one-billing"
  gem "subsystem_one-analytics"
end

To support this decomposition, the main application's Gemfile would have to be updated:

<system_root>/Gemfile

# Declare dependencies on subsystems
path "gems" do
  gem "subsystem_one"
  gem "subsystem_two"
  gem "subsystem_three"
end

# Add the path to find gems required by subsystem_one
path "subsystem_one/gems"

Updating the top level Gemfile for subsystems undergoing structural refactors like this is somewhat painful but reasonably maintainable provided there is only one, conventional "gems" repository for each layer.

When it becomes unmaintainable

This strategy becomes unmaintainable with one more level of layering.

For example, consider a super_system project that pulls in the root of the above project (named here as system_1) and the root of another project (system_2):

<super_system>/systems/system_1.gemspec

 s.add_runtime_dependency("subsystem_one")
 s.add_runtime_dependency("subsystem_two")
 s.add_runtime_dependency("subsystem_three")

<super_system>/Gemfile

path "systems" do
  gem "system_1"
  gem "system_2"
end

The super_system project would have to include paths in its Gemfile for the gems required by system_1:

<super_system>/Gemfile

# Declare dependencies on the systems
path "systems" do
  gem "system_1"
  gem "system_2"
end

# Add paths to find gems required by system_1 and its subsystems
path "systems/system_1/gems"
path "systems/system_1/gems/subsystem_one/gems"

The last line couples super_system to the internal structure of system_1.

Contrived?

This example may seem a bit contrived, but I work on something similar to this whereby a system is comprised of many Rails Applications. Each Rails Application supplies its own inline gem that the system depends on in its Gemfile. The inline gems provided by the Rails Applications are used by the system to interact with the applications. In the usage above, every restructuring internal to system_1 would need to be handled by the Gemfile for super_system.

With require_gemfile, using a convention whereby systems specify their path repositories in a Gempaths file, the above would be simpler:

<super_system>/Gemfile

# Reuse gem paths defined by system_1
require_gemfile "systems/system_1/Gempaths"

path "systems" do
  gem "system_1"
  gem "system_2"
end

<super_system>/systems/system_1/Gemfile

# Use gem paths defined for system_1
require_gemfile "Gempaths"

<super_system>/systems/system_1/Gempaths

# Reuse gem paths defined by the subsystems
require_gemfile "gems/subsystem_one/Gempaths"
require_gemfile "gems/subsystem_two/Gempaths"
require_gemfile "gems/subsystem_three/Gempaths"

<super_system>/systems/system_1/gems/subsystem_one/Gemfile

# Use gem paths defined for subsystem_one
require_gemfile "Gempaths"

<super_system>/systems/system_1/gems/subsystem_one/Gempaths

path "gems"

With the above, no layer knows about the internals of the layer beneath it.

[dtognazzini]

@johnnyshields Per rubygems/bundler/issues/2688 and rubygems/rubygems/issues/702, you may find the proposal presented in this issue interesting.

I too work on Rails apps that are composed of many engines and I've felt the pain of having to copy-and-paste git: and path: declarations across Gemfiles.

With require_gemfile, you could put common declarations in a shared Gemfile to be reused in other Gemfiles. Although, I still use gemspecs for declaring library dependencies.

[indirect]

In the general case, gems do in fact declare how to satisfy dependencies: via gem servers, which are declared in the Gemfile using source. If you have application specific needs (a patch that isn't upstream yet, a vendored and unpacked internal library) you can use git or path in your Gemfile as a sort of escape hatch.

While Bundler supports gems from paths or git repos, the happy path is still (and is intended to be) installing gems directly from servers. Most users don't have more than one Gemfile in a single repository, since most users only have one application in each git repo. For repos that have many gems, like Rails, there is a single unified gemfile that declares the Rails gem in the current directory. Bundler is smart enough to automatically find all of the Rails sub-dependencies inside the same directory, and they don't have to be explicitly spelled out, path by path.

To zoom way, way out here... what exactly is it that you want to do with Bundler that led you to this feature request? Maybe it can be solved in a way that doesn't requiring adding something this complex to Bundler.

[dtognazzini]

With respect to dependency resolution, I mean that gemspecs by themselves do not specify how their dependencies are satisfied, only that they have dependencies.

I took a look at the Rails repo and I see what you mean: there is a single top level Gemfile that uses the gemspecDSL method and the related active* gems are found locally.

I've described 2 uses cases above. I realize they're a bit difficult to read. One of the use cases is very similar to the Rails layout, although, I store the repo-local gems under a sub-directory, which means I have to specify path "sub-directory" in my Gemfile.

The Rails engine guide advises creating mountable engines wrapped in repo-local gems and suggests pulling in these gems into the main Rails application via: gem 'blorgh', path: "/path/to/blorgh". This is not the general case, but it is becoming fairly common for Rails projects. Using repo-local gems in this fashion is not a workaround, but intentional, as the repo-local gems will never be upstreamed to a gem server.

Rails engines come with a nested dummy Rails application to run tests against. Development for the engine uses a Gemfile local to the Rails engine; it's not the same as the main Rails application in the repo. One use of require_gemfile would be to reuse common Gemfile setup across engine-local Gemfiles.

I'll pull together a sample Rails project to exhibit what I mean. I won't have any time to do that until next week, though.

[johnnyshields]

This proposal seems good to me.

Like @dtognazzini, I am also requiring internal gems via the :path to their gemspec. The main limitation I face is that the gemspec syntax doesn't allow :git gem references. Requiring these internal gems via their Gemfile instead of gemspec would would be a reasonable way to resolve this limitation (since Gemfiles can use git directives). An alternative would be to support :git directives inside gemspecs, but as per previous conversations with the Rubygems authors this looks unlikely to happen. Another alternative is to host your own gem server, but that's quite a bit of extra work (pushing updates for each modification, etc).

[indirect]

The local trick works automatically on subdirectories. You don't need to do anything special to get the sub-gems, even if they are in subdirectories.

For projects that all share a lot of git dependencies, I'm not really clear of what is easier about git push than rake release to your private gem server. Gemfury offers private servers with a couple of clicks, and then you gain all the benefits of releases and versions that git lacks.

On Thu, Oct 30, 2014 at 9:55 AM, Johnny Shields notifications@github.com
wrote:

This proposal seems good to me.

Like @dtognazzini, I am also requiring internal gems via the :path to their gemspec. The main limitation I face is that the gemspec syntax doesn't allow :git gem references. Requiring these internal gems via their Gemfile instead of gemspec would would be a reasonable way to resolve this limitation (since Gemfiles can use git directives). An alternative would be to support :git directives inside gemspecs, but as per previous conversations with the Rubygems authors this looks unlikely to happen. Another alternative is to host your own gem server, but that's quite a bit of extra work (pushing updates for each modification, etc).

Reply to this email directly or view it on GitHub:
#65 (comment)

[johnnyshields]

Suppose a sub-gem is dependent on a Github-only (unreleased) version of a gem, there's no way to maintain this reference in the sub-gem's gemspec currently. The best you can do is specify the Github reference in the parent Gemfile, but this is not ideal.

As far as using Gemfury, I have over 20 internal gems in my project and growing. It would be to much work to have to release every change for each gem.

[indirect]

You already have to git push each change for each gem; rake release is the same amount of work. I guess I’m pretty unclear on your workflow. Can you explain it in detail, like Donnie did for his use-case?

On Oct 30, 2014, at 11:53 AM, Johnny Shields notifications@github.com wrote:

Suppose a sub-gem is dependent on a Github-only (unreleased) version of a gem, there's no way to maintain this reference in the sub-gem's gemspec currently. The best you can do is specify the Github reference in the parent Gemfile, but this is not ideal.

As far as using Gemfury, I have over 20 internal gems in my project and growing. It would be to much work to have to release every change for each gem.

—
Reply to this email directly or view it on GitHub #65 (comment).

[johnnyshields]

I have all my gems inside one project. Each gem is inside the /engines/ folder. In my Gemfile, I have:

%w( subgem1 subgem2 subgem3 ... ).each do |engine|
  gem engine, path: File.expand_path("../engines/#{engine}", __FILE__)
end

[indirect]

@johnnyshields You don't need to do that. You can just do this:

# Gemfile
gemspec

# app.gemspec
Gem::Specification.new do |s|
  [...]
  s.add_dependency "subgem1"
  s.add_dependency "subgem2"
  s.add_dependency "subgem3"
end

[johnnyshields]

@indirect even using this more concise syntax, the problem remains that it references the gemspecs of the sub-gems, and gemspecs can't do things like reference Github dependencies.

[indirect]

@johnnyshields I may not be fully understanding you, but it sounds like you're saying that Bundler should add a very complicated feature that is likely to cause a lot of bugs so that you can run git push instead of rake release when you update sub-dependencies. Is that true? Is there something I'm missing?

[johnnyshields]

@indirect firstly I wish to be respectful for the great work you and others have done on Rubygems/Bundler--the "Extreme Makover" work you have done in the past 2 years has been a lifesaver for the community at large.

My issue is best explained by example. Suppose I have a "Parent" Rails project and "Child A" and "Child B" as subgems in a subdir of the project. I wish to declare the following:

1. Parent depends on Child A
2. Child A depends on Child B
3. Child B depends on a git-only version of a commonly used gem, e.g. "devise"

Conceptually I'd like to represent this as something like:

# Parent Gemfile
gem 'child_a', path: '/subgems/child_a'


# Child A gemspec (or Gemfile)
gem 'child_b', path: '../child_b'  # depends on Child B


# Child B gemspec (or Gemfile)
gem 'devise', git: 'plataformatec/devise', branch: 'not_yet_released'

However, the syntax in Child A (:path) and Child B (:git) are not supported in gemspecs. So the closest I can do is:

# Parent Gemfile
gem 'child_a', path: '/subgems/child_a'
gem 'child_b', path: '/subgems/child_b'
gem 'devise', git: 'plataformatec/devise', branch: 'not_yet_released'


# Child A gemspec
# reference to Child B omitted, since Child B doesn't exist on Rubygems


# Child B gemspec
Gem::Specification.new do |s|
  s.add_dependency 'devise'    # no version/git constraint
end

In addition, in order to test Child B isolation against the repo-version of devise, Child B must have a Gemfile which has the git devise reference. This is a duplication of the reference in the Parent Gemfile. This is non-DRY and becomes cumbersome when working with ~150 gem dependencies across the project.

Two possible solutions are:

1. Rubygems could support :git / :path dependencies in its gemspecs, OR

2. Bundler could support embedding Gemfiles within other Gemfiles (a la @dtognazzini's require_gemfile proposal).

It is true that I could host all my internal gems a private gem server, and rake release each time I make a change to a given gem, but I feel this is too cumbersome when the number of subgems (~25) greatly outnumbers the number of people in the organization (~5).

[johnnyshields]

One last note, even if I were to use a private gem server, it would not solve the issue of having to reference the repo version of devise in my Parent Gemfile (when I'd rather reference it from Child B's Gemfile / gemspec). Granted I could also release that repo version of devise as a private gem on my gem server, but all this becomes a time burden when managing 25 internal and 150+ external gems.

[indirect]

Okay, I think I understand what you're saying now. I have doubts about this entire scheme of organization as being overly complex. :)

Using gems from git is slower, adds complexity, and honestly is a (functional and helpful, but still a) hack. Using released gems greatly increases the speed of Bundle install, bundle exec and everything else Bundler does. Git gems are intended to be temporary when they are absolutely needed, and should not be standard operating procedure.

Again, it seems like your described situation requires git push and bundle update every time you update a sub-gem, so I'm not sure why you're objecting to rake release and bundle update, since it's literally zero more steps.

[johnnyshields]

My usage above does not require a git push / bundle update for my sub-gems. I reference sub-gems via :path, not :git. I do also think it's a valid use case to reference sub-gems via :git--if I could I might do that for certain gems which I've hired vendors to build--but it's not what I do currently.

The main issue I have with :git is that I cannot a reference a Git version of a commonly used library in the sub-gem gemspec. As a workaround I put this :git reference in my parent Gemfile, but this is unnatural since it's actually the sub-gem which has the dependency.

I agree with your statement that git references are a hack, but given the reality of version upgrades, bugs, gem maintainers gone AWOL, etc. they a necessary evil.

[johnnyshields]

Regarding the "complexity" of the scheme, the ultimate goal is use internal gems to reduce the complexity of the app. Gems are a very useful paradigm to "isolate" code into loosely-coupled components with well-defined APIs and dependencies. I often to delegate tasks to my team and vendors by requesting them to build a subsystem as a gem.

if we could resolve the :git / :path limitations I think many more people/teams would do things in this manner--it would even be a "best practice" for scaling out Ruby apps using loose-coupling IMHO.

[segiddins]

@johnnyshields I think the solution here is to use a gem server to enforce loose coupling, rather than using ':git/:path` as a permanent solution.

[johnnyshields]

I still think a gem server is overkill. Why should I have to setup a server and roundtrip it when I have the code right there in a subdir of my project? Why shouldn't we have other options to best suit the team's workflow?

[TimMoore]

@johnnyshields can you talk more about the team's workflow? Do people work on these gems independently of the parent project? Are the gems shared between projects (it sounds like no)? Are all of the gems in the same git repo as the parent project (it sounds like yes)?

If it's the case that these gems are specific to a project and not reusable outside of it, what is the benefit you get from structuring them as gems rather than simple subdirectories on your load path?

What is the current process that a new developer would use to set up the project and start working on it? How do you imagine an ideal process would be different?

What problems do you encounter by specifying all of the git gem overrides in the top-level Gemfile that would be solved by splitting them up?

[johnnyshields]

There are two types of sub-gems I use:

• Standalone Gems (in my /gems/ dir) are private gems that are completely isolated. Structure-wise they look like the gems on RubyGems, but I don't make them public.
• Rails Engines (in my /engines/ dir) are mountable rails apps as described here. Each Rails Engine has a gemspec so it is technically a "gem", but structure-wise these look like Rails apps (e.g. with /app/, /config/, etc dirs)

Do people work on these gems independently of the parent project?

Yes. I've had several cases where I've asked vendors to build specific gems based on an API spec and have not given them access to the main project repo, with great results.

Are the gems shared between projects (it sounds like no)?

Yes. I have multiple apps on different domains which share internal gems, for example tablesolution.com which is restaurant mgmt app and tablecheck.jp which is a public-facing reservation booking app, the core reservation engine shared by both is packaged as a gem. (Currently I have these two apps as separate "engines" in the same project, but I'd like to split them into separate projects if I could use :git references in my gemspecs.) In addition, I've had third parties contact me about licensing some of my components, and since they are already packaged as gems it makes it very easy to share the code.

Are all of the gems in the same git repo as the parent project (it sounds like yes)?

Currently yes, but I would like to move some gems to separate private git repos if I could get around the aforementioned limitations.

[TimMoore]

If they are all in the same git repo, then how do people work on them independently of the main project or share them between projects?

[johnnyshields]

I've put them in a separate repo for the vendor, then copy-pasted periodically :(

(Again this is due to the aforementioned limitations; allowing :git reference would save this headache!)

[TimMoore]

@johnnyshields so what would the ideal configuration & workflow look like for you? Let's say you had each gem/engine in a separate git repo... how would that work?

It sounds to me like the only solution that actually works in that case would be to release the gems (including forked gems) to a gem server, because as soon as the Gemfile for the child gem is not in the same repo as the parent project, the hypothetical require_gemfile wouldn't be an option.

[johnnyshields]

Let's say you had each gem/engine in a separate git repo... how would that work?

• I would reference most of my "engine gems" by path and most of my "standalone gems" by git.
• For my "engine gems", I would want to declare dependencies to both git versions of public gems (e.g. gem "devise", github: "plataformatec/devise", branch: "not-yet-released) and also to my other engines/gems.
• For my "standalone gems", I'd reference them by git tag or branch and use git as if it were gem server (minus the need to setup/maintain an extra server and run rake release each time).

If I only had "standalone gems" I'd probably be OK to use a gem server. It's really the "engines" that are the pain point, since:

• I have a lot of them
• it's far more expedient to reference them by path
• I can't declare dependencies from one engine to another
• I can't declare dependencies from an engine to a repo version of public gems.

as soon as the Gemfile for the child gem is not in the same repo as the parent project, the hypothetical require_gemfile wouldn't be an option.

As per above, this would not really be an issue for "engines" since they're referenced by path. But in order to support this, the require_gemfile could have a :git option itself which instructs to load from git but using the Gemfile in the repo instead of the gemspec.

But I think a better/cleaner solution would be to allow :git and :path references in the gemspec files instead, perhaps with an internal_mode! directive which enables these options but prevents release to Rubygems. I proposed this here: rubygems/rubygems#702

[segiddins]

Specifications should never declare how dependencies are fulfilled, only what they are.

[johnnyshields]

@segiddins the "what" which I wish to reference is the code at a certain system path or git tag. Unfortunately it can't be referenced without also specifying "how". Allowing this for private gems (not released to Rubygems) seems valid to me, because the "what" is not in the public domain and is directly under your control. In order to release to Rubygems one wouldn't be allowed to do this--all of the "whats" for a public gem must also be in the public domain.

[segiddins]

And that's what gem servers are for :)

-Samuel E. Giddins

On Nov 4, 2014, at 10:02 PM, Johnny Shields notifications@github.com wrote:

@segiddins the "what" which I wish to reference is the code at a certain system path or git tag. Unfortunately it can't be referenced without specifying "how". Allowing this for private gems (not released to Rubygems) seems valid to me, because often times this "what" is not in the public domain and is directly under your control. In order to release to Rubygems one wouldn't be allowed to do this--all of the "whats" in a public gem must also be in the public domain.

—
Reply to this email directly or view it on GitHub.

[johnnyshields]

Gem servers are not an ideal solution for the engines (path-based) use case as I and @dtognazzini have outlined above.

[johnnyshields]

I will put up a $3,000 bounty to resolve this limitation, i.e. allowing :git and :path references inside nested gems. I'm agnostic as to whether the issue is resolved in Bundler (Gemfiles) or Rubygems (gemspecs), so long as it is merged into master of either library.

[johnnyshields]

@ccutrer thanks for your awesome comment. I totally get it.

Recently I'm turning my attention towards node.js. Their package manager NPM fully supports the :git and :file recursive dependency use cases I've outlined above. Granted, NPM has it's own issues, but at least the Node community does not get hung up on these silly semantics of "what" versus "how".

[indirect]

@ccutrer The entire point of a lock file is to guarantee that the exact same gem versions are available in development and production. What you're describing is the opposite of that: you want to be able to run different versions of the same gems on different machines or different production environments. The closest we can allow to that use-case is using a built gem, and then supplementing one of that gem's dependencies by declaring a git or path gem in the Gemfile. That still produces a single logical outcome—one set of gems for all machines. How do you imagine the lock could possibly work in cases like the one you are asking for?

[ccutrer]

I know, and I wish we could do that (natively). Right now we essentially lock down our versions so tight in the Gemfile that the lockfile nearly matches the Gemfile. I think in an ideal world, there would be a lockfile corresponding to each Gemfile that all contribute to the master Gemfile, locking the gems in that specific file (and a final runtime check that none of those lockfiles conflict).

In practice, when we build a new release package, we include the lockfile in it, so we don't have strange problems later if a dependency changed (we also package the gems).

Another not-so-perfect solution is one suggested for per-user gemfile inclusion - having a separate lockfile. Then we could commit the generic lockfile excluding any additional plugins not included in the base project. On deploy, or for developers working with additional plugins, they get a separate lockfile of their own that's not commited (it would be nice if it was "bootstrapped" with the main lockfile, so they're not too far off, but that seems complicated to impossible to keep up to date).

[johnnyshields]

@ccutrer supposing gemspecs could specify :git and :file dependencies, which would be evaluated at higher priority than version numbers >= 0.3.0 (i.e. Rubygems), does that solve your problem?

[ccutrer]

I should mention that in a way we already monkey-patch bundler from our Gemfile to support a separate lockfile - if there is a specific environment variable to enable (experimental) support for Rails 4 in our app, it uses Gemfile.lock4. This allows us to both switch between "normal" and Rails 4 mode very quickly during development (not having to bundle update between them, or play some other trick swapping out lockfiles), but also when slowly rolling to production, we have a single release with both lockfiles, and we can A/B test them, or slow roll. Probably not something recommended for everyday use of bundler, though.

[ccutrer]

@johnnyshields it would slightly alleviate one of our use cases (i.e. using a Gemfile fragment in addition to a gemspec in order to specify the source of a dependent, non-released gem; which is really not a big deal). It does not solve maintaining a massive Gemfile by breaking it into multiple fragments, or managing a lockfile that may or may not have additional proprietary dependencies.

[johnnyshields]

IMHO the reason for the massive Gemfile is that git gem dependencies can't be specified at a per-gem level.

I don't see the point of breaking a large Gemfile into a lot of smaller Gemfiles. In theory you should be breaking your project into lots of smaller gems, each with it's own gemspec. Breaking up the Gemfile alone does nothing to reduce the complexity of your project codebase, your tests take the same amount of time to run, etc.--no real benefit other than purely cosmetic because you prefer to read N files of size M rather than 1 file of size N*M

[johnnyshields]

For the lockfile, for the sake of argument let's assume Bundler knows how to read the :git and :file references from the gemspecs and builds an appropriate lock-file considering those proprietary dependencies.

[ccutrer]

This is our "Gemfile", sans anything needed as dependencies of private gems: https://github.com/instructure/canvas-lms/tree/stable/Gemfile.d. While it would be trimmed down if we weren't explicitly mentioning dependencies-of-dependencies (for locking purposes), it would still be large and unwieldy. Actually, I should probably break rails3 vs. rails4 dependencies into their own files.
Also notice that we do indeed have a multitude of local gems (under the gems folder, and each mentioned by name in the other_stuff.rb part of Gemfile.d).

So, in a nutshell, no, our huge Gemfile (and wanting to break it up) has nothing to do with gem dependencies needing to specify their own dependencies as coming from git source, but a combination of it flat out being a massive project (historically monolithic, but slowly being broken up), and not properly using a lockfile (due to the proprietary plugin problem).

[johnnyshields]

OK it looks like you're using some conditional build logic depending on what the user wants, i.e. "postgres on Rails 4" would build one output Gemfile vs. "mysql on Rails 3" -- something like that?

[ccutrer]

Kind of. More like "is there stuff in gems/plugins? pull in all the gems there". so non-open sourced stuff just sits there, and you don't have to do anything else - the "master" Gemfile (more specifically, Gemfile.d/plugins.rb) dynamically pulls in the additional gems (or Gemfile.d/~after.rb for legacy plugins in vendor/plugins that aren't gems, but have a Gemfile to declare their dependencies. also support Gemfile fragments from plugins in gems/plugins is in that file, for them to declare the source for non-released gems). Having two layers of dependencies on local gems is actually quite rare, and even in that case is handled by them being siblings, and the master Gemfile automatically pulling in all the local gems.

[johnnyshields]

Just for reference, Node.js NPM actually allows you to load two different versions of the same package (=~ gem) at the same time if there are conflicting dependencies, due to the way it's module system works. Rubygems can't do this because it's gems pollute the global namespace with their constants, e.g. attempting to load ActiveRecord 4.0 would overwrite ActiveRecord 3.0 in memory if you've already loaded it. This sort of thing could be emulated in Ruby but is not the standard, see here for more detailed discussion: http://andrew.ghost.io/emulating-node-js-modules-in-ruby/

[dtognazzini]

It's been awhile since I've looked at this and it's quite difficult even for me to distill my exact use cases.

To summarize, in a single Rails project comprised of internal, project-local gems (or Rails Engines), I would use this feature for reusing the following specifications across Gemfiles used by the test environments for each project-local gem:

• path specifications to ensure all test environments can find other project-local gems.
• source specifications to ensure all test environments use the same version for gems external to the project.
• group specifications to share commonly used gems in the Rails' test environment.

All of these uses are policy decisions controlled by the project. If/when a project-local gem finds usefulness independent of the project, I would carve out the gem into a new repository and serve via a gem server.

This discussion explores one strategy of accomplishing this with require_gemfile in more detail.

Still I wonder about this:

Perhaps a different implementation whereby you could use plain-old require in a Gemfile would be sufficient. Rake supports this by extending the main object with the Rake DSL. Whereas Bundler instance_evals the Gemfile. I wonder if it would be possible to rework some of the internals to allow for using plain-old require.

In that world a path specification would be relative to the file it's used in.

[bbozo]

Hello 😃

we have a project dealing with 2 rails 4 apps hanging on the same database, a jruby project handling a mobile API and a C ruby CMS of sorts.

To be able to manage this we have extracted the common code into a rails engine (mostly models, some libs) and one day the API server started generating hashes for paperclip in a different way from the CMS server which with S3 means that S3 assets can't be accessed - and after much digging I figure that it's indirectly a freaky dependency issue - an easy mistake to make when you have have 2 Gemfile and 2 Gemfile.lock which share pretty much everything except simple form and choice of rails server.

So I set out to fix the problem once and for all so it doesn't repeat, and OK, I know that you can put the common gems into the gemspec of the engine so I did that. However when I tried that, I realized that those gems don't have git support, local path support and that they don't get required on their own, so it's a bit of a no go, or at least in many ways a step down from what we have now.

Then I google for "how to handle common gem dependencies with engine" and this was the first link, and only link

I was very happy, now I'll to go eat chocolate and think options 😞

[johnnyshields]

@bbozo, yes exactly. Welcome to my world. The current Gemfile / gemspec paradigm does not scale well with complex apps, and a large part of the problem is how github dependencies are handled. As the Ruby community seems to care little about this problem, I think the best answer is:

• build a time machine
• go back in time to when you were deciding which framework/language to write your app(s) with
• use node.js

[bbozo]

Ι'm starting to get the feeling that bundler source code has gone a bit amok,

basically there's a long line of closed or "low priority" issues that hold back the "split complex rails app into gems/engines" mantra that the Rails team has been pushing a couple years now.

For example:

1. Support requiring Gemfiles inside of Gemfiles (i.e. Gemfileception) #65 - this issue prevents keeping gem versions in sync among apps using a common engine
2. bundle install --local does not install packaged gem from git source bundler#3571 - :path is not supported with "package" - this means that any deployment mechanism depending on bundle package can't work with local gems
3. Way to bundle update conservatively a single gem bundler#2016 - workaround for last issue is to use git push on the engine and bundle update on the rails apps using it, which is irritating but OK - however when you go this route then every time you want to push a 1-liner code change bundler will try to update ALL of rails dependencies because an engine gem depends on rails. I tried removing rails from the engine gemspec, but it was no use, all of rails dependencies still got pulled

Unfortunately @indirect has been very defensive about all of these issues, according to him:

1. using :git is a hack, you should publish private engines to rubygems.com
2. using :path means "you handle packaging"
3. not updating rails engine dependencies when you want to patch a piece of code from the engine is simply hard and not going to happen

Unfortunately, one gets a feeling that this defensiveness comes from the possible fact that bundler has become to unwieldy to follow Rails development goals and emerging deployment standards,

I get this feeling because in this issue @indirect said that 3000$ isn't enough money for the trouble of developing a feature that should be a priority for the rails community anyway - the ability to handle large projects - so I'm guessing Rails has come to the end of the line where Bundler is concerned and that the rift will continue to grow.

Also, I get a bad feeling from the fact that this issue is the only one to which so far I have seen @indirect help someone circumvent a Bundler design problem, and this was only after he confirmed that @johnnyshields would pay 500$ for a hackish solution.

In fact he says it clearly:

#65 (comment)
(To be super clear to anyone reading this ticket later: the above is a clever but dirty hack, it could break in future versions of Bundler, and the Bundler team only provides help with things like this if you provide monetary compensation.)

All of this leaves a very bad taste in my mouth because I've personally lost at least 2 weeks of my life handling these issues, and some of these discussions read as an Oracle support worker trying to explain to his clients that what they're seeing is an intended feature and not a serious design issue that needs addressing, and any help will cost ya 500$

Not what I've come to expect from OS community....

Anyway, my 2 cents

[bbozo]

Rails core topic: https://groups.google.com/forum/#!topic/rubyonrails-core/Q0yj6XJEaew

[segiddins]

@bbozo I'd ask you to please be polite to André. He has spent thousands of hours on bundler, and he, I, and the rest of the bundler team care greatly about making bundler the best tool it can be.

As for this thread, I think it's gone on long enough. The official position of the bundler team is to reccomend people to publish their gems to their own gem servers, and that reccomendation is unlikely to change without a drastic change in how rubygems or bundler work.

As always, pull requests to bundler are very welcome to improve things you find to be rough edges, but please do keep in mind that we're the team stuck maintaining everything in the long run, and to that end we've made the decision that this feature is not one we wish to officially support. Thanks everyone for chiming in!

[johnnyshields]

The big problems with problems in my view:

1. Gemspec and gemfile are separate. Gemfile allows using :github/:path, the gemspec does not. Gemfile is for "projects", gemspec is for "gems".

• In NPM (node.js) there is a single file called "package.json" which handles both responsibilities in one schema.

2. Ruby gems pollute the global namespace. If I bundle two gems that define the namespace Foobar, one will monkey patch the other.

• Javascript (ES6) has a module system where each file must import and export. Thus I can do something like import Foobar from FoobarGem1 as Foobar and import Foobar from FoobarGem2 as Bazqux to avoid kludges.
• Further point: when NPM gets dependencies, it gets a tree of dependencies specific to each library. Hence you never get into the situation where two gems (A and B) with a common third dependency have a clash (A-->C >= 1.0 and B-->C < 1.0)--each gem gets the correct version of it's own dependencies.

Both of these are major obstacles to scaling Ruby in production and are costing me real money in wasted effort. Frankly speaking I much prefer coding in Ruby than Javascript, and Rails is a great framework. But the JS community has it's shit together, the Ruby community does not. Not to disparage the work Bundler and Andre have done, but Bundler has fundamental design flaws some of which stem from design flaws in Ruby (lack of ES6 or Python-like module system).

(Copying to Ruby Core thread)

[bbozo]

I appreciate your good work here, but some of the support practices seem to have went a bit off (500$ for a hack that was already implemented by the canvas-lms team https://github.com/instructure/canvas-lms/commits/stable/Gemfile.d/sqlite.rb) and also if there's a problem with code maintenance that can't be fixed - it'd be good to let people know so they can plan accordingly with new projects

Also, knowing the cost of the tech support would be good to know before making a Bundler.require in a new project

@segiddins @indirect please let me know if I can updates parts of my post to not be impolite while still reflecting reality, it is not my intent to insult ❤️

[johnnyshields]

@indirect has done a fantastic job solving the bundler problem under certain assumptions (global namespaces, separation of Gemspec vs. gemfile). If we remove those assumptions however (as NPM has done) a much more scalable solution becomes possible--imagine no more gem conflicts!

While this would certainly be huge undertaking, it is possible, and it could be done in a backwards compatible fashion. It's worth noting that Javascript has achieved such a transition, though it's taken several iterations. For example, there were a variety of competing attempts to solve the module (import/export) system via libs--RequireJS, AMD, etc--before it was ultimately incorporated at the language level in EcmaScript 6.

[johnnyshields]

@brixen interested to hear your thoughts on this.

[bf4]

@johnnyshields @bbozo @dtognazzini There's a lot of words here, and I may be misunderstanding your use case, but you might want to try this in your Gemfile so you can bundle a gem from git or from a local path: http://www.benjaminfleischer.com/2011/11/01/bundler-ruby-gem-development-tricks/

if ENV['APP_GEMS_DIR']
  # for local gem development, use local gem directory so gem changes don't need to be pushed to test
  gem 'my_gem', :path => "#{ENV['APP_GEMS_DIR']}/my_gem", :require => "namespace/my_gem"
else
  gem 'my_gem', :require => "namespace/my_gem", :git => path_to_git_repo, :tag => tagname
end

Rubygems isn't going to change to package and install gems (which is what the path/git options in bundler do. They do the job of gem build, in addition to gem install).

Whenever I've worked on code that started to get crazy with interactions between a gemspec and a Gemfile, I wanted to change the app architecture, not the tools. :)

If you think about it a gemspec's job is to declare dependencies for a library. A Gemfile's job is to declare dependencies for an app.

Also maybe see https://groups.google.com/forum/#!topic/ruby-bundler/G35fht6T3yA/

[johnnyshields]

@bf4 please read #65 (comment) carefully.

The problem is not "how to kludge together multiple Gemfiles", and cannot be fixed by a bandaid. There are fundamental design assumptions in Bundler / Rubygems that make scaling Rails beyond "weekend hobby" apps difficult. The more gems in your Gemfile, the more you feel the pain (I'm at 200 and counting)

[bf4]

@johnnyshields It's hard to say without seeing your code, but statements like scaling Rails beyond "weekend hobby" apps difficult. and Ruby gems pollute the global namespace. make it sound like you're missing something. Plenty of Rails app scale beyond weekend hobby. And the notion of unintentionally re-opening/re-defining classes in Ruby is one of design, rather than a language issue. In the old days, Rake had a module called Task. So, a Rails app with a Task model would interact with Rake in weird ways. Nowadays, the module in Rake would be Rake::Task. That is, good library conventions put the whole library under a namespace. That's just how it is.

I'm going to stop responding since I'm pretty certain what you want isn't possible without changing the nature of Ruby and rewriting much of its ecosystem.

[johnnyshields]

@bf4 have you looked at NPM and ES6 modules?

[johnnyshields]

In the old days, Rake had a module called Task. So, a Rails app with a Task model would interact with Rake in weird ways.

That's exactly what I mean by "polluting the global namespace." The problem is not merely name conflicts (which are a big pain), but even more so because you can only have a single version of a given gem living in your app ecosystem at once. Please study NPM / ES6, which once had the same limitations as Ruby, but has overcome them because the JS community is not content to say "That's just how it is."

[indirect]

@bbozo Bundler is both free and open source software. If you don't like how it works, you are welcome to take the source code and make it work the way you want it to instead. Many, many other companies have been able to make Rails engines scale for them without needing this feature. It might be more productive for you to ask them how they did it, rather than demand hundreds or thousands of hours of free software development.

@johnnyshields Bundler is only able to work within the constraints of Ruby and RubyGems. If you want changes to RubyGems, ask the RubyGems team, not the Bundler team. If you want changes to Ruby, ask the Ruby team. If you want to be using NPM instead, please use Node.

Unfortunately, the discussion in this issue has moved beyond a feature request for Bundler, and has turned into demands that gemspecs and even the nature of constants in Ruby itself should be changed. We appreciate the feedback, but we don't accept feature requests for either RubyGems or Ruby here, so I'm going to lock this ticket.

The Bundler team values feedback, a lot, and we are happy to discuss and work with anyone having problems to try to help them find a solution. Personal attacks and demands for hundreds or thousands of hours of development work are both counter-productive if what you want is a solution to your problem, and a good way to get threads locked and possibly even get yourself banned from the Bundler issue tracker.