weird bundle exec - nohup behavior where SIGHUP gets raised to child process

[oneamtu]

At the company I work for we ran into this weird problem when running processes through nohup and bundle exec. nohup is a utility that is supposed to trap SIGHUPs from bubbling to a child process, thus allowing a user to daemonize a command by running it with nohup and then closing the parent shell.

For whatever reason, with more recent versions of bundler past 1.12.5 SIGHUPs bubble up to the child process. 1.12.5 works great. We also found a workaround with more recent versions of bundler where the ordering of the commands changes the behavior - nohup bundle exec fails and bundle exec nohup works. Here's an example of the test which I uploaded here - https://github.com/oneamtu/bundler_nohup_test:

[19:56:02] oneamtu:test $ nohup rake loop_de_loop >> loop.log 2>&1 &
[1] 7553
[19:56:27] oneamtu:test $ kill -hup 7553
[19:56:34] oneamtu:test $ kill -kill 7553
[1]  + 7553 killed     nohup rake loop_de_loop >> loop.log 2>&1
[19:56:44] oneamtu:test $ nohup bundle exec rake loop_de_loop >> loop.log 2>&1 &
[1] 7657
[19:56:52] oneamtu:test $ kill -hup 7657
[1]  + 7657 exit 1     nohup bundle exec rake loop_de_loop >> loop.log 2>&1
[19:56:59] oneamtu:test $ bundle exec nohup rake loop_de_loop >> loop.log 2>&1 &
[1] 7746
[19:57:12] oneamtu:test $ kill -hup 7746
[19:57:17] oneamtu:test $ kill -kill 7746
[1]  + 7746 killed     bundle exec nohup rake loop_de_loop >> loop.log 2>&1

[6:09:53] oneamtu:test $ nohup --version
nohup (GNU coreutils) 8.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jim Meyering.

Environment

Bundler       1.16.0
  Platforms   ruby, x86_64-linux
Ruby          2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
  Full Path   /home/oneamtu/.asdf/installs/ruby/2.4.2/bin/ruby
  Config Dir  /home/oneamtu/.asdf/installs/ruby/2.4.2/etc
RubyGems      2.6.13
  Gem Home    /home/oneamtu/.asdf/installs/ruby/2.4.2/lib/ruby/gems/2.4.0
  Gem Path    /home/oneamtu/.gem/ruby/2.4.0:/home/oneamtu/.asdf/installs/ruby/2.4.2/lib/ruby/gems/2.4.0
  User Path   /home/oneamtu/.gem/ruby/2.4.0
  Bin Dir     /home/oneamtu/.asdf/installs/ruby/2.4.2/bin
Tools
  Git         2.7.4
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2017-10-31
Git SHA           10f20fa33
Released Version  true

Bundler settings

path
  Set for your local app (/home/oneamtu/test/test/.bundle/config): "vendor/bundle"
disable_shared_gems
  Set for your local app (/home/oneamtu/test/test/.bundle/config): true

Gemfile

Gemfile

source 'https://rubygems.org'

gem 'rake', '~> 11.0'

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    rake (11.3.0)

I've been meaning to dig into bundler a bit and patch it, but I figured this info would be good if anyone else is experiencing this issue since it includes a workaround. I'm happy to own it, let me know if you have any particular directions I should go in.

[colby-swandale]

Thanks for the detailed report! A PR fixing this issue would be very welcome 👍

[atomical]

What would be the work around for bundle exec executable_name ? Where the executable name is only available when doing bundle exec?

[segiddins]

I wonder if this change in 1.13 is responsible

[shayonj]

@segiddins I am unable to follow that link, but, I believe it started with this change introducing the unexpectedness with Signals

https://github.com/bundler/bundler/blob/93e11fc7e3b9d067affe637c0e4c264d5b82bcd1/lib/bundler/cli/exec.rb#L63

from:

93e11fc#diff-fcaa44c40b054ace4b039a38e3ffe82fR63

Before I proceed, IMPORTANT NOTE: This bug only exists when you vendor in your gems (which requires Bundler::CLI::Exec to call the executable). If you were to try the steps from description without vendoring, this bug does not exists.

Explanation:

Like I mentioned, this only happens when you vendor in.

• The issue starts with bundler swallowing HUP signal here

https://github.com/bundler/bundler/blob/060e2f953c8f08dd7bed9711863e6341eebe5fe2/lib/bundler/cli/exec.rb#L74

• Now, because rake is vendored in (per the description) (and is not executed by ruby_executable_hooks), the same is loaded here

https://github.com/bundler/bundler/blob/060e2f953c8f08dd7bed9711863e6341eebe5fe2/lib/bundler/cli/exec.rb#L75

So, when a HUP signal is sent, it gets rescued/swalloed well before nohup gets a chance to work with it.

This entire phenomenon, can by tried by this simple script:

test.rb (

IGNORE_SIGNALS = %w[SEGV BUS ILL FPE VTALRM KILL STOP]
signals = Signal.list.keys - IGNORE_SIGNALS
signals.each do |s|
  trap(s, "DEFAULT")
end

sleep 100

Now, try using nohup on the file:

nohup ruby test.rb 
appending output to nohup.out

...

# send a `kill -hup $PID` on the process

You will see the nohup died.

Next, add HUP to IGNORE_SIGNALS and try again. You will see that nohup no longer dies and its not being trapped/swalled anymore.

I will try to cut a PR soon.

[shayonj]

I am curious, why does bundler needs to swallow any Signal at all? The changes that were introduced to fix the issue: #4568, (which I think is the probable cause for this current issue) seems a little unclear to me, so I am curious :). I am also unable to reproduce the original issue from #4568

bundle _1.12.0_ exec puma
Puma starting in single mode...
* Version 3.4.0 (ruby 2.3.0-p0), codename: Owl Bowl Brawl
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:9292
Use Ctrl-C to stop
^C- Gracefully stopping, waiting for requests to finish
=== puma shutdown: 2017-12-13 21:33:27 -0800 ===
- Goodbye!
echo $?
0

I am trying to determine whether it makes sense to add HUP to

https://github.com/bundler/bundler/blob/060e2f953c8f08dd7bed9711863e6341eebe5fe2/lib/bundler/cli/exec.rb#L9

Or just not have signals get swallowed by bundler at all, but leave it to to executables to manage it. For example, puma does a lot of custom stuff to handle signals

[indirect]

@shayonj the change was to allow signals to be passed to children, but perhaps it is no longer needed? Can you also test this after bundle config disable_exec_load true?

@segiddins any thoughts here? we didn't end up reverting the exec/load stuff, did we?

[shayonj]

@indirect ah! I believe that makes sense, thanks!

And, yep, I can no longer replicate the original issue after setting disable_exec_load to true

Looks like disable_exec_load is in play here

https://github.com/bundler/bundler/blob/060e2f953c8f08dd7bed9711863e6341eebe5fe2/lib/bundler/cli/exec.rb#L27

[indirect]

Hmm. So maybe this means we can stop reserving signals in some versions of Ruby? Or maybe Puma changed the way that it traps signals, and our work-around is no longer needed.

[segiddins]

Given that we're just setting the signal handlers all back to their default, I wonder if Ruby's default handler for SIGHUP is weird?

[shayonj]

hm, perhaps I am not following 😅

I believe along with trapping signal handlers, we are also swallowing them (i.e we are not re-raising them), which is why HUP is never caught by nohup (in this specific example). Here is a simple example:

test.rb

IGNORE_SIGNALS = %w[SEGV BUS ILL FPE VTALRM KILL STOP]
signals = Signal.list.keys - IGNORE_SIGNALS
signals.each do |s|
  trap(s, "DEFAULT")
end

Kernel.exec("ruby ./test1.rb")
# Kernel.load("./test1.rb")

test1.rb

Signal.trap("HUP") do
  puts "--> Received hup from test.rb"
end
sleep 100

Now, run this via

nohup ruby test.rb

And kill via

nohup ruby test.rb

You will see that, nohup died/return, because we are trapping/swallowing HUP. If you, add HUP to IGNORE_SIGNALS, the behavior is now expected and will also see that the signal is being received by executed file (test1.rb), in the logs.

---

In short, I am wondering if we need to trap any signals at all. I still can't replicate the puma specific behavior from the past issue:D.

[segiddins]

But all we’re doing is trapping with the default handler, which in theory should be no different than not calling trap at all

[shayonj]

[UPDATE]

Yeah, though not entirely. Setting DEFAULT means, setting the SIG_DFL handler on the process.

I went digging and it turns out, Signal.trap(:HUP, "DEFAULT") overrides the signal set by nohup which was SIG_IGN / IGNORE (ruby source code here). Hence, when nohup runs the bundler code via execvp, bundler here overrides the signal handler setup by nohup.

That said, a better solution might be: bundler restores a signal back to previous status if a present one, otherwise sets the DEFAULT signal?

(I discussed the same issue here https://bugs.ruby-lang.org/issues/14196)

---

[UPDATE 2]

Here is the dump of the signals that Bundler is overriding from Exec (and ran via nohup):

SIGEXIT: Before Handler: (), Current Handler: (DEFAULT)
SIGHUP: Before Handler: (IGNORE), Current Handler: (DEFAULT)
SIGINT: Before Handler: (#<Proc:0x00007f8e100534f8@/Users/<>/.rvm/gems/ruby-2.4.2/gems/bundler-1.16.0/exe/bundle:5>), Current Handler: (DEFAULT)
SIGQUIT: Before Handler: (DEFAULT), Current Handler: (DEFAULT)
SIGTRAP: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGABRT: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGIOT: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGEMT: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGSYS: Before Handler: (), Current Handler: (DEFAULT)
SIGPIPE: Before Handler: (), Current Handler: (DEFAULT)
SIGALRM: Before Handler: (DEFAULT), Current Handler: (DEFAULT)
SIGTERM: Before Handler: (DEFAULT), Current Handler: (DEFAULT)
SIGURG: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGTSTP: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGCONT: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGCHLD: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGCLD: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGTTIN: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGTTOU: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGIO: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGXCPU: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGXFSZ: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGPROF: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGWINCH: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)
SIGUSR1: Before Handler: (DEFAULT), Current Handler: (DEFAULT)
SIGUSR2: Before Handler: (DEFAULT), Current Handler: (DEFAULT)
SIGINFO: Before Handler: (SYSTEM_DEFAULT), Current Handler: (DEFAULT)

[segiddins]

Thanks. Looks like we might need to re-consider setting everything to default, and manually keep a list of all the signals that bundler itself is trapping and just restore those handlers

[shayonj]

yes.

and manually keep a list of all the signals that bundler itself is trapping and just restore those handlers

Do we know if there are some specific signals that Bundler wish to trap?

If I am getting this right, anything thats not in IGNORE_SIGNALS is being being trapped by bundler. Hence, these are the ones we should be trapping and restoring? But, if we wish to restore, then should we even trap them in the first place ? Or, by extension, what are the benefits of trapping
any signal at all? 😅 🙈 (don't mean to be PITA, apologies, if so)

[segiddins]

Or, by extension, what are the benefits of trapping any signal at all?

We've historically trapped signals such as SIGINT to ensure that ^C properly kills bundler during downloads

[colby-swandale]

@segiddins can this be closed?

[lkosewsk]

@colby-swandale do you have a (rough, even) timeline of when you'll be promoting this tag and and submitting 1.16.2 to rubygems.org?

Just seeing whether I'm OK to wait (this problem has bit me recently) a few, say, days, or whether I should tool my own workaround by building bundler from the 1-16-stable branch.