Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(POOLER-184) Pool manager retry and exit on failure #398

Merged
merged 2 commits into from
Sep 3, 2020
Merged

(POOLER-184) Pool manager retry and exit on failure #398

merged 2 commits into from
Sep 3, 2020

Conversation

sbeaulie
Copy link
Contributor

@sbeaulie sbeaulie commented Sep 2, 2020

Adding a reconnect retry for redis, which by default would retry 10 times,
for a total wait time of ~80 seconds

@sbeaulie sbeaulie requested a review from a team as a code owner September 2, 2020 16:13
@sbeaulie
Copy link
Contributor Author

sbeaulie commented Sep 2, 2020

I tested it locally by pointing to localhost but without a redis running, and it returned the error after 1min20 seconds.

So the retry works, but I cannot reproduce the initial ticket findings that it would keep running after hitting the stack trace.

root@a3389b84408e:/# VMPOOLER_DEBUG=true docker-entrypoint.sh manager
[2020-09-02 16:07:36] starting vmpooler
[2020-09-02 16:08:57] Cannot connect to the redis server: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
warning: thread "Ruby-0-Thread-1: /usr/local/bundle/gems/vmpooler-0.14.5/bin/vmpooler:43" terminated with exception (report_on_exception is true):
Redis::CannotConnectError: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
  establish_connection at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:362
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:107
        with_reconnect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:308
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:106
      ensure_connected at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:381
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:233
               logging at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:320
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:232
                  call at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:126
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:848
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
       mon_synchronize at /opt/jruby/lib/ruby/stdlib/monitor.rb:235
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:844
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1567
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:29
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:28
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:21
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1565
                <main> at /usr/local/bundle/gems/vmpooler-0.14.5/bin/vmpooler:44
Redis::CannotConnectError: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
  establish_connection at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:362
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:107
        with_reconnect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:308
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:106
      ensure_connected at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:381
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:233
               logging at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:320
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:232
                  call at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:126
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:848
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
       mon_synchronize at /opt/jruby/lib/ruby/stdlib/monitor.rb:235
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:844
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1567
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:29
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:28
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:21
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1565
                <main> at /usr/local/bundle/gems/vmpooler-0.14.5/bin/vmpooler:44
root@a3389b84408e:/# echo $?
1

(POOLER-184) Pool manager retry and exit on failure
a35d66c 
Adding a reconnect retry for redis, which by default would retry 10 times,
for a total wait time of ~80 seconds
mattkirby
mattkirby reviewed Sep 2, 2020
View reviewed changes
Copy link
Contributor

@mattkirby mattkirby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm +1, but had a comment about preferring we move the default for 'redis_connection_attempts' to vmpooler.rb.

bin/vmpooler Outdated Show resolved Hide resolved
@sbeaulie sbeaulie marked this pull request as draft September 2, 2020 22:17
@sbeaulie sbeaulie marked this pull request as ready for review September 3, 2020 13:47
add the default in the standard location, also adding
1c53625 
support for an ENV var. Update the example yaml file with the new config
@genebean genebean merged commit debd0bd into master Sep 3, 2020
@genebean genebean deleted the POOLER-184 branch September 3, 2020 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mattkirby mattkirby mattkirby approved these changes

@genebean genebean genebean approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sbeaulie @genebean @mattkirby