Upgrading from 8.12 to 8.13.1 produced 500 error due to Redis monitoring throwing NameError

[mbsmartee]

^^ Provide a general summary of the issue in the title above. ^^

After upgrading from version 8.12 to version 8.13.1 of newrelic_rpm we began getting a 500 error due to the Redis monitoring component of the New Relic agent not finding "client".

Description

newrelic_rpm breaking Rails app on Heroku using Redis because of NameError

This shows up in the Rails Console when trying to access the Rails cache in an app using the Redis cache.

NameError (undefined local variable or method client' for RedisClient::Middlewares:Module`

Expected Behavior

It is expected that it will working normally and not break the app.

Troubleshooting or NR Diag results

Provide any other relevant log data.
TIP: Scrub logs and diagnostic information for sensitive information

Steps to Reproduce

Using newrelic_rpm 8.13.1, open a Rails console where the New Relic agent is enabled and the 8.13.1 version of the newrelic_rpm is being used.
In the Rails console, try to access the keys in the Rails cache with Rails.cache.redis.keys
What should happen is that a listing of keys returns. What happens instead is that there is a NameError.
This can be fixed by disabling the agent or reverting to the 8.12 versions.

Your Environment

Ruby 3.12, newrelic_rpm 8.13.1

Additional context

N/A

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://issues.newrelic.com/browse/NEWRELIC-5499

[kaylareopelle]

Thanks for bringing this to our attention, @mbsmartee

What versions of the following gems are you running:

• rails
• redis

Are you using Sidekiq or another gem with a dependency on Redis?

Does this error also occur if you try to perform an operation directly on your Redis client?

Example:

redis = Redis.new 
redis.get("foo")

[mbsmartee]

Thanks for bringing this to our attention, @mbsmartee

What versions of the following gems are you running:

• rails

7.0.4

• redis

Redis 6 I believe, it is the latest Heroku Redis.

Are you using Sidekiq or another gem with a dependency on Redis?

Redis Objects.

Does this error also occur if you try to perform an operation directly on your Redis client?

Example:

redis = Redis.new 
redis.get("foo")

It happens in the running Rails app and can be reproduced in the Rails console. I'm able to work around the issue by setting NEW_RELIC_AGENT_ENABLED to false. I'm using the Redis Rails cache setting that has been present in Rails since 5.2 I believe.

With the agent enabled I'm clearing connecting:

irb(main):001:0> Rails.cache.redis
=> #<Redis client v5.0.5 for redis://[REDACTED]/0>

But if I try to grab keys it errors out:

irb(main):001:0> 
irb(main):002:0> Rails.cache.redis
=> #<Redis client v5.0.5 for redis://[REDACTED]/0>
irb(main):003:0> Rails.cache.redis.keys
/app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:33:in `call_pipelined_with_tracing': undefined local variable or method `client' for RedisClient::Middlewares:Module (NameError)                   
                                                                                
      with_tracing(operation, statement: statement, database: client.config.db) { yield }
                                                              ^^^^^^            
irb(main):004:0> 

[mbsmartee]

If I disable New Relic, I'm able to grab the keys, no problem:

Loading development environment (Rails 7.0.4)
irb(main):001:0> Rails.cache.redis
=> #<Redis client v5.0.5 for redis://REDACTED/0>
irb(main):002:0* Rails.cache.redis.keys
irb(main):002:0> Rails.cache.redis.keys
=> 
["program:hecm:home_value_limit",                                               
 "lender:m39fow6072:proprietary_home_value_limit",                              
 "lender:r05m66opsr:proprietary_home_value_limit",                              
 "lender:SmartFi:proprietary_home_value_limit",                                 
 "lender:vwh5uffv8g:proprietary_home_value_limit",                              
 "lender:5ivdvtl3ng:proprietary_home_value_limit"]                              
irb(main):003:0> 

[kaylareopelle]

Thanks for the additional context, @mbsmartee. We're still trying to reproduce this locally.

Since the error seems to be related to Redis instrumentation, you can disable the instrumentation while we look for a solution and continue to run New Relic for all other parts of your app.

To disable the instrumentation, update your newrelic.yml file to include the following under the common: &default_settings key:

instrumentation.redis: disabled

If you're using environment variables, you can pass this instead:

NEW_RELIC_INSTRUMENTATION_REDIS=disabled

[kaylareopelle]

@mbsmartee, would you be willing to share how you're configuring Redis in your environment/development.rb file and any initializers related to Redis?

For example, the config.cache_store keys you're using, how many URLs you're passing, etc. would be helpful. We don't need any personal/proprietary data though.

We're having trouble reproducing the issue and want to get as close to your environment as we can.

[kaylareopelle]

Also, could you provide a longer stack trace? We'd love to understand more about how the call to call_pipelined_with_tracing is getting lit up by running Rails.cache.redis.keys

[patatepartie]

@kaylareopelle If it helps, I'm getting the same error as mbsmartee, but in the context of action cable.
This happened after upgrading newrelic_rpm from 8.12.0 to 8.13.1.
Here's the full stack trace.

#<Thread:0x00007fce075ba158 /app/vendor/bundle/ruby/3.1.0/gems/actioncable-7.0.4/lib/action_cable/subscription_adapter/redis.rb:150 run> terminated with exception (report_on_exception is true):                                                   
/app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:33:in `call_pipelined_with_tracing': undefined local variable or method `client' for RedisClient::Middlewares:Module (NameError)
                                                                                                                                                                                                                                                    
      with_tracing(operation, statement: statement, database: client.config.db) { yield }                                                                                                                                                           
                                                              ^^^^^^                                                                                                                                                                                
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/middleware.rb:12:in `call_pipelined'                                                                                                      
    from /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.9.0/lib/redis_client.rb:676:in `connect'                                                                                                                                                 
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:25:in `block in connect'                                                                                                       
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:10:in `block in connect_with_tracing'                                                                                  
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:48:in `block in with_tracing'                                                                                          
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/tracer.rb:355:in `capture_segment_error'                                                                                                                        
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:48:in `with_tracing'                                                                                                   
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:10:in `connect_with_tracing'                                                                                           
    from /app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.13.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:25:in `connect'                                                                                                                
    from /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection'                                                                                                                                          
    from /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected'                                                                                                                                        
    from /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.9.0/lib/redis_client.rb:631:in `ensure_connected'                                                                                                                                        
    from /app/vendor/bundle/ruby/3.1.0/gems/redis-5.0.5/lib/redis/client.rb:104:in `disable_reconnection'                                                                                                                                           
    from /app/vendor/bundle/ruby/3.1.0/gems/redis-5.0.5/lib/redis.rb:79:in `without_reconnect'                                                                                                                                                      
    from /app/vendor/bundle/ruby/3.1.0/gems/actioncable-7.0.4/lib/action_cable/subscription_adapter/redis.rb:80:in `listen'                                                                                                                         
    from /app/vendor/bundle/ruby/3.1.0/gems/actioncable-7.0.4/lib/action_cable/subscription_adapter/redis.rb:154:in `block in ensure_listener_running'                                                                                              
Exiting                                                                                                                                                                                                                                             
** [NewRelic][2022-11-27 23:37:35 +0000 ba5ec3404b13 (7)] INFO : Starting Agent shutdown                                                                                                                                                            

We're using redis 5.0.5 (with a redis server 3.2.X) and rails 7.0.4.

We only have this configuration related to redis:

production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/7" } %>

Disabling redis instrumentation, as you suggested, correctly works around the issue.

[kaylareopelle]

Thanks for sharing your experience, @patatepartie. Glad to hear turning off the instrumentation helps workaround the problem. We'll try to reproduce the bug using Action Cable and get back to you with more information.

[fallwith]

@mbsmartee @patatepartie Hi there. We've been able to reproduce the problem when using redis-client gem v0.9.0 as shown as active in @patatepartie's stack trace. When we update that gem to v0.11.2, the problem goes away. I believe v8.13.0 of the New Relic Ruby agent was developed against v0.11.x of that gem, so it makes sense that we didn't notice compatibility issues with older versions.

When convenient, would you please try upgrading to redis-client v0.11.2 for your applications and try testing with v8.13.1 of the New Relic Ruby agent with Redis instrumentation enabled?

If you're able to confirm that the problem is fixed with redis-client v0.11.2, we can update the New Relic Ruby agent to either require that as a minimum version or be backwards compatible with older redis-client versions, depending on how tricky that might be.

[fallwith]

@mbsmartee @patatepartie We have prepped #1673 as a compatibility fix between redis-client versions < 0.11 and newrelic_rpm v8.13+.

In addition to upgrading your apps to use redis-client v0.11+, another option is to test with this new holiday_spiced_sider branch of the New Relic Ruby agent.

[patatepartie]

I tried independently upgrading redis-client and using the newrelic_rpm branch: both work (as in I don't see the stack trace anymore when enabling redis instrumentation).

Thanks @fallwith!

[fallwith]

Hi @mbsmartee @patatepartie. The fix has now been released in v8.14.0 of the newrelic_rpm gem.

• v8.14.0 should work just fine with older versions of the RedisClient gem
• We have added additional Redis instrumentation related error handling just in case any other undiscovered compatibility issues still remain.

Both of you have been officially recognized in our CHANGELOG. Thanks again for bringing this issue to our attention!