redis-cache blocks the eventloop

[bartm-dvb]

Describe the bug

The Redis cache implementation seems to block the eventloop. It causes the HTTP throughput to be very poor, we see that the worker_pool_queue_delay_seconds_max metric keeps getting higher and higher, it was at 60s before we turned off the load test. It also seems to cause a very high number of open files on the Linux system.

We verified this by switching to caffeine cache, the throughput increased by a factor 3. The number of open files dropped and worker_pool_queue_delay_seconds_max stays low.

Expected behavior

The Redis cache implementation does not block the eventloop.

Actual behavior

The Redis cache implementation blocks the eventloop.

How to Reproduce?

1. Create a sample application with Resteasy Reactive and Redis cache
2. Create a simple endpoint and a method annotated with @CacheResult
3. Run a load test with > 100req/s
4. The console should now log thread blocked errors

Output of uname -a or ver

Linux 4.18.0-477.27.1.el8_8.x86_64

Output of java -version

openjdk version "17.0.5" 2022-10-18 LTS OpenJDK Runtime Environment (Red_Hat-17.0.5.0.8-2.el8_6) (build 17.0.5+8-LTS) OpenJDK 64-Bit Server VM (Red_Hat-17.0.5.0.8-2.el8_6) (build 17.0.5+8-LTS, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.4.1

Build tool (ie. output of mvnw --version or gradlew --version)

3.9.2

Additional information

No response

[quarkus-bot]

/cc @cescoffier (redis), @gsmet (redis), @gwenneg (cache), @machi1990 (redis)

[geoand]

Without an example application that exhibits this problem, it will be very hard to track down the problem

[bartm-dvb]

Without an example application that exhibits this problem, it will be very hard to track down the problem

I will add an example application a bit later

[bartm-dvb]

Sample application:

https://github.com/bartm-dvb/quarkus-redis-bug

It seems to happen only when the Redis client is configured in cluster mode. Running a load test on this will cause it to run out of open files. I will run tests tomorrow with more open files allow to see if I will run in to thread-blocked warnings

[cescoffier]

\CC @Ladicek

[Ladicek]

It also seems to cause a very high number of open files on the Linux system.

I'm fairly sure this is caused by vert-x3/vertx-redis-client#365 (connection pooling basically doesn't work at all), which indeed only manifests when using clustered Redis client.

This should be fixed in the next Vert.x release, which should be this week. Once that is integrated in Quarkus, we can see whether there's more to this issue.

[bartm-dvb]

Thanks, I see the new Vert.x version was released yesterday. I will rerun the test when the new Quarkus version is released so we can close this issue

[cescoffier]

Vert.x 4.4.6 seems to fix the issue, at least improve a lot.
I still see Redis error, but I believe it's a docker issue more than a Quarkus/Vert.x issue.

[bartm-dvb]

That's good news! Thanks for taking the effort to run the test.