Memory leak in HikariCP connection pool

[chuwy]

I'm testing my RESTful service built on top of http4s and doobie and encountering cryptic (as always) OOM.

It suspect it happens with any load, but below results are were produced with stress-testing via apache benchmark (ab -n 1000000 -c 800 from the same host).

Memory dump, couple of minutes after launch:

The connectionEC is the only object in memory that looks slightly (yet) suspicious. Working queue has 778 elements.

I dump memory every 5 minutes and see more or less the same picture. connectionEC is always on the first line, but never significantly above 3M. Until ~1hr later I get the OOM and heap produced with -XX:+HeapDumpOnOutOfMemoryError shows following picture:

Basically it filled my heap with... something doobieish inside something IOish (sorry, not very faimiliar with cats-effect internals).

I think this particular setup was poorly configured: small 1CPU/2GB machine, 16 threads in connection pool, 5 maximumPoolSize (which looks too big), fairly heavy load and heavy DB response (~400Kb over the network), but other benchmarks have shown similar results.

I'm running this doobie 0.7.0 (same problem was with 0.6.0), http4s-blaze 0.20.1 NIO1. Here's the pool initialization code: https://github.com/snowplow-incubator/iglu-server/blob/release/0.6.0/src/main/scala/com/snowplowanalytics/iglu/server/storage/Storage.scala#L95-L116 (in case I miss somethiing very obvious).

This looks suspciciously similar to #824 / http4s/http4s-okhttp-client#3, except that I'm not using http4s client.

[chuwy]

Just in case - the message I'm getting along with OOM:

[iglu-hikaricp-pool housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - iglu-hikaricp-pool - Thread starvation or clock leap detected (housekeeper delta=1m238ms398µs104ns).
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /iglu/java_pid6.hprof ...
Heap dump file created [388489171 bytes in 10.352 secs]
Aborting due to java.lang.OutOfMemoryError: Java heap space
...
#  Internal Error (debug.cpp:321), pid=6, tid=72
#  fatal error: OutOfMemory encountered: Java heap space
...
Exception in thread "pool-3-thread-1"

[chuwy]

It seems the problem was in http4s Timeout middleware. The problem went away after we removed it.

I'm leaving the issue open as I don't know whether the issue is in http4s (unlikely as Timeout implementation is so primitive that I cannot believe it can be wrong) in doobie (maybe something is not cancellable in fact?) or cats-effect.

[chuwy]

I think this PR into cats-effect might have fixed it: typelevel/cats-effect#568. Will try to reproduce with both versions.

[darekmydlarz]

@chuwy any updates on that?

[mdedetrich]

Was about to make an issue but I also have a leak, not sure if its entirely related though https://github.com/mdedetrich/task-doobie-memory-leak

[chuwy]

@darek1024, sorry for delay - not yet. My problem went away once I removed Timeout middleware in http4s code. I still didn't have a chance to write minimal project to reproduce and check if typelevel/cats-effect#568 really fixed it.

[chuwy]

@mdedetrich based on this (monix/monix#912) Task also doesn't yet use setRemoveOnCancelPolicy, so it certainly can be related.

[mdedetrich]

The leak happens even when you don't use Task

[chuwy]

Which implementation do you use? I assume that only IO from cats-effect-2.0.0 and ZIO don't suffer from it.

[mdedetrich]

In the repository which I linked to, I just used IO (well I ran IO.unsafeToFuture) without any task and the leak was still there.

[jatcwang]

Closing this as it's probably not an issue anymore. Please reopen if that's not true