Memory leak with Doobie + Task #934
Comments
|
I ran both example projects for half an hour and both of them seem to have slowly increasing old gen. Maybe the reason it is visible in the graph is due to Doobie GCing 5x more often? Quill Doobie Quill Doobie |
|
Hmm, I can't really replicate your results, what OS are you running on? I will also try and run the tests for a longer period of time. |
|
I am running Linux 5.1.15 and using GraalVM |
|
Hmm I am not using GraalVM, instead I was using OpenJDK 11 (also on linux). When I get home I will run |
|
This is what I get running Java 11 with these java options: Quill Doobie |
|
Okay so I verified a couple of things. Yes its true that the Quill version has memory increasing slowly over time, but this is just the OldGen and it actually gets cleaned up when you approach the current heap size where as the doobie version will continuously increase over time (while bumping the current heap) which is what we are experiencing in production. I just modified the repository to increase the number of transactions being made (so its quicker to reproduce) and I also lowered the initial heap size to 64M so we don't have to wait so long. These are the results For quill, as you can see the memory does increase over time but once you approach the initial heap size the OldGen GC sweep kicks in and releases the memory On the other hand when we run the doobie version when it approaches the initial heap size it will just continuously increase the heap (which indicates a leak)
|
|
For further evidence, I also ran the tests setting The associated graph is here As you can see its allocated memory until it hits the max heap size at which point you start getting memory related errors. The same with quill
|
|
Okay I get the same results with the changes. It's because the connect executor has an unbound queue and the number of produced database operations exceeds the number that can be processed. I seem to be able to process ~3,900 operations a second but there are
In production I run with a bounded queue and handle the |
|
Okay so question is we are having this leak in production and we have no way near that amount of transactions per second (we typically get 200 transactions per second max, lets say 500 max to be generous). I am going to tweak the connect executor to see if I can reproduce but I think its something more. Furthermore for whatever reason, Quill doesn't have this problem at all. It can handle this load and it also doesn't slow down |
|
When I run the quill example I get a |
|
Hmm I don't get this, I even posted the graph which was running for 20 minutes, let me run once more |
|
Strange. There are over 11,000 monix-io threads listed in VisualVM when it crashes. |
|
Latest run, also running the suite on a friends laptop and its working fine. In any case, this is somewhat concerning because it appears that what you are saying is that Doobie can't process transactions as fast as they are incoming (talking about our production server where we are experiencing the problem, not this reproduction repo which is deliberately running a lot of transactions to speed up the leak). Using a bounded queue isn't really an option because we are then dropping transactions. |
|
Okay so I ran val transactEC: ExecutionContext = ExecutionContext.fromExecutor(
new ThreadPoolExecutor(
100,
Integer.MAX_VALUE,
60,
TimeUnit.SECONDS,
new LinkedBlockingQueue[Runnable](1000)
)
)And I am still getting the problem,
Also here is a screenshot of it
Not sure what is happening with Quill, maybe its specific to OpenJDK? In any case there are still problems with Doobie even when using a bounded queue (or should I use more appropriate settings?) EDIT: Forgot the second parameter, let me re-run |
|
Okay so I reran with the correct parameters, i.e. val transactEC: ExecutionContext = ExecutionContext.fromExecutor(
new ThreadPoolExecutor(
100,
100,
60,
TimeUnit.SECONDS,
new LinkedBlockingQueue[Runnable](1000)
)
)And still the same result
|
|
Yeah I have no idea. If I run with I get this
I have removed references to
|
|
Okay thanks, I will try and replicate this tomorrow. Maybe its the timeout of 60 seconds that is causing the queue to fill. This still begs the question though, having an unbounded queue to me seems like a bandaid because essentially this is saying that Doobie can't process transactions as fast as its receiving them. This is understandable for the repo which has the isolated leak but for production we honestly do not have that much load and if I understand correctly I definitely do not want to be dropping transactions on a bounded queue because its filling up (and also I am curious as to why Quill is not exhibiting this behavior, at least when you slow down the transaction frequency in the production repo) |
|
Sorry for taking a while, but I tried replicating your results. The good: I sometimes can replicate your results and interestingly it doesn't leak here. I am attaching screenshots of the past 3 runs which I did The second run is the one which is ideal however it doesn't happen materialistically. These are the settings I am using for the transact executor import java.util.concurrent.RejectedExecutionHandler
import java.util.concurrent.ThreadPoolExecutor
class MyRejectedExecutionHandler extends RejectedExecutionHandler {
override def rejectedExecution(worker: Runnable, executor: ThreadPoolExecutor): Unit = {
logger.error(s"Worker ${worker.toString} is rejected")
}
}
val connectEC: ExecutionContext =
ExecutionContext.fromExecutor(Executors.newFixedThreadPool(32))
val executor = new ThreadPoolExecutor(
100,
100,
60,
TimeUnit.SECONDS,
new LinkedBlockingQueue[Runnable](1000),
new MyRejectedExecutionHandler
) |
Note that this may be related to #913
I finally managed to diagnose a memory leak that is happening with Doobie +
monix.eval.Task/cats.effect.IO. Note that this only occurs with Doobie.The repo reproducing the leak can be found here https://github.com/mdedetrich/task-doobie-memory-leak
The text was updated successfully, but these errors were encountered: