-
Notifications
You must be signed in to change notification settings - Fork 3k
Rapid Recovery
HikariCP is designed to recover from database restart or network partition events. However, the pool can only do so for connections currently under its control. Once a connection is "out" of the pool, borrowed by the application, HikariCP is largely powerless to ensure rapid recovery.
The reason that HikariCP is powerless to recover connections that are out of the pool is due to unacknowledged TCP traffic. TCP is a synchronous communication scheme that requires "handshaking" from both sides of the connection as packets are exchanged (SYN and ACK packets).
When TCP communication is abruptly interrupted, the client or server can be left awaiting the acknowledgement of a packet that will never come. The connection is therefore "stuck", until an operating system level TCP timeout occurs. This can be as long as several hours, depending on the operating system TCP stack tuning.
In order to avoid this condition, it is imperative that the application configures the driver-level TCP socket timeout. Each driver differs in how this timeout is set, but nearly all drivers support it.
HikariCP recommends that the driver-level socket timeout be set to (at least) 2-3x the longest running SQL transaction, or 30 seconds, whichever is longer. However, your own recovery time targets should determine the appropriate timeout for your application.
See the specific database sections below for some common configurations.
Another impediment to recovery can be DNS caching performed by the JVM, or the operating system itself. For example, if the hostname of the database remains constant, but the IP address may change during recovery, a DNS cache can interfere with recovery by returning a stale address.
See this page, whose content is applicable across all databases.
The PostgreSQL database driver supports socket time outs via the socketTimeout property. See this page for relevant information.
The MySQL database driver (Connector/J) supports socket time outs via the socketTimeout property. See this page for relevant information.
The Microsoft SQL Server driver supports socket time outs via the socketTimeout property. See this page for relevant information.