Relax connection termination policy in routing driver #424
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously routing driver terminated all connections towards a particular address when one of active connections had a network error. Connections were also terminated when new routing table did not contain some address that was present in the previous routing table. Such behaviour might be problematic because it results in terminated queries. Network errors might have been temporary but always resulted in termination of active queries. This commit makes driver keep existing connections and terminate only idle ones on network error. Address will still be excluded from the routing table and subsequent queries will not be pointed on it. Corresponding connection pool is transferred into a "passive" state where it does not serve new connections and disposes returned connections immediately. Pool is transferred back to "active" state during next rediscovery only if corresponding address is present in the new routing table. Old behaviour still remains and can be activated using `-DpurgeOnError=true` system property. It exists as a fallback only for exceptional cases and will most likely be silently removed in future.
* `ConnectionPools` which is a map from addresses to connection pools * `RoutingTable` which contains addresses for routers, writers, and readers For a routing driver, once an address is removed from the current `RoutingTable`, the address is no longer accessable. a.k.a. no new connection will be created in the corresponding connection pool. When updating routing table, we also need to signal the addresses in `ConnectionPools` to be active if they are newly added into the current routing table or passive if they have already been removed from the routing table. For the pools connected to addresses that have been removed, when a connection is free, the connection should be terminated rather than reused. When there is no connection in the pool, the pool could be safely removed from `ConnectionPools`. So the logic that need to be changed: * When a new `RoutingTable` is available, compute `added_addr = distinct_addr_in(new_routingTable) - distinct_addr_in(pre_routingTable)` `removed_addr = distinct_addr_in(pre_routingTable) - distinct_addr_in(new_routingTable)` * Mark all addresses in set `added_addr` in `ConnectionPools` to be `active` connection pools * Mark all addresses in set `removed_addr` in `ConnectionPools` to be `passive` connection pools * Remove `passive` connection pools if no connection is `inUse` (all connections are idle) * When returning a connection to a `passive` connection pool, terminate the connection directly, [and remove the connection pool if no connections is `InUse`]
zhenlineo
reviewed
Nov 2, 2017
| { | ||
| expirationTimeout = cluster.expirationTimestamp(); | ||
| // todo: what if server is added as reader and removed as writer? we should not treat it as removed |
There was a problem hiding this comment.
To answer your todo, there is no need to know if the status of an address in RoutingTable, we only care about the whole distinct addresses in the RoutingTable. The ConnectionPools only need to know if an address is active (can reuse connections) or passive (can be terminated once no inUse connections in the pool)
See more explanation in lutovich#1
Simplify the logic of compute added and removed addresses in routing table
That tried to count inserted nodes in the cluster using read transaction and no bookmarks.
Use transaction function to write and receive bookmark. Previously `Session#run()` was used and it does not produce a bookmark.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously routing driver terminated all connections towards a particular address when one of active connections had a network error. Connections were also terminated when new routing table did not contain some address that was present in the previous routing table. Such behaviour might be problematic because it results in terminated queries. Network errors might have been temporary but always resulted in termination of active queries.
This commit makes driver keep existing connections and terminate only idle ones on network error. Address will still be excluded from the routing table and subsequent queries will not be pointed on it. Corresponding connection pool is transferred into a "passive" state where it does not serve new connections and disposes returned connections immediately. Pool is transferred back to "active" state during next rediscovery only if corresponding address is present in the new routing table.
Old behaviour still remains and can be activated using
-DpurgeOnError=truesystem property. It exists as a fallback only for exceptional cases and will most likely be silently removed in future.