Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configurable warm up for remote datacenter connections #1198

Merged
merged 2 commits into from
Jun 20, 2019
Merged

Add configurable warm up for remote datacenter connections #1198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db3b61c
Add configurable warm up for remote datacenter connections
cgtz Jun 20, 2019
7044b18
Address comments
cgtz Jun 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions ambry-api/src/main/java/com.github.ambry/config/RouterConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,23 @@ public class RouterConfig {

/**
* The percentage of {@link RouterConfig#routerScalingUnitMaxConnectionsPerPortSsl} or
* {@link RouterConfig#routerScalingUnitMaxConnectionsPerPortPlainText} to warm up in the startup.
* {@link RouterConfig#routerScalingUnitMaxConnectionsPerPortPlainText} to warm up for data nodes in the local
* datacenter during startup.
* {@link RouterConfig#routerConnectionsWarmUpTimeoutMs} may need to be adjusted.
*/
@Config("router.connections.warm.up.percentage.per.port")
@Config("router.connections.local.dc.warm.up.percentage")
@Default("25")
public final int routerConnectionsWarmUpPercentagePerPort;
public final int routerConnectionsLocalDcWarmUpPercentage;

/**
* The percentage of {@link RouterConfig#routerScalingUnitMaxConnectionsPerPortSsl} or
* {@link RouterConfig#routerScalingUnitMaxConnectionsPerPortPlainText} to warm up for data nodes in remote
* datacenters during startup.
* {@link RouterConfig#routerConnectionsWarmUpTimeoutMs} may need to be adjusted.
*/
@Config("router.connections.remote.dc.warm.up.percentage")
@Default("0")
public final int routerConnectionsRemoteDcWarmUpPercentage;
/**
* The max time allowed to establish connections to local DC in the startup
*/
Expand Down Expand Up @@ -300,8 +310,10 @@ public RouterConfig(VerifiableProperties verifiableProperties) {
verifiableProperties.getIntInRange("router.scaling.unit.max.connections.per.port.plain.text", 5, 1, 100);
routerScalingUnitMaxConnectionsPerPortSsl =
verifiableProperties.getIntInRange("router.scaling.unit.max.connections.per.port.ssl", 2, 1, 100);
routerConnectionsWarmUpPercentagePerPort =
verifiableProperties.getIntInRange("router.connections.warm.up.percentage.per.port", 25, 0, 100);
routerConnectionsLocalDcWarmUpPercentage =
verifiableProperties.getIntInRange("router.connections.local.dc.warm.up.percentage", 25, 0, 100);
routerConnectionsRemoteDcWarmUpPercentage =
verifiableProperties.getIntInRange("router.connections.remote.dc.warm.up.percentage", 0, 0, 100);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the default value be 25 here? We are setting default in the declaration decorator to 25.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah thanks for catching that. The default should be 0 since I feel that warming up cross DC connections only applies to specific ambry use cases, but I will fix the decorator.

routerConnectionsWarmUpTimeoutMs =
verifiableProperties.getIntInRange("router.connections.warm.up.timeout.ms", 5000, 0, Integer.MAX_VALUE);
routerConnectionCheckoutTimeoutMs =
Expand Down
22 changes: 15 additions & 7 deletions ambry-router/src/main/java/com.github.ambry.router/NonBlockingRouter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadLocalRandom;
Expand Down Expand Up @@ -553,14 +554,21 @@ private class OperationController implements Runnable {
*/
OperationController(String suffix, String defaultPartitionClass, AccountService accountService) throws IOException {
networkClient = networkClientFactory.getNetworkClient();
// Warm up connections to dataNodes in local DC.
// Warm up connections to dataNodes in local and remote DCs.
List<ResponseInfo> responseInfos = new ArrayList<>();
networkClient.warmUpConnections(clusterMap.getDataNodeIds()
.stream()
.filter(dataNodeId -> clusterMap.getDatacenterName(clusterMap.getLocalDatacenterId())
.equals(dataNodeId.getDatacenterName()))
.collect(Collectors.toList()), routerConfig.routerConnectionsWarmUpPercentagePerPort,
routerConfig.routerConnectionsWarmUpTimeoutMs, responseInfos);

String localDatacenter = clusterMap.getDatacenterName(clusterMap.getLocalDatacenterId());
Map<Boolean, List<DataNodeId>> localAndRemoteNodes = clusterMap.getDataNodeIds()
.stream()
.collect(Collectors.partitioningBy(dataNodeId -> localDatacenter.equals(dataNodeId.getDatacenterName())));
logger.info("Warming up local datacenter connections to {} nodes", localAndRemoteNodes.get(true).size());
networkClient.warmUpConnections(localAndRemoteNodes.get(true),
routerConfig.routerConnectionsLocalDcWarmUpPercentage, routerConfig.routerConnectionsWarmUpTimeoutMs,
responseInfos);
logger.info("Warming up remote datacenter connections to {} nodes", localAndRemoteNodes.get(false).size());
networkClient.warmUpConnections(localAndRemoteNodes.get(false),
routerConfig.routerConnectionsRemoteDcWarmUpPercentage, routerConfig.routerConnectionsWarmUpTimeoutMs,
responseInfos);
// Update ResponseHandler immediately if connections lost to certain nodes.
for (ResponseInfo responseInfo : responseInfos) {
if (responseInfo.getRequestInfo() == null) {
Expand Down
10 changes: 4 additions & 6 deletions ambry-router/src/test/java/com.github.ambry.router/NonBlockingRouterTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,8 @@ private Properties getNonBlockingRouterProperties(String routerDataCenter) {
properties.setProperty("router.delete.success.target", Integer.toString(DELETE_SUCCESS_TARGET));
properties.setProperty("router.connection.checkout.timeout.ms", Integer.toString(CHECKOUT_TIMEOUT_MS));
properties.setProperty("router.request.timeout.ms", Integer.toString(REQUEST_TIMEOUT_MS));
properties.setProperty("router.connections.local.dc.warm.up.percentage", Integer.toString(67));
properties.setProperty("router.connections.remote.dc.warm.up.percentage", Integer.toString(34));
properties.setProperty("clustermap.cluster.name", "test");
properties.setProperty("clustermap.datacenter.name", "dc1");
properties.setProperty("clustermap.host.name", "localhost");
Expand Down Expand Up @@ -859,12 +861,8 @@ public void testWarmUpConnectionFailureHandling() throws IOException {
MockServerLayout mockServerLayout = new MockServerLayout(mockClusterMap);
mockSelectorState.set(MockSelectorState.FailConnectionInitiationOnPoll);
setRouter(props, mockServerLayout, new LoggingNotificationSystem());
List<DataNodeId> localNodes = mockClusterMap.getDataNodes()
.stream()
.filter(node -> node.getDatacenterName().equals("DC3"))
.collect(Collectors.toList());
for (DataNodeId node : localNodes) {
assertTrue("Local node should be marked as timed out by ResponseHandler.", ((MockDataNodeId) node).isTimedOut());
for (DataNodeId node : mockClusterMap.getDataNodes()) {
assertTrue("Node should be marked as timed out by ResponseHandler.", ((MockDataNodeId) node).isTimedOut());
}
router.close();
mockSelectorState.set(MockSelectorState.Good);
Expand Down