why grpc-java uds is slower than dns？

[zhangchao1171]

Question:
We did a test to compare the performance between uds and dns. According to what said on the Internet, uds should be faster because it lacks many network layer protocols. However, the actual tested found that uds is at most the same as dns,in most cases The performance will be worse. Can anyone help explain ? The following are test cases and test results.

ENV:
jdk: jdk-17.0.10
grpc: 1.61.1
test machine: centos 7.9 & 64 cpu cores & 251 GB RAM

code detail
uds server core code:

 String UDS_FILE="/tmp/grpc_uds.socket";
 server = NettyServerBuilder.forAddress(new DomainSocketAddress(UDS_FILE))
                    .bossEventLoopGroup(new EpollEventLoopGroup())
                    .workerEventLoopGroup(new EpollEventLoopGroup())
                    .channelType(EpollServerDomainSocketChannel.class)
                    .maxInboundMessageSize(4*1024*1024)
                    .addService(BrSpringApplication.getBean(GreeterServiceImple.class))
                    .build().start();
  blockUtilShutDown();

uds client core code:

 String socketPath="/tmp/grpc_uds.socket";
 channel = NettyChannelBuilder.forAddress(new DomainSocketAddress(socketPath))
                    .eventLoopGroup(new EpollEventLoopGroup())
                    .channelType(EpollDomainSocketChannel.class)
                    .maxInboundMessageSize(4 * 1024 * 1024)
                    .usePlaintext().build();
GreeterGrpc.GreeterBlockingStub blockStub = GreeterGrpc.newBlockingStub(channel);
HelloRequest request = HelloRequest.newBuilder().setName(paramJson.toJSONString()).build();
        HelloReply response = getBlockStub().sayHello(request);

GreeterServiceImpl.java

@Override
    public void sayHello(HelloRequest request, StreamObserver<HelloReply> responseObserver) {
        long startTime = System.currentTimeMillis();
        logger.warn("request:{}", request.getName());
        try {
            if (Context.current().isCancelled()) {
                logger.warn("deadline cancalled,deadline:{}", Context.current().getDeadline());
                responseObserver.onError(new Exception("Cancelled by client"));
                return;
            }
            JSONObject msgRequest= JSONObject.parseObject(request.getName());
            int num=msgRequest.getIntValue("factor");
            String msg=msgRequest.getString("message");
            StringBuilder messageBack = new StringBuilder("rep-");
            for (int i = 0; i < num; i++) {
                messageBack.append(msg).append("|");
            }
//            logger.warn("response:{}", messageBack);
            HelloReply response = HelloReply.newBuilder().setMessage(messageBack + ",ms:" + (System.currentTimeMillis() - startTime)).build();
            responseObserver.onNext(response);
            responseObserver.onCompleted();

        } catch (BrException e) {
            logger.error("BrException,code:{},message:{}", e.getCode(), e.getMessage());
            HelloReply response = HelloReply.newBuilder().setMessage(e.getMessage()).build();
            responseObserver.onNext(response);
            responseObserver.onCompleted();
        } catch (Exception ex) {
            logger.error("sys error", ex);
            HelloReply response=HelloReply.newBuilder().setMessage(ex.getMessage()).build();
            responseObserver.onNext(response);
            responseObserver.onCompleted();
        }
    }

dns sever core code:

String port="80";
server = NettyServerBuilder.forPort(port)
                    .maxInboundMessageSize(4*1024*1024)
                    .addService(BrSpringApplication.getBean(GreeterServiceImple.class))
                    .build().start();
blockUtilShutDown();

dns client core code:

String ip="localhost:80"
channel = ManagedChannelBuilder.forTarget(ip).usePlaintext().build();
GreeterGrpc.GreeterBlockingStub blockStub = GreeterGrpc.newBlockingStub(channel);
HelloRequest request = HelloRequest.newBuilder().setName(paramJson.toJSONString()).build();
        HelloReply response=getBlockStub().sayHello(request);

test result:

client message size	response message size	thread	totalnum	type	num	total cost(ms)	qps
12B	60B	50	5000000	dns	1	127359ms	5535.790199357721
12B	60B	50	5000000	dns	2	126051ms	5593.233722858208
12B	60B	50	5000000	dns	3	128828ms	5472.666687366101
12B	60B	50	5000000	uds	1	133212ms	5292.561510974987
12B	60B	50	5000000	uds	2	128315ms	5494.546265050852
12B	60B	50	5000000	uds	3	127177ms	5543.712337922738
12B	120B	50	5000000	dns	1	125919ms	5599.097070338868
12B	120B	50	5000000	dns	2	122034ms	5777.346509989019
12B	120B	50	5000000	dns	3	120439ms	5853.857172510566
12B	120B	50	5000000	uds	1	123538ms	5707.010830675582
12B	120B	50	5000000	uds	2	126988ms	5551.963209122122
12B	120B	50	5000000	uds	3	126450ms	5575.584847765916
12B	6kb	50	5000000	dns	1	123176ms	5723.783074624926
12B	6kb	50	5000000	dns	2	127242ms	5540.880401125414
12B	6kb	50	5000000	dns	3	121556ms	5800.06502352825
12B	6kb	50	5000000	uds	1	131939ms	5343.626251525327
12B	6kb	50	5000000	uds	2	133399ms	5285.1423473939085
12B	6kb	50	5000000	uds	3	130091ms	5419.534817935138
5kb	5kb	50	5000000	dns	1	133404ms	5284.9442595424425
5kb	5kb	50	5000000	dns	2	130532ms	5401.225017620201
5kb	5kb	50	5000000	dns	3	130145ms	5417.286134695916
5kb	5kb	50	5000000	uds	1	140809ms	5007.0144948121215
5kb	5kb	50	5000000	uds	2	140685ms	5011.427685965099
5kb	5kb	50	5000000	uds	3	140992ms	5000.515660463005

I expected grpc uds to be faster than dns , but it's not. all test cases are on the localhost machine , and i put the uds path to the ssd disk , but fount it's performance as same as hdd. I alse changed server and client memory from 2GB to 6 GB ,but it doesn't any improvement

[He-Pin]

How about test with newer version of netty grpc and netty

[zhangchao1171]

tested on grpc lastest version 1.69.0 and netty version 4.1.110.Final，it's same as 1.61.1 , grpc uds still slower than dns . and comparing the results，it seams uds on the 1.69.0 is slower than 1.61.0 . here are test result on grpc version on 1.69.0:

client message size	response message size	thead	totalnum	type	num	total cost(ms)	qps
5kb	5kb	50	5000000	dns	1	133404ms	5284.9442595424425
5kb	5kb	50	5000000	dns	2	133904ms	5265.210180427769
5kb	5kb	50	5000000	dns	3	129208ms	5456.571605473346
5kb	5kb	50	5000000	uds	1	146765ms	4803.820420399959
5kb	5kb	50	5000000	uds	2	143096ms	4926.990999049589
5kb	5kb	50	5000000	uds	3	144900ms	4865.650131124914

[He-Pin]

Strange, how about test with a smaller and larger message sizes，we are using uds in grpc too, but have not tested it, another thing is, how about cache the response data instead of generating them every time

[zhangchao1171]

The behavior does not change regardless of the message size, and the server logic is the same in both methods. As for caching data on the server side, the above result is like this，send request message is equal to the response.

[He-Pin]

Do you test it on mac or Linux box？ bare metal or not？

[zhangchao1171]

all test are on linux , use prod machine

[ejona86]

The answer is going to be of various forms of "don't know" and "that would be interesting for you to figure out." A good way to start that would be to use perf and make a flame graph, as that can include syscall overhead.

One thing to note, it seems your benchmark is closed-loop and you have fewer concurrent RPCs than cores. So your benchmark may not be a throughput benchmark as much as a latency benchmark. Unless you can show it takes about the same amount of time when you vary the number of concurrent RPCs, then you should keep in mind it may be a latency benchmark.

The answer could turn out to be very easy. For example, TCP might have bigger buffer sizes in the kernel. Or the Linux scheduler just so happens to do a better job with TCP.

As a shot-in-the-dark, you might try setting flowControlWindow() to various MBs (1,4,8 MB) in NettyChannelBuilder/NettyServerBuilder and see what happens.

[zhangchao1171]

yeah , It's a latency benchmark， and refre to this https://github.com/grpc/grpc-java/issues/11763 （Bottom of page）. I did a test on netty abount uds vs dns , but it performance not as grpc ，it's uds is faster than dns . grpc is based on netty communication ，that's why I'm confused .