gRPC streaming keepAlive doesn't work with docker swarm

[npuichigo]

In a swarm setup using overlay networks, idle connections between grpc-gateway and backend service will end up in a broken state after 15 minutes. (moby/moby#31208)

So I tried the following two ways. Firstly, I set net.ipv4.tcp_keepalive_time to less than 900 seconds, to make sure the TCP connection between grpc-gateway and IPVS doesn't expire. Secondly, I configured grpc.WithKeepaliveParams to set the ping time interval to 30 seconds. However, neither method works.

Is there anyway to configure grpc-gateway to support docker swarm?

[johanbrandhorst]

How did you configure the KeepaliveParams? The gRPC-Gateway uses a gRPC client to talk to the backend so there is not much we can do in terms of changing the gateway. The parameters must be specified to the Dial call. If that's not working, I'm afraid you will have to continue debugging your network setup.

I'm going to close this as it is a docker swarm issue with a gRPC client workaround. Please let me know if you want me to clarify anything.

[npuichigo]

Thank you for your reply. I configure the KeepaliveParams like this, so I'm sure the parameters are passed to the Dial call.

ctx := context.Background()
        ctx, cancel := context.WithCancel(ctx)
        defer cancel()

        mux := runtime.NewServeMux()
        opts := []grpc.DialOption{
                grpc.WithInsecure(),
                grpc.WithKeepaliveParams(keepalive.ClientParameters{
                        Time:                30 * time.Second,
                        Timeout:             20 * time.Second,
                        PermitWithoutStream: true,
                }),
        }
        err := gw.RegisterMyHandlerFromEndpoint(ctx, mux, *echoEndpoint, opts)

[johanbrandhorst]

That looks correct, but unless there's a bug with the gateway not using the parameters in the call to gRPC.Dial, I don't see what we could do to help you. Can you investigate whether the keep alive is being used in the gRPC library?

[npuichigo]

Okay, I'll check it. Thanks.