"read: connection reset by peer" error under *moderate* load

[zarmin]

I see these error messages in the console under moderate load (more than 10 write / sec)

2017-11-06 11:03:23.845682 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55152: read: connection reset by peer
2017-11-06 11:03:23.865135 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55164: read: connection reset by peer
2017-11-06 11:03:23.957875 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55218: read: connection reset by peer
2017-11-06 11:03:23.982248 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55230: read: connection reset by peer
2017-11-06 11:03:24.051271 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55272: read: connection reset by peer
2017-11-06 11:03:24.071915 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55284: read: connection reset by peer
2017-11-06 11:03:24.081377 I | etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 10.156.0.4:2379->10.156.0.4:55290: read: connection reset by peer

The values are successfully written and the cluster's healthy, but I worry about these errors.
The error is only present when using version 3.2.x, with 2.2.5 I have no errors.
The network is stable.

Version:

etcd Version: 3.2.0
Git SHA: 66722b1
Go Version: go1.8.3
Go OS/Arch: linux/amd64

Cluster setup shell script:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.156.0.4
HOST_2=10.156.0.5
HOST_3=10.156.0.3

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}

CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

./etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

(https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md)

[nvtkaszpir]

Check your clients which connects to etcd. We've seen this when client tried to fetch too much data via grpc ( usually over 4MB).

[xiang90]

how can i reproduce this issue?

[xiang90]

@zarmin kindly ping. if there is no response, i will close this issue due to low activity.

[SaranBalaji90]

@zarmin may be this could be related to grpc/grpc-go#1062