libp2p_kad::handler No older substream waiting to be reused. Dropping new substream.

[lynic]

I try to setup a kad network, I enabled kad,relay,ping,identify,gossipsub in my code, and setup one as server with public access.

Then I spawn 10~20 clients to connect to this server, client enables kad,ping,identify,gossipsub, and use kad find_closest_peers to update DHT.

After some time, the server logs shows many message like below.

[2022-09-28T16:32:45Z WARN  libp2p_kad::handler] New inbound substream to PeerId("12D3KooWLMtdcMzvCQH5NseLaBbBqETrRLQuQCVSjsGsb4qzutD8") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.

Would like to understand what does this message mean? and how to remediate it?

[thomaseizinger]

To protect against DoS attacks, the kademlia implementation limits the amount of inbound streams that a single remote peer can open on one connection. This limit is currently set to 32 and not configurable.

I don't know kademlia in depth (@mxinden does) but 32 streams PER connection sounds like a fair amount to me. It essentially means that a single peer is currently keeping us busy with 32 concurrent requests. Once a request is serviced, the same substream can be re-used again. That is the reuse the message is talking about.

Essentially, this warning means that your node is under high load and (temporarily) refusing to service even more requests.

[dignifiedquire]

FYI these happen a lot

[dignifiedquire]

I am not entirely clear how I can do sth on my end to prevent this, I have a lot of queries to run 😅

[thomaseizinger]

With #2878, these will go away and we will just not accept any more inbound streams from the remote. Depending on the muxer, this may lead to timeouts on the remote in opening new streams.

[mxinden]

With #2878, these will go away and we will just not accept any more inbound streams from the remote. Depending on the muxer, this may lead to timeouts on the remote in opening new streams.

I am not convinced a global limit is the way to go. See also #2878 (comment).

I am not entirely clear how I can do sth on my end to prevent this, I have a lot of queries to run sweat_smile

In my eyes the best long-term fix is to add a maximum number of concurrent streams to the Kademlia specification. I do think it is reasonable to assume that a node can not make use of a parallelization factor of say 16 to a remote peer. @dignifiedquire would you be willing to drive this?

[thomaseizinger]

With #2878, these will go away and we will just not accept any more inbound streams from the remote. Depending on the muxer, this may lead to timeouts on the remote in opening new streams.

I am not convinced a global limit is the way to go. See also #2878 (comment).

The latest version of #2878 allows the ConnectionHandler to decide on the max number of inbound streams. Once hit, the muxer on this connection will no longer poll for inbound streams.

For Kademlia, this is now set to the 32 that was enforced previously as part of the event handler in the ConnectionHandler. With "this will go away", I was referring to the log message. That one is removed because the ConnectionHandler is now guaranteed to not be given more substreams than what it reported in max_inbound_streams :)

[shamil-gadelshin]

I have a similar issue with just two peers: #3048