[ipfs/go-bitswap] Connection State Tracking

[Stebalien]

Currently, we handle enqueuing broadcast wants synchronously when we process "connect" events. Instead, we should do the minimal amount of work possible when processing events, handing broadcast wants in the background.

General Design

There are two parts to this design:

1. Connectivity tracking: reference count connections to keep track of whether or not we're connected to a peer.
2. Availability tracking: records whether or not the peer is available as a bitswap peer.

States

• Disconnected (has no connections)
• Connected (has at least one connection)
• Available

Transitions

• disconnected -> connected
  • When:
    • We receive the first connection
  • Effect:
    • Start reference counting.
• • -> disconnected
  • When:
    • We drop the last connection.
  • Effect:
    • Forget the peer.
• connected -> available
  • When:
    • We receive a "connect" event, check the peerstore, and see that the peer speaks the bitswap protocol.
    • We receive an identify event, see that the peer is connected and that the peer speaks the bitswap protocol.
    • We receive a bitswap message from the peer.
  • Effect:
    • Tell the PeerManager that the peer is available.
• available -> connected
  • When:
    • The peer is marked unresponsive because we fail to send them a message.
  • Effect:
    • Drop the message queue for the peer.
    • Don't drop the ledger or anything else.

Ideas

I'm having trouble completely fleshing out this design so I'm going to write some other ideas:

1. I'd have one goroutine for processing events, transitioning from disconnected, to connected, to available.
2. I'd have one goroutine syncing the peer's state with the peer manager.

The important part is that if any subsystem signals that the peer should transition from available to connected, the peer must transition to connected, even if it ends up transitioning back to available.

I have some ideas on how to do this, but I'll have to try to write them up later.