Graceful shutdown support

[terencefan]

The target of this PR is to ensure all incoming and outgoing packages will be processed before a scheduled shutdown.

UPDATE 11/13

1. Add test cases for HubDispatcher and MultiEndpointContainer.

UPDATE 11/12

1. Lift the ShutdownAsync method up to the HubDispatcher level.

UPDATE 11/06

1. Lift the ShutdownAsync method up to the MultiEndpointServiceConnectionContainer level.
2. Add OfflineAsync method to IServiceConnectionContainer.

UPDATE 11/05

1. Move CloseAsync method to ServiceConnectionContainer level.

UPDATE 11/01

1. Replace CloseConnectionMessage with PingMessage.
2. Wait on TCS instead of connection outgoing tasks.
3. Prevent rebalanced (ondemand) connection from starting in the shutdown process.

UPDATE 10/31

Add test cases for corner cases.

UPDATE 10/29

Add test cases for common scenarios.

UPDATE 10/28

Reimplement CloseAsync process by using TaskCompletionSource which makes the code more readable.

UPDATE 10/24

After discussed thoroughly with @KKhurin and @vicancy, we decided NOT do anything with the messages during the shutdown process in this scenario. We just remove server connections from the routing table to prevent new client connections to be assigned to this server and wait until either client or server to close client connections by themselves.

Since the server-side may not have the ability to notify the client to reconnect, maybe we will introduce a mechanism between server and ASRS to let the server has this ability in the future.

And there will also a TIMEOUT (by default it's 30s).
We simply drop every connection after reaching the time limit.

The CloseAsync (for each server connection) contains the following 6 steps:

1. Try acquiring closeConnectionLock to ensure at most 1 closeAsync in progress.
2. Send a CloseConnectionMessage with a FIN to ASRS
   • Once ASRS receives a FIN, it will remove the server connection immediately from its route table (to prevent new client connections from assigning to this server connection)
3. Wait until ASRS has sent back a CloseConnectionMessage with a FINACK.
4. Wait until all client connections have stopped.
   • Our ASRS will handle the client connections and find a proper time to close them, or in some cases we just let our customer close them by themselves. So we intentionally do nothing on the SDK side during this step.
5. Complete incoming pipe
6. Wait until the connection (StartAsync) has stopped safely.

[davidfowl]

Spoke to @terencefan offline:

• There should be no changes to the route API. Instead there should be a TimeSpan that can be set on ServiceOptions that let the user control how long to wait for shutdown.
• By default we should Inject the IApplicationLifetime and call ShutdownAsync on all connections ourselves.
• There should be no locks, CloseAsync on ServiceConnectionBase should Write the Fin and wait on task returned from ProcessIncomingAsync with the timeout configured in ServiceOptions.

[KKhurin]

Shouldn't we drain the outgoing pipe as the first step of the whole shutdown process or at least after we dispose the hub and close client connections but before we send the fin?
The reason is the messages queued in the pipe (and the implicit queue inside the semaphore lock) could require acks and it feels wrong to even have a possibility to send any messages after sending fin.

[terencefan]

@KKhurin Actually we shouldn't block the outgoing pipe even after sending the FIN.

The FIN here is to notify the ASRS Runtime that the app server wants to close its connection, not already closed. There are still messages in both incoming and outgoing pipes and need to be processed at the moment.

Once our app server receives a FINACK, we could make sure no more incoming messages will be sent from ASRS Runtime, and after we close the incoming pipe gracefully, we could drain the outgoing pipe and close the connection.

And once our ASRS Runtime receives a FIN, it will block the upcoming messages from the client/browser, not the server. All pending messages (even the messages triggered by the unprocessed messages in the incoming pipe) in the outgoing pipe will be sent in my proposal.

A typical disconnecting process would contain 4 steps, we reduce it into 2 (FIN + FINACK) just because we are not expecting a graceful shutdown on the client-side.

[vicancy]

move to local static readonly field, serialized?

[davidfowl]

Why didn't this get fixed in the PR?

[davidfowl]

Here as well.

[davidfowl]

OfflineAsync isn't a verb. MarkOfflineAsync? MakeOfflineAsync?