http_client does not gracefully handle remotely-closed keepalive connections (Linux/ASIO)

[Olipro]

From my discussion with Robert, the issue manifests itself like so:

1. Have a keepalive socket as an http_client to some server.
2. Server sends a FIN ACK as it decides the connection has been idle for too long which puts the socket into CLOSE_WAIT
3. something comes along and makes a new request with the http_client instance, headers are sent on the socket which is in CLOSE_WAIT so they are still received.
4. Since the server can't actually respond, the connection gets an RST from the server end and cpprestsdk finds out about it when it moves onto trying to write the body, consequently throwing an exception to the user.

Firstly, we should try other connections if we end up in this state, however, retrying all connections will result in one unlucky caller spending time acting as a garbage collector for all the dead sockets, which could be numerous.

One option for dealing with this is to retry the pool of existing connections in a limited fashion such that anyone who encounters 2 failures will give up and (try to) establish a new connection - so they'll still have cleaned up more than they created.

[vadz]

This is a really bad bug and I don't understand why more people are not affected by it as it seems like it always happens whenever you use the same http_client for multiple but not immediately consecutive requests. E.g. it is reliably triggered in my GUI applications if the user enters a wrong password in the login form and then corrects it and tries to log in again and connection is closed by the server in the meanwhile. And as the default timeout for Apache is just 5s, it means that it happens all the time.

Moreover, asio_connection_pool timeout is currently hardcoded as 30s in the code and there doesn't seem to be any way to prevent the connection from being reused (or did I miss a way to close it from the client code?), so I don't even see how to work around it... At the very least, I think the hardcoded timeout should be changed to be 5s or less to make this problem less likely to happen in practice.

But ideally I think cpprestsdk should parse the Keep-Alive header and avoid reusing the connection if it's older than the timeout specified in it. Of course, it could still happen that the connection times out even then, so the code should still handle this more robustly and I'll submit my updated patch from #297 together with another fix required to actually make it work to do this.

[vadz]

P.S. Another improvement would be to skip the resolving step for the reused-but-already-closed-by-server connections: we could store the list of endpoints retrieved during the initial resolution and just reconnect to them when recreating a new connection.

[zowers]

timeout of 30 seconds is also too big for Amazon S3, 5 seconds works fine