Describe queues in client and server connections.

tempesta_fw/connection.h

@@ -102,6 +102,39 @@ typedef struct {
 
 #define TFW_CONN_TYPE(c)	((c)->proto.type)
 
+/*
+ * Queues in client and server connections provide support for correct
+ * handlng of requests and responses.
+ *
+ * Incoming requests are put on client connection's @seq_queue in the
+ * order they come in. When responses to these requests come, they're
+ * sent back to client in exactly the same order the requests came in.
+ * @seq_queue is contended by threads that process requests, as well
+ * as by threads that process responses. In the latter case that may
+ * not lead to sending a response. Thus a separate @ret_qlock is used
+ * for sending responses to decrease the time @seq_qlock is taken for.
+ *
+ * Unless serviced from cache, each request is forwarded to a server
+ * over specific server connection. It's put on server connection's
+ * @fwd_queue, and also on @nip_queue if it's non-idempotent. Requests
+ * must be forwarded in the same order they're put on @fwd_queue, so
+ * it must be done under the queue lock. Otherwise pairing of requests
+ * to responses may get broken. When a response comes then the first
+ * request is taken out of @fwd_queue, and that's the paired request.
+ * There're two types of requests in @fwd_queue: those that were sent
+ * out, and those that were not sent out yet. @msg_sent points at the
+ * latest request that was sent out. That is helpful when repairing
+ * a server connection that had gone bad.
+ *
+ * A request is in @seq_queue until it's deleted, and may also be in
+ * @fwd_queue if it's forwarded to a server. @nip_queue supplements
+ * @fwd_queue and may be considered as part of @fwd_queue for this
+ * description. A response is never put on any queue. Instead, it's
+ * attached to a paired request as @req->resp. A request is always
+ * processed in the context of just one queue at any given moment.
+ * That way NO locking hierarchy is involved. Please see the code.
+ */
+
 /*
  * These are specific properties that are relevant to client connections.
  *

tempesta_fw/http.c

@@ -690,6 +690,14 @@ tfw_http_req_fwd_unsent(TfwSrvConn *srv_conn, struct list_head *equeue)
  * removed when the holding non-idempotent request is followed by
  * another request from the same client. Effectively, that re-enables
  * pipelining. See RFC 7230 6.3.2.
+ *
+ * Requests must be forwarded in the same order they are put in the
+ * queue, and so it must be done under the queue lock, otherwise
+ * pairing of requests with responses may get broken. Take a simple
+ * scenario. CPU-1 locks the queue, adds a request to it, unlocks
+ * the queue. CPU-2 does the same after CPU-1 (the queue was locked).
+ * After that CPU2 and CPU2 are fully concurrent. If CPU2 happens to
+ * proceed first with forwarding, then pairing gets broken.
  */
 static void
 tfw_http_req_fwd(TfwSrvConn *srv_conn, TfwHttpReq *req)