Skip to content

Commit

Permalink
Update exchange protocol documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
arhimondr committed Jul 18, 2024
1 parent 0521740 commit 96c0361
Showing 1 changed file with 14 additions and 9 deletions.
23 changes: 14 additions & 9 deletions presto-docs/src/main/sphinx/develop/worker-protocol.rst
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,16 @@ results or by a downstream worker to fetch intermediate results from the
upstream worker.

* A ``GET`` on ``{taskId}/results/{bufferId}/{token}`` returns the next batch
of results from the specified output buffer.
of results from the specified output buffer. Acknowledges the receipt of the
previous batch.
* A ``GET`` on ``{taskId}/results/{bufferId}/{token}/acknowledge`` acknowledges
the receipt of the results and allows the worker to delete them.
* A ``DELETE`` on ``{taskId}/results/{bufferId}`` deletes all results from the
specified output buffer in case of an error.
* Optionally, a ``HEAD`` request can be made to ``{taskId}/results/{bufferId}``
to retrieve any non-data page sequence related headers. Use this to check if
the buffer is finished, or to see how much data is buffered without fetching
the data.
the data. Acknowledges the receipt of the previous batch of results.

Coordinator and workers fetch results in chunks. They specify the maximum size
in bytes for the chunk using ``X-Presto-Max-Size`` HTTP header. Each chunk is
Expand All @@ -76,20 +77,24 @@ response includes:
* The sequence number to use to acknowledge the receipt of the chunk and to
request the next chunk as ``X-Presto-Page-End-Sequence-Id`` HTTP header,
* An indication that there are no more results as ``X-Presto-Buffer-Complete``
HTTP header with the value of "true".
HTTP header with the value of ``true``.
* The remaining buffered bytes in the output buffer as ``X-Presto-Buffer-Remaining-Bytes``
HTTP header. This should return a comma separated list of the size in bytes of
the pages that can be returned in the next request. This can be used as a hint
to upstream tasks to optimize data exchange.

The body of the response contains a list of pages in :doc:`SerializedPage wire format <serialized-page>`.

After receiving the first chunk of results, the client sends an ack via a GET
on ``{taskId}/results/{bufferId}/{token}/acknowledge`` with the token set to
the value of the ``X-Presto-Page-End-Sequence-Id`` HTTP header received earlier
along with the results. Then, the client uses that sequence number to request
the next chunk of results. The client keeps fetching results until it receives
``X-Presto-Buffer-Complete`` HTTP header with the value of "true".
After receiving the first chunk of results, the client uses the
``X-Presto-Page-End-Sequence-Id`` sequence number to request the next chunk of results.
Requesting the next chunk automatically acknowledges reception of the previous chunk.
The client keeps fetching results until it receives ``X-Presto-Buffer-Complete`` HTTP header
with the value of ``true``.

When a client decides not to fetch the next chunk of data right away, it sends an
explicit ack using a GET on ``{taskId}/results/{bufferId}/{token}/acknowledge``. The client
sets the token to the value of the ``X-Presto-Page-End-Sequence-Id`` header
received earlier.

If the worker times out populating a response, or the task has already failed
or been aborted, the worker will return empty results. The client can attempt
Expand Down

0 comments on commit 96c0361

Please sign in to comment.