Add support for content range requests when getting blobs #537

rchincha · 2024-05-29T17:24:52Z

OCI artifacts support has landed in various OCI specs v1.1.0 which allows for arbitrary artifact types, small and large.

Large artifacts (even existing container images) pose a particular challenge that:

it takes too long to download
it takes too long to unpack

This PR begins to address 1. above.

The client can initiate a HEAD request to get the size and later multiple GET range requests to download a blob in parallel.

Fixes #354

sudo-bmitch · 2024-05-29T17:37:09Z

Does this need to be included in the dist-spec, vs delegating to the HTTP 1.1 RFC: https://datatracker.ietf.org/doc/html/rfc7233#section-3.1

rchincha · 2024-05-29T17:38:09Z

#536

rchincha · 2024-05-29T17:39:11Z

Does this need to be included in the dist-spec, vs delegating to the HTTP 1.1 RFC: https://datatracker.ietf.org/doc/html/rfc7233#section-3.1

If it is not included here, registry operators have no guidance on whether to implement this or not. We are explicitly stating that we are doing/requiring this for perf reasons, etc.

samuelkarp

If we recommend behavior here, that should align with both the Notational Conventions of this specification and with the behavior described in RFC 9110.

samuelkarp · 2024-05-29T19:57:20Z

spec.md

@@ -190,6 +190,11 @@ If present, the value of this header MUST be a digest matching that of the respo

 If the blob is not found in the registry, the response code MUST be `404 Not Found`.

+A GET request may also include a `Range` request header to download part of a


Suggested change

A GET request may also include a `Range` request header to download part of a

A GET request MAY also include a `Range` request header to download part of a

samuelkarp · 2024-05-29T19:57:47Z

spec.md

+blob. The response code can either be `216 (Partial Content)` or `416 (Range
+Not Satisfiable)` in case of an invalid range. If the registry doesn't support


Suggested change

blob. The response code can either be `216 (Partial Content)` or `416 (Range

Not Satisfiable)` in case of an invalid range. If the registry doesn't support

blob. The response code SHOULD either be `216 (Partial Content)` or `416 (Range

Not Satisfiable)` in case of an invalid range. If the registry doesn't support

samuelkarp · 2024-05-29T19:59:41Z

spec.md

@@ -190,6 +190,11 @@ If present, the value of this header MUST be a digest matching that of the respo

 If the blob is not found in the registry, the response code MUST be `404 Not Found`.

+A GET request may also include a `Range` request header to download part of a
+blob. The response code can either be `216 (Partial Content)` or `416 (Range


Suggested change

blob. The response code can either be `216 (Partial Content)` or `416 (Range

blob in accordance with [RFC 9110](https://datatracker.ietf.org/doc/html/rfc9110#name-range-requests). The response code can either be `216 (Partial Content)` or `416 (Range

samuelkarp · 2024-05-29T20:02:48Z

spec.md

+Not Satisfiable)` in case of an invalid range. If the registry doesn't support
+range requests, it can respond with `Accept-Ranges: none`.


Suggested change

Not Satisfiable)` in case of an invalid range. If the registry doesn't support

range requests, it can respond with `Accept-Ranges: none`.

Not Satisfiable)` in case of an invalid range. A registry MAY ignore the `Range` header.

An older registry unaware of this change may ignore the header, but a newer registry may want to send a response back to discourage clients from sending the Range request.

rchincha · 2024-05-29T21:09:52Z

We can only download as fast as the pipe allows of course, but maybe this helps with head-of-line blocking when transferring a large file by getting many smaller pieces in parallel - how many pieces? etc is tbd math. Also, perhaps more importantly, opens possibility of getting pieces from multiple registries in parallel. Furthermore, download failures more likely with large files are resumable.

rchincha · 2024-05-30T18:50:38Z

Updated as per comments in the OCI meeting 05/30/2024
The only point of contention then is MAY vs SHOULD

Candidate for dist-spec v1.1.1?

sudo-bmitch

In addition to the reword, it looks like a separate commit got pulled in by accident. Please fix with a rebase on main.

spec.md

OCI artifacts support has landed in various OCI specs v1.1.0 which allows for arbitrary artifact types, small and large. Large artifacts (even existing container images) pose a particular challenge that: 1) it takes too long to download 2) it takes too long to unpack This PR begins to address 1) above. The client can initiate a HEAD request to get the size and later multiple GET range requests to download a blob in parallel. Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

sudo-bmitch

LGTM

rchincha · 2024-05-31T18:46:02Z

Note that another reading of the spec found this:
https://github.com/opencontainers/distribution-spec/blob/main/spec.md#resumable-pull
So we are just making this explicit with this PR.

sudo-bmitch · 2024-05-31T21:40:27Z

I've linked with #354.

rchincha · 2024-06-08T05:39:54Z

Just preliminary testing ...

zot localhost over http with 10GiB artifact blob

Single-threaded client start to finish: 1min 32secs

{"level":"info","module":"http","component":"session","clientIP":"127.0.0.1:59934","method":"GET","path":"/v2/artifact/blobs/sha256:732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d","statusCode":200,"latency":"1m32s","bodySize":10737418240,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["Go-http-client/1.1"]},"goroutine":167,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-06-08T03:34:08.916403767Z","message":"HTTP API"}

Multi-theaded client (still HTTP/1.1) but with range based download with 2GiB chunks: ~33 secs
HTTP/1.1 doesn't do true pipelining, however at least, the server backend becomes aware of all requests so can start working on I/Os in parallel. golang HTTP/2.0 needs TLS as a default (without doing transport gymnastics) - will test this next.

{"level":"info","module":"http","component":"session","clientIP":"127.0.0.1:52134","method":"GET","path":"/v2/artifact/blobs/sha256:732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d","statusCode":206,"latency":"18s","bodySize":2147483649,"headers":{"Range":["bytes=6442450944-8589934592"],"User-Agent":["Go-http-client/1.1"]},"goroutine":184,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-06-08T04:11:01.590093476Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"127.0.0.1:52176","method":"GET","path":"/v2/artifact/blobs/sha256:732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d","statusCode":206,"latency":"18s","bodySize":2147483648,"headers":{"Range":["bytes=8589934592-10737418240"],"User-Agent":["Go-http-client/1.1"]},"goroutine":195,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-06-08T04:11:01.831432295Z","message":"HTTP API"}

Multi-theaded client (but now HTTP/2.0) but with range based download with 2GiB chunks: ~58 secs (surprisingly slower)

{"level":"info","module":"http","component":"session","clientIP":"127.0.0.1:56086","method":"GET","path":"/v2/artifact/blobs/sha256:732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d","statusCode":206,"latency":"25s","bodySize":2147483649,"headers":{"Range":["bytes=0-2147483648"],"User-Agent":["Go-http-client/2.0"]},"goroutine":726149,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-06-12T22:27:50.496794257Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"127.0.0.1:56086","method":"GET","path":"/v2/artifact/blobs/sha256:732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d","statusCode":206,"latency":"29s","bodySize":2147483649,"headers":{"Range":["bytes=4294967296-6442450944"],"User-Agent":["Go-http-client/2.0"]},"goroutine":726027,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-06-12T22:27:55.172800387Z","message":"HTTP API"}

rchincha · 2024-06-17T02:33:16Z

@shizhMSFT reported this for Azure. Thanks!

Summary of findings that indeed parallel range requests does improve download performance however hits Azure infra limits fairly quickly.

shizhMSFT · 2024-06-17T15:01:21Z

@shizhMSFT reported this for Azure. Thanks!

Here are the detailed data points for the above diagram.

Connections	Download Speed (MiB/s)
1	132
2	142
3	147
4	144
5	145
6	143
7	147
8	142
9	145
10	142
11	146
12	141
13	143
14	140
15	144
16	144

The benchmark was generated by running for i in {1..16}; do aria2c -x 16 -s $i $url; done in an Azure D2s v3 VM with Premium SSD to download a 10 GiB blob in an Azure Blob Storage instance within the same region.

Detailed conversation can be found in the CNCF Slack channel #oras: https://cloud-native.slack.com/archives/CJ1KHJM5Z/p1718311259363889

rchincha requested review from dmcgowan, jzelinskie, jonjohnsonjr, jdolitsky, mikebrow, stevvooe and sudo-bmitch as code owners May 29, 2024 17:24

rchincha mentioned this pull request May 29, 2024

Add support for erofs and squashfs layer media types opencontainers/image-spec#1191

Open

samuelkarp reviewed May 29, 2024

View reviewed changes

rchincha force-pushed the perf branch 3 times, most recently from ab2e9e1 to 64cb41e Compare May 29, 2024 20:33

rchincha force-pushed the perf branch from 64cb41e to dd71826 Compare May 30, 2024 18:49

sudo-bmitch requested changes May 30, 2024

View reviewed changes

spec.md Outdated Show resolved Hide resolved

rchincha force-pushed the perf branch from dd71826 to 9d4c4a9 Compare May 31, 2024 04:41

sudo-bmitch approved these changes May 31, 2024

View reviewed changes

rchincha mentioned this pull request Jun 9, 2024

performance: what can dist-spec do to improve downloads of large images/layers? #536

Open

dmcgowan approved these changes Jun 20, 2024

View reviewed changes

dmcgowan merged commit 2291163 into opencontainers:main Jun 20, 2024
4 checks passed

rchincha mentioned this pull request Jun 20, 2024

Initial commit for OCI dist spec v1.1.0 agent support kserve/kserve#3539

Draft

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for content range requests when getting blobs #537

Add support for content range requests when getting blobs #537

rchincha commented May 29, 2024 •

edited by sudo-bmitch

Loading

sudo-bmitch commented May 29, 2024

rchincha commented May 29, 2024

rchincha commented May 29, 2024

samuelkarp left a comment

samuelkarp May 29, 2024

samuelkarp May 29, 2024

samuelkarp May 29, 2024

samuelkarp May 29, 2024

rchincha May 29, 2024 •

edited

Loading

rchincha commented May 29, 2024 •

edited

Loading

rchincha commented May 30, 2024

sudo-bmitch left a comment

sudo-bmitch left a comment

rchincha commented May 31, 2024

sudo-bmitch commented May 31, 2024

rchincha commented Jun 8, 2024 •

edited

Loading

rchincha commented Jun 17, 2024 •

edited

Loading

shizhMSFT commented Jun 17, 2024 •

edited

Loading

		@@ -190,6 +190,11 @@ If present, the value of this header MUST be a digest matching that of the respo

		If the blob is not found in the registry, the response code MUST be `404 Not Found`.

		A GET request may also include a `Range` request header to download part of a

		blob. The response code can either be `216 (Partial Content)` or `416 (Range
		Not Satisfiable)` in case of an invalid range. If the registry doesn't support

	blob. The response code can either be `216 (Partial Content)` or `416 (Range
	blob in accordance with [RFC 9110](https://datatracker.ietf.org/doc/html/rfc9110#name-range-requests). The response code can either be `216 (Partial Content)` or `416 (Range

		Not Satisfiable)` in case of an invalid range. If the registry doesn't support
		range requests, it can respond with `Accept-Ranges: none`.

Add support for content range requests when getting blobs #537

Add support for content range requests when getting blobs #537

Conversation

rchincha commented May 29, 2024 • edited by sudo-bmitch Loading

sudo-bmitch commented May 29, 2024

rchincha commented May 29, 2024

rchincha commented May 29, 2024

samuelkarp left a comment

Choose a reason for hiding this comment

samuelkarp May 29, 2024

Choose a reason for hiding this comment

samuelkarp May 29, 2024

Choose a reason for hiding this comment

samuelkarp May 29, 2024

Choose a reason for hiding this comment

samuelkarp May 29, 2024

Choose a reason for hiding this comment

rchincha May 29, 2024 • edited Loading

Choose a reason for hiding this comment

rchincha commented May 29, 2024 • edited Loading

rchincha commented May 30, 2024

sudo-bmitch left a comment

Choose a reason for hiding this comment

sudo-bmitch left a comment

Choose a reason for hiding this comment

rchincha commented May 31, 2024

sudo-bmitch commented May 31, 2024

rchincha commented Jun 8, 2024 • edited Loading

rchincha commented Jun 17, 2024 • edited Loading

shizhMSFT commented Jun 17, 2024 • edited Loading

rchincha commented May 29, 2024 •

edited by sudo-bmitch

Loading

rchincha May 29, 2024 •

edited

Loading

rchincha commented May 29, 2024 •

edited

Loading

rchincha commented Jun 8, 2024 •

edited

Loading

rchincha commented Jun 17, 2024 •

edited

Loading

shizhMSFT commented Jun 17, 2024 •

edited

Loading