Efficient streaming from S3/Azure Storage

[yishaigalatzer]

What should we add or change to make your life better?

In NuGet (and many other apps) you want to use storage to handle your large file storage, but getting file streaming right (and highly effective) is hard and is implemented by individual frameworks separately.

The key issues are (that are solved to some extent by aspnetcore, but you need to code to them)

1. Mapping storage in your URL space. This is important to create a uniform URL space that is easy to track and manage firewalls for.
2. Ability to work around speed limitations on storage services, e.g. the differences between large and small blobs, automatically restarting failed requests, working with multiple replicas of the data to optimize speed to first byte and overall transfer speed.
3. Ability to run high load at an effective price point. I.e. how many requests can be served through the proxy simultaneously without a major impact on latency
4. Measure the results at high percentile request P99 -> P99.99 and also optimize the P100
5. Ability to programmatically pick different sources (e.g. table storage for small files, but storage for large files)
6. Specifically I worry about long running requests that will get their memory promoted to gen2, and with a long running process, and a mix of large/small files or fast/slow connections this can trigger high percentile latency delays.

Why is this important to you?

I've now seen this pattern show up in multiple services my team's have built over the years. We sometimes choose (like in NuGet3) to offload the streaming straight to storage, but that has performance side effect because we expose the customer directly to the limitation of storage when a service may have a different SLA in mind.

A proxy can solve this in a nice way for multiple micro services in a kubernetes cluster, where the streaming work is delegated by all other services to a dedicated service that can handle these concerns without having to optimize and configure the code in each micro service.

When running the streaming through a service owned proxy, there are better opportunity to capture and aggregate relevant metrics, perform additional validations inline (if chosen) that may or may not be available or are cost effective on the storage service(s)

[analogrelay]

A lot of this seems very coupled to a specific application pattern. What kind of things do you think the general-purpose proxy would provide, as compared to how any app would configure it? Certainly many of these requirements involve YARP providing specific features and we can track that work.

1. Mapping storage in your URL space. This is important to create a uniform URL space that is easy to track and manage firewalls for.

This seems to be something our existing routing can cover, but I may be wrong.

2. Ability to work around speed limitations on storage services, e.g. the differences between large and small blobs, automatically restarting failed requests, working with multiple replicas of the data to optimize speed to first byte and overall transfer speed.

These seem to be covered by features such as Retry (#56) and Mirroring (#105).

3. Ability to run high load at an effective price point. I.e. how many requests can be served through the proxy simultaneously without a major impact on latency

4. Measure the results at high percentile request P99 -> P99.99 and also optimize the P100

6. Specifically I worry about long running requests that will get their memory promoted to gen2, and with a long running process, and a mix of large/small files or fast/slow connections this can trigger high percentile latency delays.

To me these all come down to performance goals we are already tracking. Keeping latency low, RPS high, and memory usage low are all goals and how specifically we do that (managing gen2 memory) will be determine by profiling. Measuring P99+ requests are a key measurement though.

5. Ability to programmatically pick different sources (e.g. table storage for small files, but storage for large files)

I don't see this as a key scenario for YARP to provide built-in. It's not going to be a kitchen sink proxy :). A key motivator for YARP is the ability to plug in custom code while still taking advantage of the rich features already provided though, so this seems like a perfect example of something that should be easy to implement with custom code in YARP.

[samsp-msft]

Triage: Not a planned YARP feature as explained by @anurse above.