Proposal: IPFS Gateway HTTP API

[anorth]

This is a project proposal for the web3 dev team.

[anorth]

This is the first proposal, being used as a process example as well as a real proposal. So we're figuring out the workflow as we go.

I think review here is only about whether this is an appropriate proposal, and for discussion/collaboration on the proposal itself. Approval signals nothing about whether we should actually undertake the proposal at any point in time. We'll discuss that in separate issues.

[anorth]

FYI @gmasgras @achingbrain @autonome

[willscott]

just the golang / server side implementation, right?

[anorth]

JS implementation will be necessary for sub-DAG queries from the client, which would be needed for complex cases. So no, not just server side. But also we needn't block creating and exposing this API on that JS selector implementation (I gather it doesn't exist?). It will just be more useful once JS clients can express queries.

[warpfork]

👍 👍

Designing an API that's usable by a lightweight client, while also being designed explicitly to avoid "N+1 query problems", seems very high-value to me.

[achingbrain]

I'm +1 for a simple API that doesn't need a complicated client to access. However this proposal or the design of the outcome of this proposal should take into account the limitations of the existing HTTP API when used in anything that implements browser Fetch.

TLDR: you cannot do full-duplex streaming, so design the API as simple request/response pairs, or only require streaming from the server to the client.

More detail:

1. Limited streaming support
   • We use multipart requests to simulate streaming from the client to the server
   • Fetch uses FormData which requires adding all parts to the request before sending - so no streaming uploads.
     • You can use Blobs as parts, but these only work in a memory-efficient way if they are Files.
     • You can create a Blob from an array or a string, so you can have JSON Blobs (for example), but these are kept in memory until the FormData is disposed of, so it doesn't scale and you still need to add them all to the request before any of it is sent
     • According to the specs you can use a ReadbleStream as the request body, but it's not implemented anywhere (yet, I think Chrome just did an 'intent to ship'). Even if it is, there's no guarantee it'll be full-duplex (see next point)
   • Browsers require sending the entire request before a single byte from the response is made available to the calling code, so no full-duplex streaming
     • You can almost use XMLHttpRequest for this by accessing the .responseText property before the response completes, but the response body is buffered in-memory as text until the .readyState is DONE, so it's not a solution either
2. No HTTP trailers
   • Errors cannot be sent in HTTP trailers, they are not readable by fetch, instead they need to be sent in the response body

In js-IPFS we settled on using gRPC over websockets which addresses these points, though the API becomes harder to use without a client which runs counter to the spirit of this proposal.

This may not be an issue, most people reach for some sort of client when dealing with an API in order to get something that respects the conventions and norms of their development platform and to not have to deal with the weird foibles of whatever the API developers thought was REST* - the people cURLing APIs are not (I believe) the majority of Web2-graduating-to-Web3 developers, not after the first few minutes anyway.

Further reading:

• Error handling for streaming API handlers in the browser, has it ever worked? ipfs/js-ipfs#2519
• feat: add grpc server and client ipfs/js-ipfs#3403

---

• I'm kidding, no-one seems to know (or at least can agree on) what REST is

[lidel]

The true scope of this proposal will depend on how capable Writable Gateway approach (added it as prior art below) turns out to be when it is not limited to files, but can operate on IPLD DAGs, control pinning.

But yes, I imagine it being simple request/response pairs with potential for leveraging https://streams.spec.whatwg.org when those features land in browsers, and perhaps, in the future support resumable uploads as well (prior art: https://tus.io, ipfs/js-ipfs#1540) to make DX/UX easier when working with huge imports.

We have distinct Gateway (/ipfs, /ipns) and API (/api/v0) endpoints today, and only the most basic read operations can be executed via Gateway. The goal here would be to shift what is possible on the Gateway, so most popular operations work without touching advanced API.

Error reporting on /api/v0 is broken in browsers, as noted by @achingbrain in ipfs/js-ipfs#2519,
so for everything not handled by Gateway (eg. pubsub, advanced streaming), we need to replace HTTP-based /api/v0 with something like RPC over websockets anyway (js-ipfs already does this: ipfs/js-ipfs#3403).

[anorth]

Thanks! I've noted the HTTP streaming limitations in the body. @lidel I have also declared you a co-author, thanks for fleshing some bits out.

[achingbrain]

Error reporting on /api/v0 is broken in browsers

It's worse than that, errors and streaming are broken anywhere that uses Fetch - that's browsers, node-fetch, electron-fetch, React Native, etc.

The only place it does work is via node's http module, but Fetch is the thing that users reach for when they need to make a network request on pretty much every platform JS runs on and it will be what they use if you're going for clientless here.

[Stebalien]

I'd like to avoid supporting two APIs if possible unless we intend to deprecate/remove one of them. I'm concerned that either:

1. This limited REST API will eventually grow to feature parity with an RPC API.
2. Applications will inevitably outgrow the REST API and need to either use both APIs or migrate to the RPC API.

That's why I'm personally in-favor of the gRPC+websocket API.

On the other hand, I have occasionally wanted a slightly more featureful gateway, mostly for integration with scripts where there is no convenient js-ipfs-http-client. But I'd still prioritize making the more general purpose API suck less first.

[lidel]

l feel we have a soft consensus(?) that we need to deprecate /api/v0 over HTTP due to reasons mentioned in #1 (comment) and a proposal to replace it with something else. JS IPFS added gRPC+websocket API as drop-in replacement that finally works like real RPC.

Perhaps that should be a separate Proposal tho?
I really feel there is a merit in making Gateways more useful, and would like to keep this one as-is so we can keep it on the list of projects.

[mikeal]

Love it.

One technical consideration I’d like to add: whenever we create an API or a protocol that uses selectors we should modularize the use of selectors so that, in the future, we could swap selectors out for another query mechanism and reuse the rest of the protocol or API. This should be easy, we just need to use a union and it shouldn’t add any extra burden to the implementation and will futureproof the work.

[mikeal]

Would love to see one more review approval from someone who works on the gateway.

[momack2]

@gmasgras @hsanjuan - could you review please?

[BigLep]

We're going this direction, but the work is covered in other proposals including in #100 . Closing as a result.