Add etag and other caching headers to all content addressed gets in api

[ianopolous]

Version information:

N/A

Type: Enhancement

Priority:

P3

Description:

After discussion in irc, I'm submitting this feature request. The origin of this requirement is the following: Previously we've had our own http api for Peergos with caching working fine, but now we're trying to move to match IPFS' http api to allow us to self host from within IPFS. However it seems that the object api currently doesn't include etag headers and are not cached. Local tests however would suggest that even including them will not make the browser (chrome only tested) cache the requests. My guess is that the hash will need to be included in the path, rather than the query to get the caching working.

e.g. https://ipfs.io/api/v0/object/get?stream-channels=true&arg=QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D

Currently we are using solely the ipfs object api (get, data, and the writing calls new, addLink, setData) However, I'd like to migrate to the simpler dag api, once it supports cbor, but it seems to have the same problem.

[ianopolous]

An easy way to demonstrate the browser's non caching behaviour, even with the correct etag header present is
https://ipfs.io/ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D?key=thiswontbecached
(chrome won't cache this, ... though it seems firefox will, so maybe the non caching with etag is a chrome bug :-( )

[ianopolous]

After talking to a chromium engineer it turns out this is by design. Pressing enter in the url bar, or F5 will ignore the cache, even if the url is set to cache for a year with immutable and etag set. It should be returned from cache when called within a page, either as an ajax call or as a subresource. So it really is just a matter of setting the same headers on object calls.

[lidel]

I agree, we could provide significant optimization by adding cache-control: immutable to responses with immutable data from API endpoints at:

• /api/v0/object/get/
• /api/v0/dag/get/

However, immutable flag should be set only if:

• Response is successful (HTTP Status Code is 200)
• .. AND the arg parameter points at immutable resource (was a valid CID or an IPFS path starting with /ipfs/).
  • We don't want to cache mutable /ipns/ forever, eg. https://ipfs.io/api/v0/object/get?arg=/ipns/ipfs.io/images/ipfs-logo.svg (or it should be cached only for a short time, eg. 1minute)

[ianopolous]

We no longer use the object api, but have replaced it with the block api, but the same thing applies there.

[lidel]

We discussed this during Friday issue triage and decided these additional HTTP semantics do not belong under /api/v0:

The HTTP API under /api/v0/ is an RPC-style API over HTTP, not a REST API.
– https://docs.ipfs.io/reference/http/api/

In short: if one needs caching of RPC API call responses, then it needs to happen at application level.

---

Note that HTTP cache control is already supported on gateway's /ipfs/ paths.

@ianopolous is getting JSON representation of directory listing the only reason for filling this issue?

I believe we could add support for returning JSON instead of HTML for dir listings
(triggered by passing Accept: application/json header or &enc=json param)
Feel free to PR or open a separate issue for adding that feature.