Support HEAD requests

[juliangruber]

From https://specs.ipfs.tech/http-gateways/trustless-gateway/#head-ipfs-cid-path-params:

1.2 HEAD /ipfs/{cid}[/{path}][?{params}]
Same as GET, but does not return any payload.

[juliangruber]

Blocks CheckerNetwork/frisbii-on-fly#3

[rvagg]

What use-case is this going to serve for you? Since Frisbii takes a no-buffering approach I think the utility would be limited to finding out if the first block is available or not. Is that all you need from it? I know Boxo takes a different approach and does a bunch of up-front buffering before it starts serving so can deal with a few other classes of errors but I'm not very keen on that approach.

[juliangruber]

We're currently using a deployed Frisbii instance as an SP we can rely on, in automated tests. Therefore for us it would be sufficient if the HEAD request just always got a 200 response, for example. It also doesn't need to have any headers. Since our use case is limited, I think we could also live off a fork (I have a very basic HEAD implementation ready locally). Wdyt?

[lidel]

@rvagg doing "quick root block presence check" for HEAD is sensible.

Boxo and Storacha already do that as optimization for HEAD /ipfs/cid?format=raw – we check the root block, nothing more, and return 200 if block is present:

$ curl "https://dag.w3s.link/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR?format=raw" -I
HTTP/2 200

$ curl "https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR?format=raw" -I
HTTP/2 200

Implementation wise, HEAD does not have to fetch entire payload nor return any extra headers GET does based on payload (RFC uses SHOULD, but not MUST). This enables us to return HTTP 200/404 fast. Our focus is on application/vnd.ipld.raw responses but I think it is fine to apply this optimization to deserialized responses too.

FYSA HEAD is basic presence check will improve performance of HTTP retrieval, we started work across IPFS ecosystem:

• HTTP retrieval is wip in HTTP retrieval proposal ipfs/boxo#747 (cc @hsanjuan)
  • we are running internal tests with ipfs.io and Storacha team to use ?format=raw instead of Bitswap
  • at some point in the future every peer that announces /tls/../http listener will be probed with HEAD first, and if HEADS are supported they will act as HAVE checks rather than WANTs to avoid GET responses from multiple peers.
    • HEAD-based check is being documented at trustless gateway: document probing path (identity cid) ipfs/specs#491

[rvagg]

Thanks for all the details @lidel, this is helpful. I did see ipfs/specs#491 pass by but how much of a limbo is that in? Is it likely to land roughly as is or should I wait and watch progress on that?

[hsanjuan]

@rvagg it will land :) I hate to work on things that don't land. (and it will not change much).

[bajtos]

This issue where @hsanjuan requested Spark to check HEAD retrievals can offer some additional context:

• Test HEAD requests before GET CheckerNetwork/spark-checker#104