Optimizing IPNS

[Kubuxu]

As for today IPNS is painfully slow, every minute you have to wait over 10 seconds for entry to be rechecked. I understand that aim of that is maximum consistency but it isn't crucial in many if not most use cases.

So long IPNS lookups and lack of proper caching makes IPNS unusable for distribution of any user viewable content. The resolution time is just too long.

Even though IPRS is coming any part of its spec do not improve the performance of IPNS. As it was mentioned if IPFS is to become popular it has to be fast, currently IPNS isn't best example of that principle.

Problem lies in looking for consistency in cases where it is not needed on this level. Current cache time, which is good for achieving high consistency, also means that in many cases record is out of the cache before it is even used again. Also with only one caching stage, even if user uses the record all the time he will meet a resolution latency as only one stage caching is used.

Proposition, allow IPNS to specify, during publishing, the cache life, it is maximum time the entry is treated as actual without running full resolution. The difference between cache life and expiration time is that if node is separated from the network entry's is still valid as node running resolution in separation will just decide that its entry is newest available and thus reset its cached time.

To work around the problem of downtime of heavily used entry, during which no valid cached entry is available, new resolution process should be started before cache time elapses. When cached entry is used its cached time should be checked, if it exceeds half of cache life background re-resolution of entry should take place.

This mechanism still allows for bes possible consistency while making performance of IPNS usable for real world applications.

If you have any questions or I am unclear in some of my points, please ask.

Current IPNS performance (0.4 from master): https://gist.github.com/Kubuxu/76de80267e2d0eb10f3d

[hackergrrl]

I think IPRS in toto captures this intent and more: letting users and applications define the meaning of "validity". Though yes, waiting for this larger piece to get in is undesirable. 😿

Somewhat relevant: for the high speed pubsub case, I have a proposal out for creating an overlay network of publishers and subscribers.

Also, cc @whyrusleeping, who I think has some ideas about speeding up IPNS in the near-ish future?

[Kubuxu]

Notion of validity do not change anything as in case caching, it will allow to change the 24h expiration time or for creation for more advanced expirations.

Expiration and caching life of the record are two separate thing.

Also pub-sub do not help much in case of websites.

[hackergrrl]

I really like this part:

To work around the problem of downtime of heavily used entry, during which no valid cached entry is available, new resolution process should be started before cache time elapses. When cached entry is used its cached time should be checked, if it exceeds half of cache life background re-resolution of entry should take place.

Being able to keep a record around but still keep a pulse on e.g. the DHT for newer records seems like a useful improvement. I think this'd be a nice piece to incorporate into the record layer in some form. 👍