Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

IPLD requirements for js-ipfs #1893

Closed
mikeal opened this issue Feb 23, 2019 · 9 comments
Closed

IPLD requirements for js-ipfs #1893

mikeal opened this issue Feb 23, 2019 · 9 comments
Assignees

Comments

@mikeal
Copy link
Contributor

mikeal commented Feb 23, 2019

The IPLD team is trying to get a handle on our roadmap and plan for the next few quarters. js-ipfs is the primary consumer of JS IPLD interfaces and we’d like to get a good list of things js-ipfs needs over the next year or so.

@alanshaw alanshaw added kind/support A question or request for support status/ready Ready to be worked labels Mar 12, 2019
@daviddias daviddias added the ipld label Oct 1, 2019
@daviddias
Copy link
Member

IPLD packs a ton of promises for the future of building distributed computing. However, rather than giving a long list of all future hopes and dreams, I'll make one request that reflects one current use case of IPLD over js-ipfs other than IPFS using it for UnixFS.

Can js-ipld reproducibility and correctly resolve data in formats used by blockchains such as the Ethereum Blockchain? This was possible before and it created a ton of momentum and value, things to note are:

It would be really great to have this stable and something that we can point people at as a great example of interoperability

@daviddias
Copy link
Member

Ah, another great example of interoperability between networks and formats is https://github.com/ipfs-shipyard/igis, right now it lacks the maintenance necessary.

@daviddias
Copy link
Member

One more ask (😇) support for Torrents #779

@mikeal
Copy link
Contributor Author

mikeal commented Oct 1, 2019

torrents

I actually thought that torrent codecs were already implemented because, at some point, you added repos for those implementations to https://github.com/ipld/ipld#implementations but apparently those are just empty repos 😱

Writing JS codecs for torrents would be pretty easy, I’ve spent time a fair amount of time in those torrent libraries already. I’m not going to consider this high priority (compared to unixfsv2) but this is something small and fun and I’ll probably get to it in Q4 cause I love you @daviddias 😘

ETH

AFAIK the codecs for ETH work and aren’t in need of any dramatic changes (@vmx already did a big refactor recently that puts them in alignment with our other async/await refactors).

Our team is pretty small and focused on improvements to the tools and primitives, this makes taking the “ETH on IPFS demo” and turning it into a longer term supported application quite difficult for our team to do directly. However, if there’s anything this demo needs from IPLD libraries we can prioritize those more highly, just let us know what it is.

ETH2 is another story and we’re following that development and even in the process of providing feedback to some of the ETH folks on one of their proposed new block formats.

In general, our focus this year has been on IPLD Schemas and IPLD “native” block formats and workflows. But it’s important that we don’t forget about the basics, like just being able to link between block formats, and when we do planning for 2020 we should consider taking on any new block formats and even making some usability improvements to existing formats like git (which works but at a very low level, we could do more here to make it easier to pull out actual file data).

@daviddias
Copy link
Member

you added repos for those implementations to https://github.com/ipld/ipld#implementations but apparently those are just empty repos 😱

Ups, sorry! I did a lot of experimentation locally and found some obstacles along the way, then my attention was routed to other things (it was mid 2017 :)). I left a list of things to be done at #779

Writing JS codecs for torrents would be pretty easy, I’ve spent time a fair amount of time in those torrent libraries already.

True, but depending on how it is integrated, it gets tricky. A torrent file is referenced by the hash (called infohash in torrent world) of the file that the torrent is sharing, but not the torrent file itself. torrent files keep mutating as new trackers get added.

Adding support to resolve torrent file chunks to IPLD should be trivial, I agree, however, in the ideal world one should be able to do something like ipfs torrent cat /infohash and it would pipe out the contents of the torrent by connecting to the torrent network and piecing the file together with bits existing there and other bits already existing in the IPFS network, if available.

I’m not going to consider this high priority (compared to unixfsv2) but this is something small and fun and I’ll probably get to it in Q4 cause I love you @daviddias 😘

I didn't mean to derail the plan for Q4, but this does make me super happy, thaaank you ❤!

ETH

Understood. It would be good to:

  • Ensure that the example is running perfectly
  • Potentially re record a video demo that can be shared
  • Reach out to @kumavis and @hermanjunge to learn what is the latest state of Mustekala. I recall that for the Blockchain usecase, Bitswap was not ideal and now with Graphsync, it can potentially make the use case much more powerful.

@mikeal
Copy link
Contributor Author

mikeal commented Oct 1, 2019

True, but depending on how it is integrated, it gets tricky. A torrent file is referenced by the hash (called infohash in torrent world) of the file that the torrent is sharing, but not the torrent file itself. torrent files keep mutating as new trackers get added.

Ya, I had read the prior threads and I’m already taking for granted the torrent-file + torrent approach at the IPLD layer.

Adding support to resolve torrent file chunks to IPLD should be trivial, I agree, however, in the ideal world one should be able to do something like ipfs torrent cat /infohash and it would pipe out the contents of the torrent by connecting to the torrent network and piecing the file together with bits existing there and other bits already existing in the IPFS network, if available.

As a first step, we should just get ipfs cat CID-of-torrent-file working.

Rather than infohash, why don’t you just use a magnet URL? We could even add a parameter to the magnet URL for the CID of the torrent-file, then it would contain everything we need for getting the file out of both networks. Maybe this sounds like cheating, but from a user perspective it’s already common to prefer magnet urls over infohashes, and you’ve already stubbed out a separate command in your example for getting torrents so the parameter doesn’t have to conform to multiaddr.

The networking side of this is outside the scope of IPLD but I can try to advise where I can since I have some experience with these JS libraries and with bittorrent in general.

@daviddias
Copy link
Member

Two other requirements:

@mikeal
Copy link
Contributor Author

mikeal commented Oct 17, 2019

I’ve thought a lot about the “patch” API and I’m still not sure it’s possible to do something atomic and performant.

A patch API, or any mutation API, needs to be efficiently batched in order to be performant when manipulating the same parts of the DAG concurrently. This is actually something I plan on writing soon on top of unixfsv2 (for mfs), taking all concurrent mutation operations and batching them into one large graph mutation that doesn’t unnecessarily create/read any part of the tree more than once.

I can produce this as a standalone library for efficient graph manipulation, but the API will have to sit on top of a larger mutex used for batching concurrent operations and I’m not sure how well that would fit into js-ipfs’s current architecture. It would have an API along the lines of:

const m = mutex(rootNode, { /* storage functions */ getBlock, setBlocks })
await m.set(/path/to/node’, newBlock)

You could use this atomically by just creating a new mutex for every operation, but it won’t be nearly as performant as keeping a single instance around and hammering it with concurrent mutations like I’ll do with the mfs code.

In short, you’ll have something for this by the end of the quarter, but it may take larger changes in js-ipfs in order to make it actually perform well.

@SgtPooki
Copy link
Member

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Status: Done
Development

No branches or pull requests

4 participants