Implement Garbage Collection

[dirkmc]

resolves #2012

Depends on

• Implement ipfs refs #2004
• GC endpoint ipfs-inactive/js-ipfs-http-client#992
• Tests for Garbage Collection ipfs-inactive/interface-js-ipfs-core#462
• Fix: make sure write starts only after read completes achingbrain/mortice#1

TODO:

• Core (mark and sweep)
• CLI
• http interface
• interface-js-ipfs-core tests Tests for Garbage Collection ipfs-inactive/interface-js-ipfs-core#462
• nodejs-specific tests
• Locking
• Tests for locking

[dirkmc]

Note: I needed to refactor the pinning code in order to share code with some GC functionality. The functionality hasn't changed, things have just moved around somewhat.

[wehriam]

@dirkmc thanks for working on this, and looking forward to the finish line.

We're using JS-IPFS to run 24/7 video streams in the browser; it is fairly quickly filling up all available disk.

Please let me know if there's anything I can do to assist.

[dirkmc]

@wehriam that's great to hear, thanks! It would be great to get some feedback from some different use cases. Would you like to try out this branch to see if it helps with your use case? I can provide instructions if you're not sure how

[wehriam]

@dirkmc thanks! We did a build off of feat/gc and are testing it now. It seems to be working well so far.

Are you planning support for Datastore options similar to those in go-ipfs? They aren't in the config parser so I wasn't sure.

We're currently using the navigator.storage.estimate() method (when available) to determine when to trigger GC.

Are all unpinned blocks removed on GC? It would be great to have a way to remove them on an LRU basis so browser-based swarms could serve as an edge cache for large files.

[dirkmc]

@wehriam the implementation of GC on this branch must be manually invoked, and removes all blocks that are unpinned and not in MFS. We are currently discussing how best to implement automatic GC, including using an LRU heuristic.

You can see usage information for the IPFS repo with ipfs.repo.stat()

[wehriam]

@dirkmc I have a fiddle running here - https://jsfiddle.net/t9a5ogc1/ - you can see usage stats in the console.

[wehriam]

This works well in our testing. Are there any blockers for a final release? It appears ipfs-inactive/interface-js-ipfs-core#460 has been successfully merged.

[dirkmc]

@wehriam thanks for helping out with testing this in a real world scenario.

We are currently all hands on deck getting ready for IPFS camp, so it's likely this will be released soon after, once people have had time to code review it.

[alanshaw]

Just a real quick first pass review - I'll circle back to this and do a proper review asap.