Skip to content
This repository has been archived by the owner on Sep 21, 2024. It is now read-only.

feat: Improved IPFS Kubo syndication #666

Merged
merged 1 commit into from
Oct 6, 2023
Merged

Conversation

cdata
Copy link
Collaborator

@cdata cdata commented Oct 5, 2023

This change proposes a significant revision to how we syndicate to IPFS Kubo from gateways.

  • We now use the same CAR streaming used in client<->gateway sync and replication
  • The bloom filter is dropped in favor of keeping faith in "pin roots" and ensuring rigorous collection of blocks in memo_body_stream
  • Syndication checkpoints now expire every 90 days, forcing re-syndication of history

Additionally, a small utility example has been implemented to verify the shape of CAR files produced by Noosphere's CAR streaming facilities.

@cdata cdata force-pushed the fix/ipfs-scaling-issues branch 6 times, most recently from 5d05e12 to 0bcd0ed Compare October 5, 2023 19:40
@cdata cdata marked this pull request as ready for review October 5, 2023 19:49
@cdata cdata requested a review from jsantell October 5, 2023 19:49
jsantell
jsantell previously approved these changes Oct 5, 2023
Copy link
Contributor

@jsantell jsantell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤘

Cargo.toml Show resolved Hide resolved
rust/noosphere-gateway/src/worker/syndication.rs Outdated Show resolved Hide resolved
rust/noosphere-ipfs/src/client/kubo.rs Show resolved Hide resolved
rust/noosphere-gateway/src/worker/syndication.rs Outdated Show resolved Hide resolved
@cdata cdata force-pushed the fix/ipfs-scaling-issues branch from 0bcd0ed to b1641e9 Compare October 6, 2023 02:44
@cdata cdata enabled auto-merge (squash) October 6, 2023 03:05
@cdata cdata disabled auto-merge October 6, 2023 03:05
@cdata cdata merged commit eeab932 into main Oct 6, 2023
@cdata cdata deleted the fix/ipfs-scaling-issues branch October 6, 2023 03:05
@github-actions github-actions bot mentioned this pull request Oct 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants