Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support refs -r pinning method #326

Closed
hsanjuan opened this issue Mar 2, 2018 · 1 comment · Fixed by #331
Closed

Support refs -r pinning method #326

hsanjuan opened this issue Mar 2, 2018 · 1 comment · Fixed by #331
Assignees
@hsanjuan
Labels
exp/intermediate Prior experience is likely helpful P2 Medium: Good to have, but can wait until someone steps up

Comments

@hsanjuan
Copy link
Collaborator

hsanjuan commented Mar 2, 2018

It should be possible to configure the ipfs connector to perform pinning with refs -r + pin rather than just
pin. This should allow to perform fetching in parallel.
@hsanjuan hsanjuan added help wanted Seeking public contribution on this issue exp/intermediate Prior experience is likely helpful status/ready Ready to be worked P2 Medium: Good to have, but can wait until someone steps up labels Mar 2, 2018
@hsanjuan
Copy link
Collaborator Author

hsanjuan commented Mar 2, 2018 

<lgierth> Lars hsanjuan: are you importing things with `pin add` or `refs -r`?
18:23:38 <hsanjuan> Hector lgierth: cluster does `pin add`
18:30:04 <lgierth> Lars ah i see, ok
18:30:12 so that's why other pins don't get worked on
18:30:27 or rather, take some time to get worked on
18:48:03 <hsanjuan> Hector lgierth: refs -r will fetch everything and works in parallel ?
18:50:42 <lgierth> Lars yeah. it's fetches slightly differently from pin (more depth-first than breadth-first) but yes you can do a ton of `refs -r` calls in parallel
18:51:19 why keeps insisting that pin fetches faster, but i'll claim that refs fetches more consistently fast

@hsanjuan hsanjuan self-assigned this Mar 6, 2018
@hsanjuan hsanjuan added status/in-progress In progress and removed help wanted Seeking public contribution on this issue status/ready Ready to be worked labels Mar 6, 2018
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
8f944d3 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
e359602 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
4ff53ab 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
6ce5baf 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
@hsanjuan hsanjuan mentioned this issue Mar 7, 2018
Merged
hsanjuan added a commit that referenced this issue Mar 9, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
fb4812e 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
@hsanjuan hsanjuan added need/review Needs a review and removed status/in-progress In progress labels Mar 9, 2018
hsanjuan added a commit that referenced this issue Mar 10, 2018
@hsanjuan
Merge pull request #331 from ipfs/feat/pin-refs
c276c31 
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
@ghost ghost removed the need/review Needs a review label Mar 10, 2018
@hsanjuan hsanjuan mentioned this issue Mar 12, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hsanjuan hsanjuan

Labels
exp/intermediate Prior experience is likely helpful P2 Medium: Good to have, but can wait until someone steps up
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feat #326: Adds "refs -r" pinning method support + multiple pin workers ipfs-cluster/ipfs-cluster
1 participant
@hsanjuan