Reprovider strategies #4113
Reprovider strategies #4113
Conversation
magik6k
force-pushed
the
feat/reprovider-starts
branch
3 times, most recently
from
fa8524f
to
92df0d6
Compare
|
Low patch coverage is likely caused by not collecting coverage stats from iptb tests. For example cc @Kubuxu |
magik6k
force-pushed
the
feat/reprovider-starts
branch
5 times, most recently
from
b33fbfb
to
a07e3f9
Compare
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
magik6k
force-pushed
the
feat/reprovider-starts
branch
from
efb86e7
to
048debe
Compare
core/core.go
Outdated
| case "pinned": | ||
| keyProvider = rp.NewPinnedProvider(n.Pinning, n.DAG, false) | ||
| default: | ||
| return fmt.Errorf("unknown reprovider strtaegy '%s'", cfg.Reprovider.Strategy) |
docs/config.md
Outdated
| @@ -15,7 +15,7 @@ a running daemon do not read the config file at runtime. | |||
| - [`Identity`](#identity) | |||
| - [`Ipns`](#ipns) | |||
| - [`Mounts`](#mounts) | |||
| - [`ReproviderInterval`](#reproviderinterval) | |||
| - [`Reproviderl`](#reprovider) | |||
exchange/reprovide/reprovide.go
Outdated
| } | ||
|
|
||
| unmute := rp.muteTrigger() |
There was a problem hiding this comment.
could I get a comment or two on the muting? I think I get it, but would be great to clearly lay out whats going on
exchange/reprovide/reprovide.go
Outdated
| ) | ||
|
|
||
| var log = logging.Logger("reprovider") | ||
|
|
||
| type keyChanFunc func(context.Context) (<-chan *cid.Cid, error) |
There was a problem hiding this comment.
lets export this type, that way you dont have to redefine it inline up above.
exchange/reprovide/providers.go
Outdated
| if !onlyRoots { | ||
| err := merkledag.EnumerateChildren(ctx, dag.GetLinks, key, set.add) | ||
| if err != nil { | ||
| return //TODO: propagate to chan / log? |
There was a problem hiding this comment.
lets definitely not ignore this error, at least log.Error it (though propogating it upwards would be optimal)
exchange/reprovide/providers.go
Outdated
| } | ||
|
|
||
| type streamingSet struct { | ||
| set map[string]struct{} |
There was a problem hiding this comment.
you could reuse the cid.Set for this, it has a visit method that does exactly what you want down in 'add'
There was a problem hiding this comment.
The reason this is done like this is to reduce i/o hit on merkledag.EnumerateChildren that happens when reprovider starts on large repo (with cid.Set I got 100% io use for about 5min with my repo)
There was a problem hiding this comment.
right, what i'm saying is that instead of having set here be a map[string]struct{}, it could instead be a *cid.Set
magik6k
force-pushed
the
feat/reprovider-starts
branch
4 times, most recently
from
ddaddaf
to
5b078b0
Compare
magik6k
force-pushed
the
feat/reprovider-starts
branch
2 times, most recently
from
729a3a8
to
fcaf2b9
Compare
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
magik6k
force-pushed
the
feat/reprovider-starts
branch
from
fcaf2b9
to
4a5b93a
Compare
Reprovider strategies
…r-starts Reprovider strategies
…r-starts Reprovider strategies
Reprovider strategies allow to specify which keys to provide to the content routing.
Supported strategies are:
all- all blocks,pinned- all pinned blocks,roots- root/directly pined blocks.Current reprovider is slow(~0.2keys/s with my setup), so constraining key set should speed the process up, announcing only pin roots should still work quite well in most cases.
TODO: