Bulk Provide/Reproviding System #34

aschmahmann · 2021-04-09T15:27:44Z

Add an alternative provide/reproviding system that:

keeps track of when reproviding was done so we don't just do it whenever we start up the provider system if we're given a datastore with persistence
requires an interface that deals with providing many objects at once rather than just one at a time
emits some basic stats around how much providing has been done

…racking of individual CID reprovide times, and have a very simple and fast stats function

…g to provide

…ng LastReprovide entry in the datastore

batched/system.go

queue/queue.go

batched/system.go

…umentation

queue/queue.go

batched/system.go

Stebalien

Go timers are... special.

batched/system.go

…vides

batched/system.go

…eturning a closed channel

…ovideCidLoop since there is no static + dynamic reprovide channels anymore

batched/system.go

Stebalien · 2021-05-12T22:24:13Z

batched/system.go

+
+func resetTimer(t *time.Timer, dur time.Duration) {
+	if !t.Stop() {
+		<-t.C


This will block forever if the timer wasn't already running.

Stebalien

More tests would be nice, but otherwise LGTM (module one small change).

batched/system.go

fix: in queue sync datastore to disk

c707e99

aschmahmann force-pushed the feat/bulk-provide branch 2 times, most recently from f9f54af to 5a4e4c4 Compare April 9, 2021 15:40

feat: add a batching provider/reprovider system

c193919

aschmahmann force-pushed the feat/bulk-provide branch from 5a4e4c4 to c193919 Compare April 12, 2021 23:06

aschmahmann mentioned this pull request Apr 13, 2021

experimental provider system ipfs/kubo#8070

Merged

aschmahmann added 3 commits April 22, 2021 09:50

simplify batch provider to remove the managed provider list, remove t…

829bc09

…racking of individual CID reprovide times, and have a very simple and fast stats function

batch provider waits for provider system to be ready before attemptin…

0fdf1d6

…g to provide

fix: handle periods with no providing, and handle the case of a missi…

93ff866

…ng LastReprovide entry in the datastore

Stebalien suggested changes May 11, 2021

View reviewed changes

aschmahmann commented May 11, 2021

View reviewed changes

batched/system.go Outdated Show resolved Hide resolved

aschmahmann added 6 commits May 12, 2021 13:11

moved import

fb01b26

batched: moved context creation to prevent leakage

bf3cf56

cleanup code a bit and add more logging

0023040

remove unnecessary goroutine

8bbdb3b

rename getTime to parseTime

9e61fa1

removed newly added syncs from the provider queue and updated its doc…

c3fd90b

…umentation

aschmahmann force-pushed the feat/bulk-provide branch from 90acf8c to c3fd90b Compare May 12, 2021 18:05

aschmahmann commented May 12, 2021

View reviewed changes

queue/queue.go Show resolved Hide resolved

aschmahmann added 2 commits May 12, 2021 14:13

batched: wait for goroutines to close before returning from Close

c52c0b0

batched: reuse timers in the Run function

34b4e40

aschmahmann commented May 12, 2021

View reviewed changes

batched/system.go Outdated Show resolved Hide resolved

renamed variable

4407f46

Stebalien suggested changes May 12, 2021

View reviewed changes

batched/system.go Outdated Show resolved Hide resolved

batched/system.go Show resolved Hide resolved

batched/system.go Show resolved Hide resolved

batched/system.go Outdated Show resolved Hide resolved

aschmahmann added 2 commits May 12, 2021 15:54

some go timer fixes

f498246

have a max collection threshold even when only processing regular pro…

cffff34

…vides

aschmahmann force-pushed the feat/bulk-provide branch from 4f3814c to cffff34 Compare May 12, 2021 20:01

aschmahmann commented May 12, 2021

View reviewed changes

batched/system.go Outdated Show resolved Hide resolved

batched/system.go Outdated Show resolved Hide resolved

aschmahmann marked this pull request as ready for review May 12, 2021 20:10

Stebalien reviewed May 12, 2021

View reviewed changes

batched/system.go Outdated Show resolved Hide resolved

batched/system.go Show resolved Hide resolved

batched/system.go Show resolved Hide resolved

batched/system.go Outdated Show resolved Hide resolved

batched/system.go Outdated Show resolved Hide resolved

aschmahmann added 5 commits May 12, 2021 16:34

max collection timer only starts once we have our first provide

6d08b5d

remove cid from provide queue if its invalid

3a94d3a

switch reproviding channel to be unbuffered

866d979

batched: do not panic if no key provider is set, instead default to r…

f84c7d9

…eturning a closed channel

batched: renamed dynamicCh and dynamicCidLoop to reprovideCh and repr…

8c07658

…ovideCidLoop since there is no static + dynamic reprovide channels anymore

aschmahmann commented May 12, 2021

View reviewed changes

batched/system.go Show resolved Hide resolved

Stebalien suggested changes May 12, 2021

View reviewed changes

aschmahmann added 7 commits May 12, 2021 19:23

batched: add test

71c6564

fix max collection check

b017f26

fixed waitgroup usage bug

d366b61

batched: internally configurable initial reprovide

843e75c

batched: add basic test

04452c4

batched: rework timer usage

edb6021

use closure

1c19caa

aschmahmann requested a review from Stebalien May 13, 2021 02:32

Stebalien approved these changes May 13, 2021

View reviewed changes

batched/system.go Outdated Show resolved Hide resolved

fix debug log usage

c86f3de

aschmahmann merged commit d391dae into master May 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bulk Provide/Reproviding System #34

Bulk Provide/Reproviding System #34

aschmahmann commented Apr 9, 2021

Stebalien left a comment

Stebalien May 12, 2021

Stebalien left a comment

Bulk Provide/Reproviding System #34

Bulk Provide/Reproviding System #34

Conversation

aschmahmann commented Apr 9, 2021

Stebalien left a comment

Choose a reason for hiding this comment

Stebalien May 12, 2021

Choose a reason for hiding this comment

Stebalien left a comment

Choose a reason for hiding this comment