Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polygon/p2p: Add blk/s and bytes/s to periodic log #9976

Merged
merged 15 commits into from
Apr 27, 2024
Merged

polygon/p2p: Add blk/s and bytes/s to periodic log #9976

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b459cec
Add blk/s and bytes/s
shohamc1 Apr 17, 2024
51751c4
Renames
shohamc1 Apr 18, 2024
438f47c
Add reset, use encoding size
shohamc1 Apr 19, 2024
c785a3d
Use size from fetch
shohamc1 Apr 23, 2024
d394cef
Merge branch 'refs/heads/devel' into shohamc1/astrid-progess-speed
shohamc1 Apr 23, 2024
8406bef
Add type constraint
shohamc1 Apr 23, 2024
5a1a8a5
Renames
shohamc1 Apr 23, 2024
c0e3c75
Fix tests
shohamc1 Apr 24, 2024
42a775d
Make FetcherResponse generic
shohamc1 Apr 24, 2024
abc293e
Reset fetchStartTime
shohamc1 Apr 24, 2024
bd2410e
Use atomic uint
shohamc1 Apr 24, 2024
28f3e8a
Use FetcherResponse
shohamc1 Apr 25, 2024
eea1854
Fix tests
shohamc1 Apr 25, 2024
47db312
Formatting
shohamc1 Apr 25, 2024
fb69419
Refactor
shohamc1 Apr 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions polygon/sync/block_downloader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,10 @@ func (d *blockDownloader) downloadBlocksUsingWaypoints(ctx context.Context, wayp

var lastBlock *types.Block
batchFetchStartTime := time.Now()
logStartTime := time.Now()
shohamc1 marked this conversation as resolved.
Show resolved Hide resolved
sum := 0
shohamc1 marked this conversation as resolved.
Show resolved Hide resolved
totalSize := float64(0)
shohamc1 marked this conversation as resolved.
Show resolved Hide resolved

for len(waypoints) > 0 {
select {
case <-ctx.Done():
Expand Down Expand Up @@ -162,7 +166,14 @@ func (d *blockDownloader) downloadBlocksUsingWaypoints(ctx context.Context, wayp
"kind", reflect.TypeOf(waypointsBatch[0]),
"peerCount", len(peers),
"maxWorkers", d.maxWorkers,
"blk/s", fmt.Sprintf("%.2f", float64(sum)/time.Since(logStartTime).Seconds()),
"bytes/s", fmt.Sprintf("%s", common.ByteCount(uint64(totalSize/time.Since(logStartTime).Seconds()))),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a helper function to calculate speed, and check for zero division there

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are safe without this check, since the log interval is set to 30 seconds. We similar divisions elsewhere in the codebase without checks
https://github.com/ledgerwatch/erigon/blob/bd378f2533052e1a97f61b2ad4cb855d6e385602/cl/phase1/stages/stage_history_download.go#L303

)

sum = 0
totalSize = 0
logStartTime = time.Now()
taratorio marked this conversation as resolved.
Show resolved Hide resolved

default:
// carry on
}
Expand Down Expand Up @@ -237,6 +248,11 @@ func (d *blockDownloader) downloadBlocksUsingWaypoints(ctx context.Context, wayp
}

d.logger.Debug(syncLogPrefix("fetched blocks"), "len", len(blocks), "duration", time.Since(batchFetchStartTime))

sum += len(blocks)
for _, block := range blocks {
taratorio marked this conversation as resolved.
Show resolved Hide resolved
totalSize += float64(block.Size())
}
batchFetchStartTime = time.Now() // reset for next time

if err := d.storage.InsertBlocks(ctx, blocks); err != nil {
Expand Down
Loading