Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: upgrade to badgerV4 #3340

Merged
merged 14 commits into from
May 30, 2023
Merged

chore: upgrade to badgerV4 #3340

merged 14 commits into from
May 30, 2023

Conversation

cisse21
Copy link
Member

@cisse21 cisse21 commented May 17, 2023

Description

Benchmarks with BadgerV4
Benchmark_Dedup/no_duplicates_1000_batch_unique-8 450210 4657 ns/op 16207560 bytes 450210 events
dedup_test.go:174: db size: 25M testdata/tmp/JrafTtIrtY/DB_Benchmark_Dedup

BenchMarks with BadgerV2
Benchmark_Dedup/no_duplicates_1000_batch_unique-8 464840 4093 ns/op 16734240 bytes 464840 events
dedup_test.go:180: db size: 59M testdata/tmp/OvBnPAnaHD/DB_Benchmark_Dedup

BadgerV2 has better performance but uses significantly higher disk size(almost 2x) than BadgerV4

Notion Ticket

Notion Link

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented May 17, 2023

Codecov Report

Patch coverage: 77.55% and project coverage change: -0.01 ⚠️

Comparison is base (471a562) 68.58% compared to head (eb25ca4) 68.58%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3340      +/-   ##
==========================================
- Coverage   68.58%   68.58%   -0.01%     
==========================================
  Files         330      330              
  Lines       52820    52804      -16     
==========================================
- Hits        36229    36213      -16     
+ Misses      14261    14259       -2     
- Partials     2330     2332       +2     
Impacted Files Coverage Δ
services/pgnotifier/pgnotifier.go 71.08% <0.00%> (-0.27%) ⬇️
...rprise/suppress-user/internal/badgerdb/badgerdb.go 78.53% <25.00%> (ø)
services/dedup/badger.go 58.57% <40.00%> (-12.09%) ⬇️
services/debugger/cache/internal/badger/badger.go 71.75% <70.00%> (+0.88%) ⬆️
processor/manager.go 94.73% <100.00%> (ø)
processor/processor.go 87.63% <100.00%> (+0.10%) ⬆️
services/dedup/dedup.go 94.66% <100.00%> (+0.54%) ⬆️
warehouse/internal/repo/upload.go 83.36% <100.00%> (ø)
warehouse/upload.go 65.06% <100.00%> (ø)

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@cisse21 cisse21 marked this pull request as ready for review May 23, 2023 11:48
services/dedup/dedup.go Outdated Show resolved Hide resolved
services/dedup/dedup.go Outdated Show resolved Hide resolved
@cisse21 cisse21 requested a review from atzoum May 30, 2023 03:41
Co-authored-by: Aris Tzoumas <atzoumas@rudderstack.com>
@cisse21 cisse21 requested a review from atzoum May 30, 2023 05:35
@cisse21 cisse21 merged commit 3ecea32 into master May 30, 2023
@cisse21 cisse21 deleted the chore.moveToBadgerV4 branch May 30, 2023 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants