Make sure to GC as much as possible #56

magik6k · 2019-03-19T22:04:39Z

No description provided.

Stebalien · 2019-03-20T00:49:59Z

I'm not noticing any difference when testing with go-ipfs. Badger still isn't GCing anyting when running ipfs repo gc.

BenLubar · 2019-03-20T12:24:20Z

I ran ipfs repo gc several times with an offline copy of my IPFS data directory (the repo had recently been garbage-collected with the latest released version of ipfs), and here are the sizes of the badgerds folder after each run:

81G 65G 63G 62G 60G 58G 58G 57G 57G 57G 57G 56G 56G 50G 47G 44G 44G 44G 41G 34G 34G 34G 34G 33G 33G 32G 32G 32G 32G 32G 31G 31G 30G 30G 30G 30G 30G 30G 30G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G 29G

For reference, running badger backup and badger restore on the same badgerds folder without any garbage collection resulted in a folder that's 22G.

hsanjuan · 2019-03-20T13:36:47Z

Would https://godoc.org/github.com/dgraph-io/badger#DB.Flatten help?

magik6k · 2019-03-20T13:37:25Z

@BenLubar thanks for testing.

Can you run

du -sh ~/.ipfs/badgerds/ --exclude='*.vlog'
du -sh ~/.ipfs/badgerds/ --exclude='*.sst'

in both repos (or at least the GCed one)?

I've also noticed that we are actually running https://github.com/dgraph-io/badger/tree/8115aed38f8f8cd248d832f51c27700efc25b201, which does not include the fix I was talking about on IRC (dgraph-io/badger#725). I've pushed an update to it now, so when you do go get github.com/ipfs/go-ds-badger@a17a6683864e61d11 in go-ipfs dir, and build that it should actually have that fix. (I'm also kind of surprised that upgrading in the first place helped anything)

@BenLubar Can you try running with that fix on GCed database?

BenLubar · 2019-03-20T18:36:20Z

The badgerds backup/restore versions:

$ du -sh badgerds-backup --exclude='*.vlog'
1.6G    badgerds-backup
$ du -sh badgerds-backup --exclude='*.sst'
79G     badgerds-backup
$ du -sh badgerds-restore --exclude='*.vlog'
1.3G    badgerds-restore
$ du -sh badgerds-restore --exclude='*.sst'
21G     badgerds-restore

I'll update this with the version from this PR once it finishes.

Edit: Here's what it looks like with the version from this PR:

$ du -sh badgerds --exclude='*.vlog'
1.6G    badgerds
$ du -sh badgerds --exclude='*.sst'
74G     badgerds

BenLubar · 2019-03-20T18:53:28Z

@magik6k Here's the output of badger info:

[2019-03-20T13:49:06-05:00] MANIFEST      13 kB MA
[       16 minutes earlier] 000255.sst    68 MB L1 bbf831b52c943721659fa79514fc60e27bb0399ed4bf81cb2fd72ca5888fe533
[       16 minutes earlier] 000256.sst    52 MB L1 b8233c9afaa6e0e47517330725ee8860bd5887f0c38d83713f367006f81b9505
[         1 second earlier] 000261.sst    70 MB L1 ebdb0dbb81be76664f30c3b36e651c7f6d10a31762290b3001185693bddeca25
[         1 second earlier] 000262.sst    70 MB L1 5673a244998212c6d43ef1e5b6e78e14778e1d8af7c3f00bece2b21cb11667f2
[         1 second earlier] 000263.sst    14 MB L1 b8eb91c90a98de6d5e90a4ea67e533cc140e70388f7dfc30cd1853191373f432
[       16 minutes earlier] 000047.sst   4.1 MB L2 5ae844d7ce5c8eb31f87f898c392af78408ba01b797f71c1ff29b938edc33ee4
[       16 minutes earlier] 000174.sst    26 MB L2 58c22b1fce81e378fc9e76c031269165f3bac621f3ceffd891cbfe152dba32c0
[       16 minutes earlier] 000210.sst    68 MB L2 98539e50c1c96dd39dd37652df6293962f3ce9b1fa21ddb658e91936a187dbf8
[       16 minutes earlier] 000212.sst    70 MB L2 e80c287fd50f4ad8c1bdb72d183ec583290a509f3d26d738df48d51ece7c4a1e
[       16 minutes earlier] 000215.sst    70 MB L2 c08efa5eca18ee7116a5c04843fe1fa40623f59b56d94ffa14871bd08b3b467d
[       16 minutes earlier] 000217.sst    70 MB L2 0a86aea7066ba893155466dfd74329438fae7804f35a8fb57d4dc2d7dc733e28
[       16 minutes earlier] 000219.sst    70 MB L2 4a8de7a753023f7da0dff9ac4384521a471514ab4e16e8d920509e899fa3446c
[       16 minutes earlier] 000220.sst    70 MB L2 b308203e418e067673b33627bbd5ed3893f003b955df5834e3c5f429ac9811f5
[       16 minutes earlier] 000221.sst    68 MB L2 23935b361980b40aba3ebf08bdef9bdecdc9c01220222f1804d044807a95aca5
[       16 minutes earlier] 000235.sst    70 MB L2 ad09e0d3786ffb8d3db34318178917b4e9b018aeef873e2f1af48b752a855151
[       16 minutes earlier] 000240.sst    70 MB L2 de9861ac7a14d2ad19896453a9ff3ed1d99f029ae8a272c569ba154a25719f42
[       16 minutes earlier] 000241.sst    68 MB L2 9de6cb105dfb29f9d3dbe257484f6dba5629f2d969cdbf80e8fa71ecd4f86909
[       16 minutes earlier] 000242.sst    70 MB L2 e9527e9934cb86e32541684eb6f0791d1ede7d226944f6304b761e583786effa
[       16 minutes earlier] 000243.sst    70 MB L2 c8d834754d148fc98e481954b4b44400cd63e4855449577320d76a165a4dd11d
[       16 minutes earlier] 000244.sst    70 MB L2 9db3282316de38a93233ec014907756b2c7512ded14a4b71ec26109467d0b4fb
[       16 minutes earlier] 000245.sst    70 MB L2 a8f0216ca7988a23d88efdb180b1f000eecfdb64310e8ea42456bb5442ba8004
[       16 minutes earlier] 000246.sst    70 MB L2 88ea9b149d4bdd38b71bea221a7e2e857408e505628579c4adba138df960feef
[       16 minutes earlier] 000247.sst    70 MB L2 ac82bb798705a66f3a3d0ef36805b5479852ad4ab894a476ff9d98fdb2f2e2ae
[       16 minutes earlier] 000248.sst    28 MB L2 b10f373c037b2fd2e24036ede50ed24c35ae54659931c432edb582abd6c2bf52
[       16 minutes earlier] 000257.sst    70 MB L2 4ac1be8b891dbb09b850ae149fcebc1e9803a39c70cf1fed9ab03c149d4dd338
[       16 minutes earlier] 000258.sst    68 MB L2 ec26fdb6d35944310a8bc2e79893ea77b5f6d1f7e7ff6d2b29f57264516ecc24
[       16 minutes earlier] 000259.sst    57 MB L2 1e7434f09e8f1b5139aa69c78d77fa89008444f75f49c7aca2c8a7739935fb9c
[       16 minutes earlier] 000001.vlog  1.1 GB VL
[       16 minutes earlier] 000003.vlog  1.1 GB VL
[       16 minutes earlier] 000004.vlog  1.1 GB VL
[       16 minutes earlier] 000005.vlog  1.1 GB VL
[       16 minutes earlier] 000006.vlog  1.1 GB VL
[       16 minutes earlier] 000007.vlog  1.1 GB VL
[       16 minutes earlier] 000009.vlog  1.1 GB VL
[       16 minutes earlier] 000010.vlog  1.1 GB VL
[       16 minutes earlier] 000011.vlog  1.1 GB VL
[       16 minutes earlier] 000012.vlog  1.1 GB VL
[       16 minutes earlier] 000013.vlog  1.1 GB VL
[       16 minutes earlier] 000015.vlog  1.1 GB VL
[       16 minutes earlier] 000016.vlog  1.1 GB VL
[       16 minutes earlier] 000017.vlog  1.1 GB VL
[       16 minutes earlier] 000018.vlog  1.1 GB VL
[       16 minutes earlier] 000019.vlog  1.1 GB VL
[       16 minutes earlier] 000020.vlog  1.1 GB VL
[       16 minutes earlier] 000021.vlog  1.0 GB VL
[       16 minutes earlier] 000023.vlog  1.1 GB VL
[       16 minutes earlier] 000024.vlog  302 MB VL
[       16 minutes earlier] 000025.vlog  1.1 GB VL
[       16 minutes earlier] 000027.vlog  934 MB VL
[       16 minutes earlier] 000028.vlog  1.1 GB VL
[       16 minutes earlier] 000029.vlog  1.1 GB VL
[       16 minutes earlier] 000031.vlog  1.1 GB VL
[       16 minutes earlier] 000032.vlog  1.1 GB VL
[       16 minutes earlier] 000033.vlog  554 MB VL
[       16 minutes earlier] 000034.vlog  181 MB VL
[       16 minutes earlier] 000035.vlog  953 MB VL
[       16 minutes earlier] 000036.vlog   71 MB VL
[       16 minutes earlier] 000037.vlog   71 MB VL
[       16 minutes earlier] 000038.vlog   71 MB VL
[       16 minutes earlier] 000039.vlog   71 MB VL
[       16 minutes earlier] 000040.vlog   72 MB VL
[       16 minutes earlier] 000041.vlog  1.1 GB VL
[       16 minutes earlier] 000042.vlog  1.1 GB VL
[       16 minutes earlier] 000043.vlog  1.1 GB VL
[       16 minutes earlier] 000044.vlog  1.1 GB VL
[       16 minutes earlier] 000045.vlog  1.1 GB VL
[       16 minutes earlier] 000046.vlog  1.1 GB VL
[       16 minutes earlier] 000047.vlog  1.1 GB VL
[       16 minutes earlier] 000048.vlog  1.1 GB VL
[       16 minutes earlier] 000049.vlog  1.1 GB VL
[       16 minutes earlier] 000050.vlog  1.1 GB VL
[       16 minutes earlier] 000051.vlog  1.1 GB VL
[       16 minutes earlier] 000052.vlog  1.1 GB VL
[       16 minutes earlier] 000053.vlog  1.1 GB VL
[       16 minutes earlier] 000054.vlog  1.1 GB VL
[       16 minutes earlier] 000055.vlog  1.1 GB VL
[       16 minutes earlier] 000056.vlog  1.1 GB VL
[       16 minutes earlier] 000057.vlog  1.1 GB VL
[       16 minutes earlier] 000058.vlog  1.1 GB VL
[       16 minutes earlier] 000059.vlog  1.1 GB VL
[       16 minutes earlier] 000060.vlog  1.1 GB VL
[       16 minutes earlier] 000062.vlog  1.1 GB VL
[       16 minutes earlier] 000063.vlog  299 MB VL
[       16 minutes earlier] 000064.vlog  1.1 GB VL
[       16 minutes earlier] 000065.vlog  1.1 GB VL
[       16 minutes earlier] 000066.vlog  1.1 GB VL
[       16 minutes earlier] 000067.vlog  1.1 GB VL
[       16 minutes earlier] 000068.vlog  1.1 GB VL
[       16 minutes earlier] 000069.vlog  1.1 GB VL
[       16 minutes earlier] 000070.vlog  1.1 GB VL
[       16 minutes earlier] 000071.vlog  1.1 GB VL
[       16 minutes earlier] 000072.vlog  1.1 GB VL
[       16 minutes earlier] 000073.vlog  1.1 GB VL
[       16 minutes earlier] 000074.vlog  1.1 GB VL
[       16 minutes earlier] 000075.vlog  1.1 GB VL
[       16 minutes earlier] 000076.vlog  1.1 GB VL
[       16 minutes earlier] 000077.vlog  1.1 GB VL
[       16 minutes earlier] 000078.vlog  1.1 GB VL
[       16 minutes earlier] 000079.vlog  1.1 GB VL
[       16 minutes earlier] 000080.vlog  1.1 GB VL
[       16 minutes earlier] 000081.vlog  1.1 GB VL
[       16 minutes earlier] 000082.vlog  1.1 GB VL
[       16 minutes earlier] 000083.vlog  1.1 GB VL
[       16 minutes earlier] 000084.vlog  936 MB VL
[       16 minutes earlier] 000085.vlog   78 MB VL
[       16 minutes earlier] 000087.vlog   84 MB VL
[       16 minutes earlier] 000088.vlog   78 MB VL
[       16 minutes earlier] 000089.vlog   90 MB VL
[       16 minutes earlier] 000090.vlog   81 MB VL
[       16 minutes earlier] 000091.vlog   78 MB VL
[       16 minutes earlier] 000092.vlog   74 MB VL
[       16 minutes earlier] 000093.vlog   77 MB VL
[       16 minutes earlier] 000094.vlog   76 MB VL
[       16 minutes earlier] 000095.vlog   78 MB VL
[       16 minutes earlier] 000096.vlog   81 MB VL
[       16 minutes earlier] 000097.vlog  582 MB VL
[       16 minutes earlier] 000098.vlog  1.1 GB VL
[       16 minutes earlier] 000099.vlog  273 MB VL
[       16 minutes earlier] 000100.vlog   82 MB VL
[       16 minutes earlier] 000101.vlog   78 MB VL
[       16 minutes earlier] 000102.vlog   76 MB VL
[       16 minutes earlier] 000103.vlog   78 MB VL
[       16 minutes earlier] 000104.vlog  145 MB VL
[       16 minutes earlier] 000105.vlog   79 MB VL
[       16 minutes earlier] 000106.vlog   77 MB VL
[       16 minutes earlier] 000107.vlog   76 MB VL
[        2 seconds earlier] 000108.vlog  630 MB VL

[Summary]
Level 0 size:          0 B
Level 1 size:       274 MB
Level 2 size:       1.4 GB
Total index size:   1.6 GB
Value log size:      78 GB

Abnormalities: None.
0 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
badger 2019/03/20 13:50:38 INFO: 7 tables out of 27 opened in 4.877s
badger 2019/03/20 13:50:39 INFO: 18 tables out of 27 opened in 6.016s
badger 2019/03/20 13:50:43 INFO: 21 tables out of 27 opened in 9.703s
badger 2019/03/20 13:50:44 INFO: All 27 tables opened in 10.373s
badger 2019/03/20 13:50:44 INFO: Replaying file id: 108 at offset: 630389408
badger 2019/03/20 13:50:44 INFO: Replay took: 6.352µs

SSTable [Li, Id, Total Keys including internal keys] [Left Key, Version -> Right Key, Version]
SSTable [L1, 255, 0519344] [2F70726F7669646572732F43495142464B325554334C3634533458434E413459593741425356594B56454F33354B4D51444F324F334D52564D503644434E43354E592F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v24000285 -> 2F70726F7669646572732F4349514C5A4B5445474F46494B364B324C564E5447504A444C4849434E47555934514D434535584D4343533243514E42595249534D34512F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v16961366]
SSTable [L1, 256, 0404459] [2F70726F7669646572732F4349514C5A4B544D34534D4F4733373647513633585A4A56494E4D534651584E49323445525A52485051434F525748435153454C4A37412F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v24062623 -> 2F70726F7669646572732F434951505A5A5A3257585533445157365859443257424751504C57414B54544A4A3332453351595933524A59364A534A4F43324A364A412F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v26018807]
SSTable [L1, 261, 0889434] [216261646765722168656164, v29053317 -> 2F626C6F636B732F4349514B55594C504244574D36374E4544343459445151564A4F52444F544A434453505A4F484E34334533485354414257374347415449, v25609649]
SSTable [L1, 262, 0951559] [2F626C6F636B732F4349514B55594C5136544C37493558355A483532343550493237373635593653364836354D493347344A4633585A5258335732324D4241, v25609651 -> 2F626C6F636B732F434951504D3548443659454B5333344642574E34443452354948465156454F47324E4C574337455143375A504D414C4F45323355374F49, v28751212]
SSTable [L1, 263, 0195442] [2F626C6F636B732F434951504D354846365951544C575554454F594E563637525555494851435A474F3258574852505A464641414E5541494E323442544251, v28751214 -> 2F66696C6573746F72652F434951505A59515942504A5A323637554254435732575156463448433734374547454744504442595A3651534F5657435733344B324A41, v29053316]
SSTable [L2, 047, 0029529] [2F70726F7669646572732F4349514B4B49425145463352444B4B464346544D4F51524336354C3237334E564D4D3336585749375759414A4D354445465434374D58492F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v5228530 -> 2F70726F7669646572732F4349514D4B3651575050584645585042324F57444C5A5951514444564D595A4E4352534A3644425544374D58573448454B364450425A412F4349514F363245534957373557414249334C4C37344A48345A565843514E4957495950474C34454532575147483545373248374A355641, v4484892]
SSTable [L2, 174, 0195812] [2F70726F7669646572732F434951474F5335504B4135564935454944374A55564753504145373645354C4C4C4548594446473745334D58424C514E524332444D54512F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v16908891 -> 2F70726F7669646572732F4349514B4B49414F5249524B4A4B4C51594F33474653575459354147533547445047324B494D513436504F52473459494A41595A5245512F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v6173097]
SSTable [L2, 210, 0872522] [216261646765722168656164, v24481445 -> 2F626C6F636B732F434951414E454F4633513750354D354C5350574F53495141414E504D435053504336594E324C3749565842585248484856414844444549, v7038148]
SSTable [L2, 212, 0935653] [2F626C6F636B732F434951414E454F465848504C4D4953514C48595332575A4D475A48544C5745323757514D464733594D45474F5834513354583459325759, v11062675 -> 2F626C6F636B732F43495143474251523735434150484F33514533474F4251504F533536535835495853564D4D485435523553545A46444737594E56564351, v16071497]
SSTable [L2, 215, 0934471] [2F626C6F636B732F4349514347425156323447484D4B48484D363359504537574B4B374D4B4D4A494D42323659564354595636533743585753434451563549, v19074644 -> 2F626C6F636B732F4349514543354D4944484133485852325643374B344F434B4859465135455637325737444A5857445A4D32514F32444E4B364E35564C51, v7914761]
SSTable [L2, 217, 0935630] [2F626C6F636B732F4349514543354D4E59565A32494B43514E4359335355414F33424E5246354453564F34484D5347504D5A364D364C525844564F37524F59, v24378290 -> 2F626C6F636B732F434951473255574936344A41573741584759555547324A4C56344F5A344F50334A54595756373657494350424D5045375753424C364349, v21468692]
SSTable [L2, 219, 0936683] [2F626C6F636B732F43495147325557494457474C534D4E37474A4F4C494E533736363254594852415A484B4B43595054504B534835494A3558434D48585349, v21468694 -> 2F626C6F636B732F4349514851535A4D50524F434A553345495A49495A54423455435245584E445A49354B53504F4732355641495750553437575041454D49, v22597253]
SSTable [L2, 220, 0936212] [2F626C6F636B732F4349514851535A5132525241444946354C525837445251454E47505549474F524D4D3357354936425A473759564E45365A4A4B34473649, v22597255 -> 2F626C6F636B732F4349514A485447565A464E5A434C4D514E4858544947414C4C49375056423256364B414A505033475A5747414B375856444B4933425959, v23756590]
SSTable [L2, 221, 0855511] [2F626C6F636B732F4349514A48544757334E464B334A42425A46505A4749534933355537483251505435414C3534435234464541454F4B5559364E4F595459, v23756592 -> 2F626C6F636B732F4349514C375533595A42544B445A4247434C5654354D5A494159414432344633414B4E48534C495955544B57524253554E37454E505141, v9480377]
SSTable [L2, 235, 0894996] [2F626C6F636B732F4349514C375534325A474451373736355A33514E444635533251475835375046534D59324733483456434D524837485A51595447334F59, v24224344 -> 2F626C6F636B732F4349514E33463232334D37575451584E513744474436374F554359485637345456554B544A5A4A32414251434C5244374A4F4E57564E51, v11885817]
SSTable [L2, 240, 0884650] [2F626C6F636B732F4349514E3346323245494542434A47514159504842554F49543537544344504B3541574D4C35333543443259364D565259475444595A41, v27112047 -> 2F626C6F636B732F4349514F413550585134323641444E464942424C4232475551343349544D3549473250534B583335544A424B504B455332494A36324241, v16685041]
SSTable [L2, 241, 0854797] [2F626C6F636B732F4349514F413550594837413337423233524756454D525235374A5A3345574F49334F53565942365644564C4B5334484B4D524B4C564A51, v14341102 -> 2F626C6F636B732F4349515053494347485649544349435657584F5A4F4836583743515636584E4B415750364E49523336475647554659494D345158583751, v10516349]
SSTable [L2, 242, 0934959] [2F626C6F636B732F43495150534943493335324B47594E504B4549445033434D3633324B46525235534D364D4F34374D42465335464A4435354D514C564241, v13493808 -> 2F66696C6573746F72652F434951423432555A534D4D454B464A464137434945555334345548374235553259564E374936585337494A5347324455434D5A45524D41, v18227370]
SSTable [L2, 243, 0953499] [2F66696C6573746F72652F4349514234325633504F51374A37434A424C3653445846434456524943324C583749494D425A4654453537584D5454373454494D435959, v7144735 -> 2F66696C6573746F72652F434951443558475946484A4332554C354332514132473352474F5246344141574A33354A594355464C4F37374C4D365834354132555A51, v4629872]
SSTable [L2, 244, 0953505] [2F66696C6573746F72652F4349514435584833525643354136454F414A464237343455483736373249505A35533647514D445044574F47464545433256564E445941, v19555352 -> 2F66696C6573746F72652F434951463756334752344F584344444A474C4E4E5241324C3733494A52463241514243345457484D4444495843414334374252324F5141, v20914445]
SSTable [L2, 245, 0953504] [2F66696C6573746F72652F434951463756334A34595147504450555234324A5843354D524E3332374F56504F46495033455752594E4A5848374846484F375A475741, v20914447 -> 2F66696C6573746F72652F4349514842544A533559363253444752564E5145415137594550594847374155484632584A50504F37554F43424E4B59415048435A5149, v22283618]
SSTable [L2, 246, 0953542] [2F66696C6573746F72652F4349514842544A5434423645534C365456334451435653454655584F4F453358454A4C37434A594D59433648325334565A5056595A5249, v22283620 -> 2F66696C6573746F72652F4349514A445250354444354A575A37535A4656375A345035594A56434F5342454342343552485458534646505355584E444A534D374151, v23671205]
SSTable [L2, 247, 0953549] [2F66696C6573746F72652F4349514A44525042474854325033524B4B5142515351494E493635434F445445585535554B575A4E424C42495A5841484D4C4257414559, v23671208 -> 2F66696C6573746F72652F4349514C465045484B4C5236514A325144335756494E4F5752585A524D353536573259503655533753514E4657354436334255504E5559, v25966191]
SSTable [L2, 248, 0392433] [2F66696C6573746F72652F4349514C465045494E435144595135344A5245554A545259474632484654334D56374C49354F4E465932595432425159425A4735534B59, v9522369 -> 2F66696C6573746F72652F4349514D41524F4555345A5452365347485249545050334736325A42493233484F365234554D4F495254464F51373233375550534D4859, v26520620]
SSTable [L2, 257, 0953457] [2F66696C6573746F72652F4349514D41524F4B414C4E464C374B334C4549594C425336434756514A464B4449485652494A3434474535553249594357564B46375451, v26520622 -> 2F66696C6573746F72652F4349514F473353554A58534D345A545942444E5249575950594A544F574D4D4658324953524F464A4D36374E42534D3348553452565951, v27976726]
SSTable [L2, 258, 0862506] [2F66696C6573746F72652F4349514F47335356334F4E5535574D5454424B4746525A435645594F4D445A414B414B344543374D57425A5945524F3258334C57443759, v27976728 -> 2F70726F7669646572732F41464B52454948584A474F58494647424557454235344159564558484A435444525355484A4E344E4256444233454E364E34323759584A544A592F4349514E36454D415336345A435952444C354943464350594A5A35554A55464F5952363542555A47544D3351583336425153584F504A49, v24490798]
SSTable [L2, 259, 0429492] [2F70726F7669646572732F41464B52454948584A4F375A374C4251344B454C4C4D375A465845444C504552484748553743534F4D57515947353754374F355050324F354D592F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v11212808 -> 2F70726F7669646572732F434951474F53354A5344334951514F345134325A335A554F42534B504F485243565650424F54495441474137424E3241585153504A54512F43495143364E4458563548434834415A5A4B53455A47373456354B345451594F505A34544F4D4A585354493636464436504741334F4751, v13036000]

Histogram of key sizes (in bytes)
Total count: 903287
Min value: 11
Max value: 176
Mean: 73.36
                   Range     Count
[         8,         16)         1
[        16,         32)         1
[        32,         64)    744609
[        64,        128)    158640
[       128,        256)        36

Histogram of value sizes (in bytes)
Total count: 903287
Min value: 0
Max value: 262170
Mean: 21964.61
                   Range     Count
[         0,          2)         1
[         4,          8)         2
[         8,         16)    158388
[        16,         32)         2
[        32,         64)        18
[        64,        128)    417972
[       128,        256)     78316
[       256,        512)     15002
[       512,       1024)      2163
[      1024,       2048)      2547
[      2048,       4096)      6081
[      4096,       8192)     12456
[      8192,      16384)     31250
[     16384,      32768)     53034
[     32768,      65536)     42414
[     65536,     131072)     25320
[    131072,     262144)     20964
[    262144,     524288)     37357

badger 2019/03/20 13:52:36 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]}

magik6k · 2019-03-20T19:55:16Z

cc @manishrjain

raulk · 2019-07-04T16:11:55Z

Ideally we'd pass a context to this. IIRC badger GC is stop-the-world, so if it runs too long it can easily starve the rest of the system (insofar datastore access is concerned). It's kinda dangerous.

manishrjain · 2019-07-05T00:34:02Z

Do you guys still need help with this? IIRC, some changes were made to Badger to improve GC.

Stebalien · 2019-08-21T17:28:14Z

@magik6k can we close this?

magik6k · 2019-08-23T11:06:40Z

I think we still want this (but possibly with a context as raul pointed out) - without this we'll only gc at most 1gb - https://github.com/dgraph-io/badger#garbage-collection

aarshkshah1992 · 2019-10-08T05:16:22Z

Changes in this PR & the suggestions in the comments have now been incorporated in #72.

Make sure to GC as much as possible

4fb18b4

ghost assigned magik6k Mar 19, 2019

ghost added the status/in-progress In progress label Mar 19, 2019

Update badger to latest master

a17a668

Stebalien mentioned this pull request Jul 4, 2019

Periodically garbage collect #51

Closed

aarshkshah1992 mentioned this pull request Oct 7, 2019

periodic GC for badger datastore #72

Merged

Stebalien closed this Oct 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make sure to GC as much as possible #56

Make sure to GC as much as possible #56

magik6k commented Mar 19, 2019

Stebalien commented Mar 20, 2019

BenLubar commented Mar 20, 2019

hsanjuan commented Mar 20, 2019

magik6k commented Mar 20, 2019

BenLubar commented Mar 20, 2019 •

edited

Loading

BenLubar commented Mar 20, 2019

magik6k commented Mar 20, 2019

raulk commented Jul 4, 2019 •

edited

Loading

manishrjain commented Jul 5, 2019 •

edited

Loading

Stebalien commented Aug 21, 2019

magik6k commented Aug 23, 2019

aarshkshah1992 commented Oct 8, 2019

Make sure to GC as much as possible #56

Make sure to GC as much as possible #56

Conversation

magik6k commented Mar 19, 2019

Stebalien commented Mar 20, 2019

BenLubar commented Mar 20, 2019

hsanjuan commented Mar 20, 2019

magik6k commented Mar 20, 2019

BenLubar commented Mar 20, 2019 • edited Loading

BenLubar commented Mar 20, 2019

magik6k commented Mar 20, 2019

raulk commented Jul 4, 2019 • edited Loading

manishrjain commented Jul 5, 2019 • edited Loading

Stebalien commented Aug 21, 2019

magik6k commented Aug 23, 2019

aarshkshah1992 commented Oct 8, 2019

BenLubar commented Mar 20, 2019 •

edited

Loading

raulk commented Jul 4, 2019 •

edited

Loading

manishrjain commented Jul 5, 2019 •

edited

Loading