correct and speed up compaction #10867

ivanmorozov333 · 2024-10-24T18:33:48Z

Improvement
Performance improvement
Bugfix

github-actions · 2024-10-24T18:37:38Z

⚪ 2024-10-24 18:37:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for ec27930 has started.
⚪ 2024-10-24 18:37:48 UTC Artifacts will be uploaded here
⚪ 2024-10-24 18:40:45 UTC ya make is running...
🔴 2024-10-24 18:43:45 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-10-24T18:37:42Z

⚪ 2024-10-24 18:37:41 UTC Pre-commit check linux-x86_64-release-asan for ec27930 has started.
⚪ 2024-10-24 18:37:52 UTC Artifacts will be uploaded here
⚪ 2024-10-24 18:40:50 UTC ya make is running...
🔴 2024-10-24 18:44:29 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-10-25T05:35:30Z

⚪ 2024-10-25 05:35:30 UTC Pre-commit check linux-x86_64-release-asan for a1f054f has started.
⚪ 2024-10-25 05:35:42 UTC Artifacts will be uploaded here
⚪ 2024-10-25 05:38:46 UTC ya make is running...
⚫ 2024-10-25 06:46:34 UTC Check cancelled

github-actions · 2024-10-25T05:37:53Z

⚪ 2024-10-25 05:37:52 UTC Pre-commit check linux-x86_64-relwithdebinfo for a1f054f has started.
⚪ 2024-10-25 05:38:03 UTC Artifacts will be uploaded here
⚪ 2024-10-25 05:41:01 UTC ya make is running...
🟡 2024-10-25 06:26:49 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15020	13340	0	141	1442	97

⚪ 2024-10-25 06:28:05 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-25 06:39:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
288 (only retried tests)	44	0	136	14	94

⚪ 2024-10-25 06:40:07 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-10-25 06:46:49 UTC ydbd size 2.8 GiB changed* by -20.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `d5caec3`	merge: `a1f054f`	diff	diff %
ydbd size	3 032 836 080 Bytes	3 032 815 080 Bytes	-20.5 KiB	-0.001%
ydbd stripped size	480 987 256 Bytes	480 983 544 Bytes	-3.6 KiB	-0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2024-10-25 06:46:51 UTC Check cancelled

github-actions · 2024-10-25T06:48:57Z

⚪ 2024-10-25 06:48:56 UTC Pre-commit check linux-x86_64-release-asan for eca744c has started.
⚪ 2024-10-25 06:49:00 UTC Artifacts will be uploaded here
⚪ 2024-10-25 06:51:56 UTC ya make is running...
🟡 2024-10-25 08:01:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10068	9501	0	192	335	40

🟢 2024-10-25 08:02:41 UTC Build successful.
🟢 2024-10-25 08:03:26 UTC ydbd size 5.7 GiB changed* by +36.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `d5caec3`	merge: `eca744c`	diff	diff %
ydbd size	6 136 966 424 Bytes	6 137 003 344 Bytes	+36.1 KiB	+0.001%
ydbd stripped size	1 531 956 848 Bytes	1 531 972 400 Bytes	+15.2 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-25T06:49:27Z

⚪ 2024-10-25 06:49:26 UTC Pre-commit check linux-x86_64-relwithdebinfo for eca744c has started.
⚪ 2024-10-25 06:49:30 UTC Artifacts will be uploaded here
⚪ 2024-10-25 06:52:29 UTC ya make is running...
🟡 2024-10-25 07:37:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15022	13332	0	152	1440	98

⚪ 2024-10-25 07:38:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-25 07:50:34 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
297 (only retried tests)	37	0	144	22	94

⚪ 2024-10-25 07:50:44 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-25 08:02:33 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
259 (only retried tests)	16	0	142	8	93

🟢 2024-10-25 08:02:40 UTC Build successful.
🟢 2024-10-25 08:03:04 UTC ydbd size 2.8 GiB changed* by +30.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `d5caec3`	merge: `eca744c`	diff	diff %
ydbd size	3 032 836 080 Bytes	3 032 867 120 Bytes	+30.3 KiB	+0.001%
ydbd stripped size	480 987 256 Bytes	480 995 672 Bytes	+8.2 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-25T10:47:37Z

⚪ 2024-10-25 10:47:36 UTC Pre-commit check linux-x86_64-release-asan for 64ed24d has started.
⚪ 2024-10-25 10:48:04 UTC Artifacts will be uploaded here
⚪ 2024-10-25 10:51:36 UTC ya make is running...
🟡 2024-10-25 11:55:53 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10063	9662	0	197	104	100

🟢 2024-10-25 11:56:52 UTC Build successful.
🟢 2024-10-25 11:57:29 UTC ydbd size 5.7 GiB changed* by -42.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `86223ca`	merge: `64ed24d`	diff	diff %
ydbd size	6 136 872 480 Bytes	6 136 829 088 Bytes	-42.4 KiB	-0.001%
ydbd stripped size	1 531 420 816 Bytes	1 531 413 328 Bytes	-7.3 KiB	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-25T10:49:18Z

⚪ 2024-10-25 10:49:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for 64ed24d has started.
⚪ 2024-10-25 10:49:27 UTC Artifacts will be uploaded here
⚪ 2024-10-25 10:52:27 UTC ya make is running...
🟡 2024-10-25 11:38:02 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15018	13300	0	153	1464	101

⚪ 2024-10-25 11:39:24 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-25 11:51:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
325 (only retried tests)	37	0	142	51	95

⚪ 2024-10-25 11:51:32 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-25 12:03:11 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
287 (only retried tests)	33	0	142	18	94

🟢 2024-10-25 12:03:18 UTC Build successful.
🟢 2024-10-25 12:03:44 UTC ydbd size 2.8 GiB changed* by -20.1 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `86223ca`	merge: `64ed24d`	diff	diff %
ydbd size	3 032 571 856 Bytes	3 032 551 272 Bytes	-20.1 KiB	-0.001%
ydbd stripped size	480 494 808 Bytes	480 491 160 Bytes	-3.6 KiB	-0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-25T17:17:06Z

⚪ 2024-10-25 17:17:06 UTC Pre-commit check linux-x86_64-release-asan for c7ff4e7 has started.
⚪ 2024-10-25 17:17:17 UTC Artifacts will be uploaded here
⚪ 2024-10-25 17:20:26 UTC ya make is running...
🟡 2024-10-25 18:35:46 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10036	9581	0	228	125	102

🟢 2024-10-25 18:36:39 UTC Build successful.
🟢 2024-10-25 18:37:12 UTC ydbd size 5.7 GiB changed* by -29.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `3a89b73`	merge: `c7ff4e7`	diff	diff %
ydbd size	6 138 191 136 Bytes	6 138 160 928 Bytes	-29.5 KiB	-0.000%
ydbd stripped size	1 531 728 816 Bytes	1 531 724 464 Bytes	-4.2 KiB	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-25T17:17:12Z

⚪ 2024-10-25 17:17:11 UTC Pre-commit check linux-x86_64-relwithdebinfo for c7ff4e7 has started.
⚪ 2024-10-25 17:17:46 UTC Artifacts will be uploaded here
⚪ 2024-10-25 17:21:19 UTC ya make is running...
🟡 2024-10-25 18:06:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15023	13321	0	150	1451	101

⚪ 2024-10-25 18:07:23 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-25 18:19:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
309 (only retried tests)	48	0	142	24	95

⚪ 2024-10-25 18:19:23 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-25 18:31:11 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
261 (only retried tests)	17	0	142	8	94

🟢 2024-10-25 18:31:18 UTC Build successful.
🟢 2024-10-25 18:31:43 UTC ydbd size 2.8 GiB changed* by -12.7 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `3a89b73`	merge: `c7ff4e7`	diff	diff %
ydbd size	3 033 039 936 Bytes	3 033 026 952 Bytes	-12.7 KiB	-0.000%
ydbd stripped size	480 534 200 Bytes	480 531 576 Bytes	-2.6 KiB	-0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-26T12:49:35Z

⚪ 2024-10-26 12:49:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for a9953b0 has started.
⚪ 2024-10-26 12:50:16 UTC Artifacts will be uploaded here
⚪ 2024-10-26 12:53:32 UTC ya make is running...
⚫ 2024-10-26 13:20:10 UTC Check cancelled

github-actions · 2024-10-26T12:51:53Z

⚪ 2024-10-26 12:51:53 UTC Pre-commit check linux-x86_64-release-asan for a9953b0 has started.
⚪ 2024-10-26 12:52:04 UTC Artifacts will be uploaded here
⚪ 2024-10-26 12:55:12 UTC ya make is running...
⚫ 2024-10-26 13:20:11 UTC Check cancelled

github-actions · 2024-10-26T13:22:41Z

⚪ 2024-10-26 13:22:41 UTC Pre-commit check linux-x86_64-release-asan for 09ff924 has started.
⚪ 2024-10-26 13:23:23 UTC Artifacts will be uploaded here
⚫ 2024-10-26 13:24:23 UTC Check cancelled

github-actions · 2024-10-26T13:25:47Z

⚪ 2024-10-26 13:25:47 UTC Pre-commit check linux-x86_64-release-asan for 7de8468 has started.
⚪ 2024-10-26 13:25:51 UTC Artifacts will be uploaded here
⚪ 2024-10-26 13:28:56 UTC ya make is running...
🟡 2024-10-26 14:36:32 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10051	9876	0	46	31	98

🟢 2024-10-26 14:37:25 UTC Build successful.
🟢 2024-10-26 14:37:59 UTC ydbd size 5.7 GiB changed* by -29.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `67d74e2`	merge: `7de8468`	diff	diff %
ydbd size	6 138 292 144 Bytes	6 138 261 912 Bytes	-29.5 KiB	-0.000%
ydbd stripped size	1 531 755 696 Bytes	1 531 751 408 Bytes	-4.2 KiB	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-26T13:28:24Z

⚪ 2024-10-26 13:28:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7de8468 has started.
⚪ 2024-10-26 13:28:35 UTC Artifacts will be uploaded here
⚪ 2024-10-26 13:31:42 UTC ya make is running...
🟡 2024-10-26 14:19:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15022	13528	0	2	1394	98

⚪ 2024-10-26 14:20:34 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-26 14:35:06 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
99 (only retried tests)	6	0	0	0	93

🟢 2024-10-26 14:35:13 UTC Build successful.
🟢 2024-10-26 14:35:33 UTC ydbd size 2.8 GiB changed* by -8.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `67d74e2`	merge: `7de8468`	diff	diff %
ydbd size	3 033 086 168 Bytes	3 033 077 168 Bytes	-8.8 KiB	-0.000%
ydbd stripped size	480 539 160 Bytes	480 540 632 Bytes	+1.4 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

fexolm · 2024-10-28T13:20:43Z

ydb/core/tx/columnshard/engines/storage/optimizer/lcbuckets/planner/optimizer.cpp

-        result->AddMovePortions(data.GetMovePortions());
-    }
+//    } else {
+//        result = std::make_shared<NCompaction::TGeneralCompactColumnEngineChanges>(


Do we still need that commented code?

fexolm · 2024-10-28T13:21:54Z

ydb/core/tx/columnshard/engines/changes/general_compaction.cpp

@@ -237,26 +237,35 @@ std::shared_ptr<TGeneralCompactColumnEngineChanges::IMemoryPredictor> TGeneralCo
 }

 ui64 TGeneralCompactColumnEngineChanges::TMemoryPredictorChunkedPolicy::AddPortion(const TPortionInfo& portionInfo) {
-    SumMemoryFix += portionInfo.GetRecordsCount() * (2 * sizeof(ui64) + sizeof(ui32) + sizeof(ui16));
+    SumMemoryFix += portionInfo.GetRecordsCount() * (2 * sizeof(ui64) + sizeof(ui32) + sizeof(ui16)) + portionInfo.GetTotalBlobBytes();


Maybe we could use sizeof(SomeStruct) here? Just wondering...

github-actions bot added the improvement label Oct 24, 2024

ivanmorozov333 added 5 commits October 26, 2024 14:59

correct and speed up compaction

bc55b4f

fix

76ca6e9

first/last only for PK

002edc6

fix

0fc917e

reduce mem

6b44365

ivanmorozov333 force-pushed the correct_compaction branch from c98a9a2 to 6b44365 Compare October 26, 2024 12:47

fix validation

8390871

fix

42ba828

ivanmorozov333 marked this pull request as ready for review October 27, 2024 15:07

ivanmorozov333 requested a review from a team as a code owner October 27, 2024 15:07

zverevgeny assigned fexolm Oct 28, 2024

fexolm approved these changes Oct 28, 2024

View reviewed changes

fexolm reviewed Oct 28, 2024

View reviewed changes

ivanmorozov333 merged commit d4eaec4 into ydb-platform:main Oct 28, 2024
10 checks passed

ivanmorozov333 deleted the correct_compaction branch October 28, 2024 14:19

shnikd mentioned this pull request Nov 7, 2024

Support CollectDiagnostics in DATA_QUERY/GENERIC_QUERY #11371

Open

niksaveliev mentioned this pull request Nov 8, 2024

Fix read session correct close ut #11407

Merged

GrigoriyPA mentioned this pull request Nov 13, 2024

YQ WM fixed race with actor context #11576

Merged

niksaveliev mentioned this pull request Nov 14, 2024

Commit for autopartitioned topics #11629

Open

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 2, 2025

correct and speed up compaction (ydb-platform#10867)

5690fc0

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 5, 2025

correct and speed up compaction (ydb-platform#10867)

ce816b0

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 5, 2025

correct and speed up compaction (ydb-platform#10867)

f3b0f5d

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 8, 2025

correct and speed up compaction (ydb-platform#10867)

77ed5a8

zverevgeny mentioned this pull request Jan 9, 2025

Integrate changes from main #13202

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

correct and speed up compaction #10867

correct and speed up compaction #10867

ivanmorozov333 commented Oct 24, 2024

github-actions bot commented Oct 24, 2024 •

edited

Loading

github-actions bot commented Oct 24, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

fexolm Oct 28, 2024

fexolm Oct 28, 2024

correct and speed up compaction #10867

correct and speed up compaction #10867

Conversation

ivanmorozov333 commented Oct 24, 2024

github-actions bot commented Oct 24, 2024 • edited Loading

github-actions bot commented Oct 24, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 25, 2024 • edited Loading

github-actions bot commented Oct 26, 2024 • edited Loading

github-actions bot commented Oct 26, 2024 • edited Loading

github-actions bot commented Oct 26, 2024 • edited Loading

github-actions bot commented Oct 26, 2024 • edited Loading

github-actions bot commented Oct 26, 2024 • edited Loading

fexolm Oct 28, 2024

Choose a reason for hiding this comment

fexolm Oct 28, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 24, 2024 •

edited

Loading

github-actions bot commented Oct 24, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading

github-actions bot commented Oct 26, 2024 •

edited

Loading