YQ-3926 RD added parallel purecalc compilation #12505

GrigoriyPA · 2024-12-11T11:24:38Z

Changelog entry

added parallel purecalc compilation and abort requests

Changelog category

Performance improvement

Additional information

github-actions · 2024-12-11T11:28:28Z

⚪ 2024-12-11 11:28:28 UTC Pre-commit check linux-x86_64-relwithdebinfo for eb4fdb2 has started.
⚪ 2024-12-11 11:28:39 UTC Artifacts will be uploaded here
⚫ 2024-12-11 11:29:07 UTC Check cancelled

github-actions · 2024-12-11T11:30:38Z

⚪ 2024-12-11 11:30:37 UTC Pre-commit check linux-x86_64-release-asan for c4b44d9 has started.
⚪ 2024-12-11 11:30:48 UTC Artifacts will be uploaded here
⚪ 2024-12-11 11:34:05 UTC ya make is running...
🟡 2024-12-11 13:08:21 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15294	15223	0	22	5	44

🟢 2024-12-11 13:09:35 UTC Build successful.
🟡 2024-12-11 13:09:58 UTC ydbd size 3.6 GiB changed* by +146.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `95a821a`	merge: `c4b44d9`	diff	diff %
ydbd size	3 897 699 816 Bytes	3 897 849 648 Bytes	+146.3 KiB	+0.004%
ydbd stripped size	1 366 522 384 Bytes	1 366 578 480 Bytes	+54.8 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-11T11:31:35Z

⚪ 2024-12-11 11:31:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for c4b44d9 has started.
⚪ 2024-12-11 11:31:46 UTC Artifacts will be uploaded here
⚪ 2024-12-11 11:35:00 UTC ya make is running...
🟡 2024-12-11 14:08:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
32790	29870	0	2	2810	108

⚪ 2024-12-11 14:11:31 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-11 14:23:06 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
632 (only retried tests)	524	0	1	1	106

⚪ 2024-12-11 14:23:22 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-11 14:34:40 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
627 (only retried tests)	520	0	1	0	106

🟢 2024-12-11 14:34:49 UTC Build successful.
🟢 2024-12-11 14:35:08 UTC ydbd size 2.1 GiB changed* by +81.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `95a821a`	merge: `c4b44d9`	diff	diff %
ydbd size	2 255 721 680 Bytes	2 255 804 888 Bytes	+81.3 KiB	+0.004%
ydbd stripped size	483 873 296 Bytes	483 893 616 Bytes	+19.8 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-18T10:12:45Z

⚪ 2024-12-18 10:12:44 UTC Pre-commit check linux-x86_64-release-asan for e3692a0 has started.
⚪ 2024-12-18 10:13:18 UTC Artifacts will be uploaded here
⚪ 2024-12-18 10:17:06 UTC ya make is running...
🟡 2024-12-18 11:57:28 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14020	13885	0	69	16	50

🟢 2024-12-18 11:58:41 UTC Build successful.
🟡 2024-12-18 11:59:10 UTC ydbd size 3.6 GiB changed* by +146.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `8744fb5`	merge: `e3692a0`	diff	diff %
ydbd size	3 894 517 928 Bytes	3 894 667 744 Bytes	+146.3 KiB	+0.004%
ydbd stripped size	1 363 834 448 Bytes	1 363 890 544 Bytes	+54.8 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-18T10:13:41Z

⚪ 2024-12-18 10:13:41 UTC Pre-commit check linux-x86_64-relwithdebinfo for e3692a0 has started.
⚪ 2024-12-18 10:13:52 UTC Artifacts will be uploaded here
⚪ 2024-12-18 10:17:20 UTC ya make is running...
🟡 2024-12-18 11:53:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31019	28066	0	2	2832	119

⚪ 2024-12-18 11:56:18 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-18 12:08:12 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
168 (only retried tests)	59	0	0	0	109

🟢 2024-12-18 12:08:19 UTC Build successful.
🟢 2024-12-18 12:08:44 UTC ydbd size 2.1 GiB changed* by +77.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `8744fb5`	merge: `e3692a0`	diff	diff %
ydbd size	2 253 716 864 Bytes	2 253 795 984 Bytes	+77.3 KiB	+0.004%
ydbd stripped size	483 049 840 Bytes	483 066 064 Bytes	+15.8 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-23T08:05:39Z

⚪ 2024-12-23 08:05:38 UTC Pre-commit check linux-x86_64-relwithdebinfo for b61d2b4 has started.
⚪ 2024-12-23 08:05:49 UTC Artifacts will be uploaded here
⚪ 2024-12-23 08:09:02 UTC ya make is running...
🟡 2024-12-23 09:36:44 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31067	28118	0	4	2831	114

⚪ 2024-12-23 09:39:16 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-23 09:51:13 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
162 (only retried tests)	53	0	0	0	109

🟢 2024-12-23 09:51:20 UTC Build successful.
🟢 2024-12-23 09:51:38 UTC ydbd size 2.1 GiB changed* by +74.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `ac47d76`	merge: `b61d2b4`	diff	diff %
ydbd size	2 234 720 224 Bytes	2 234 796 808 Bytes	+74.8 KiB	+0.003%
ydbd stripped size	477 909 392 Bytes	477 929 072 Bytes	+19.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-23T08:05:58Z

⚪ 2024-12-23 08:05:57 UTC Pre-commit check linux-x86_64-release-asan for b61d2b4 has started.
⚪ 2024-12-23 08:06:10 UTC Artifacts will be uploaded here
⚪ 2024-12-23 08:09:36 UTC ya make is running...
🟡 2024-12-23 09:42:50 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14044	13974	0	19	5	46

🟢 2024-12-23 09:43:59 UTC Build successful.
🟡 2024-12-23 09:44:22 UTC ydbd size 3.6 GiB changed* by +141.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ac47d76`	merge: `b61d2b4`	diff	diff %
ydbd size	3 864 972 872 Bytes	3 865 117 664 Bytes	+141.4 KiB	+0.004%
ydbd stripped size	1 350 438 672 Bytes	1 350 497 488 Bytes	+57.4 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

dorooleg · 2024-12-23T15:31:14Z

ydb/core/fq/libs/row_dispatcher/purecalc_compilation/compile_service.cpp

@@ -41,15 +65,122 @@ class TPurecalcCompileService : public NActors::TActor<TPurecalcCompileService>
        }

        if (status.IsFail()) {
-            LOG_ROW_DISPATCHER_ERROR("Compilation failed for request with id: " << ev->Cookie);
-            Send(ev->Sender, new TEvRowDispatcher::TEvPurecalcCompileResponse(status.GetStatus(), status.GetErrorDescription()), 0, ev->Cookie);
+            LOG_ROW_DISPATCHER_ERROR("Compilation failed for request");


А куда id потерялся? Как понять какая именно компилияция зафейлилась?

id переехало в LogPrefix, чтобы точно не забыть поставить его везде:

ydb/ydb/core/fq/libs/row_dispatcher/purecalc_compilation/compile_service.cpp

Line 42 in 6c8320c

, LogPrefix(TStringBuilder() << "TPurecalcCompileActor " << request->Sender << " [id " << request->Cookie << "]: ")

dorooleg · 2024-12-23T15:32:36Z

ydb/core/fq/libs/row_dispatcher/purecalc_compilation/compile_service.cpp

        } else {
-            LOG_ROW_DISPATCHER_TRACE("Compilation completed for request with id: " << ev->Cookie);
-            Send(ev->Sender, new TEvRowDispatcher::TEvPurecalcCompileResponse(std::move(programHolder)), 0, ev->Cookie);
+            LOG_ROW_DISPATCHER_TRACE("Compilation completed for request");


Сюда тоже нужен какой-то идентификатор, иначе спам будет бесполезный

Тут тоже, id переехало в LogPrefix:

ydb/ydb/core/fq/libs/row_dispatcher/purecalc_compilation/compile_service.cpp

Line 42 in 6c8320c

, LogPrefix(TStringBuilder() << "TPurecalcCompileActor " << request->Sender << " [id " << request->Cookie << "]: ")

GrigoriyPA requested a review from a team as a code owner December 11, 2024 11:24

github-actions bot added the new-feature label Dec 11, 2024

GrigoriyPA requested review from dorooleg and kardymonds December 11, 2024 11:25

github-actions bot added performance and removed new-feature labels Dec 11, 2024

GrigoriyPA added the rebase-and-check Rebase PR with the current base branch and check label Dec 18, 2024

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Dec 18, 2024

GrigoriyPA added 3 commits December 23, 2024 07:47

Added parallel compilation

833b161

Fixed default value

0f21c64

Added ActorName into compile actors

6c8320c

GrigoriyPA force-pushed the YQ-3926-RD-add-parallel-purecalc-compilation branch from 977faeb to 6c8320c Compare December 23, 2024 07:53

dorooleg requested changes Dec 23, 2024

View reviewed changes

GrigoriyPA requested a review from dorooleg December 23, 2024 16:20

dorooleg approved these changes Dec 23, 2024

View reviewed changes

GrigoriyPA merged commit 22f4d0c into ydb-platform:main Dec 23, 2024
10 checks passed

GrigoriyPA deleted the YQ-3926-RD-add-parallel-purecalc-compilation branch December 23, 2024 16:24

GrigoriyPA added a commit to GrigoriyPA/ydb that referenced this pull request Dec 23, 2024

YQ-3926 RD added parallel purecalc compilation (ydb-platform#12505)

22678ae

asmyasnikov pushed a commit to asmyasnikov/ydb that referenced this pull request Dec 23, 2024

YQ-3926 RD added parallel purecalc compilation (ydb-platform#12505)

8d30b72

GrigoriyPA mentioned this pull request Dec 25, 2024

Fixed empty error from CMS create tenant request #12977

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YQ-3926 RD added parallel purecalc compilation #12505

YQ-3926 RD added parallel purecalc compilation #12505

GrigoriyPA commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

dorooleg Dec 23, 2024

GrigoriyPA Dec 23, 2024

dorooleg Dec 23, 2024

GrigoriyPA Dec 23, 2024

YQ-3926 RD added parallel purecalc compilation #12505

YQ-3926 RD added parallel purecalc compilation #12505

Conversation

GrigoriyPA commented Dec 11, 2024 • edited Loading

Changelog entry

Changelog category

Additional information

github-actions bot commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 18, 2024 • edited Loading

github-actions bot commented Dec 18, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 • edited Loading

dorooleg Dec 23, 2024

Choose a reason for hiding this comment

GrigoriyPA Dec 23, 2024

Choose a reason for hiding this comment

dorooleg Dec 23, 2024

Choose a reason for hiding this comment

GrigoriyPA Dec 23, 2024

Choose a reason for hiding this comment

GrigoriyPA commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading