Fix MakeBlocks usage in BlockMapJoinCore computation node #8686

igormunkin · 2024-09-03T15:11:32Z

This patch fixes block output processing in case when not all data created via MakeBlocks has been yielded to the computation node callee.

Since we have no guarantee which kind of array data IArrayBuilder produces, TBlockState.Slice creates the "minimal-common-size" chunks of the TBlockState.Deques vectors, to be returned to the callee. Before the fix, anything left in TBlockState.Deques was flushed in scope of FillArrays subroutine with the consecutive MakeBlocks call. As a result of the patch, HasBlocks method has been introduced in TBlockJoinState, to check whether anything is left in TBlockState.Deques. The further input processing is proceeded only, when all blocks in TBlockState.Deques have been returned.

Changelog category

Bugfix

github-actions · 2024-09-03T15:14:00Z

⚪ 2024-09-03 15:13:59 UTC Pre-commit check linux-x86_64-release-asan for ebd9283 has started.
⚪ 2024-09-03 15:18:36 UTC ya make is running...
🔴 2024-09-03 16:56:43 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12535	12451	0	22	34	28

🟢 2024-09-03 16:57:47 UTC Build successful.
🟡 2024-09-03 16:58:19 UTC ydbd size 5.5 GiB changed* by +198.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ec42e43`	merge: `ebd9283`	diff	diff %
ydbd size	5 954 870 600 Bytes	5 955 073 856 Bytes	+198.5 KiB	+0.003%
ydbd stripped size	1 489 504 304 Bytes	1 489 560 496 Bytes	+54.9 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-03T15:14:57Z

⚪ 2024-09-03 15:14:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for ebd9283 has started.
⚪ 2024-09-03 15:17:53 UTC ya make is running...
🟡 2024-09-03 16:34:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
74725	60958	0	6	13735	26

⚪ 2024-09-03 16:42:20 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-03 16:50:52 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41 (only retried tests)	21	0	0	0	20

🟢 2024-09-03 16:50:58 UTC Build successful.
🟡 2024-09-03 16:51:33 UTC ydbd size 8.3 GiB changed* by +364.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ec42e43`	merge: `ebd9283`	diff	diff %
ydbd size	8 918 549 776 Bytes	8 918 923 376 Bytes	+364.8 KiB	+0.004%
ydbd stripped size	481 813 544 Bytes	481 840 648 Bytes	+26.5 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-03T15:15:00Z

⚪ 2024-09-03 15:14:59 UTC Pre-commit check linux-x86_64-release-clang14 for ebd9283 has started.
⚪ 2024-09-03 15:17:51 UTC ya make is running...
🟢 2024-09-03 15:37:59 UTC Build successful.

github-actions · 2024-09-04T18:12:38Z

⚪ 2024-09-04 18:12:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for d08d6fc has started.
⚪ 2024-09-04 18:15:53 UTC ya make is running...
🟡 2024-09-04 19:24:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75544	61745	0	44	13735	20

⚪ 2024-09-04 19:31:34 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-04 19:41:25 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73 (only retried tests)	22	0	37	0	14

⚪ 2024-09-04 19:41:33 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-04 19:48:39 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
56 (only retried tests)	29	0	13	0	14

🟢 2024-09-04 19:48:46 UTC Build successful.
🟢 2024-09-04 19:49:22 UTC ydbd size 8.4 GiB changed* by +4.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `93bd594`	merge: `d08d6fc`	diff	diff %
ydbd size	9 000 764 872 Bytes	9 000 769 232 Bytes	+4.3 KiB	+0.000%
ydbd stripped size	486 645 096 Bytes	486 645 736 Bytes	+640 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-04T18:14:04Z

⚪ 2024-09-04 18:14:03 UTC Pre-commit check linux-x86_64-release-asan for d08d6fc has started.
⚪ 2024-09-04 18:17:59 UTC ya make is running...
🔴 2024-09-04 19:52:47 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13347	13273	0	17	34	23

🟢 2024-09-04 19:53:57 UTC Build successful.
🟢 2024-09-04 19:54:27 UTC ydbd size 5.6 GiB changed* by +3.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `93bd594`	merge: `d08d6fc`	diff	diff %
ydbd size	6 019 220 008 Bytes	6 019 223 352 Bytes	+3.3 KiB	+0.000%
ydbd stripped size	1 507 000 912 Bytes	1 507 001 936 Bytes	+1.0 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-04T18:14:24Z

⚪ 2024-09-04 18:14:23 UTC Pre-commit check linux-x86_64-release-clang14 for d08d6fc has started.
⚪ 2024-09-04 18:17:38 UTC ya make is running...
🟢 2024-09-04 18:23:58 UTC Build successful.

github-actions · 2024-09-05T03:09:11Z

⚪ 2024-09-05 03:09:10 UTC Pre-commit check linux-x86_64-release-clang14 for 7ebf4e6 has started.
⚪ 2024-09-05 03:12:19 UTC ya make is running...
🟢 2024-09-05 04:01:22 UTC Build successful.

github-actions · 2024-09-05T03:09:15Z

⚪ 2024-09-05 03:09:14 UTC Pre-commit check linux-x86_64-release-asan for 7ebf4e6 has started.
⚪ 2024-09-05 03:12:06 UTC ya make is running...
🔴 2024-09-05 04:50:18 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13358	13256	0	40	37	25

🟢 2024-09-05 04:51:25 UTC Build successful.
🟢 2024-09-05 04:51:55 UTC ydbd size 5.6 GiB changed* by +7.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `7f64b2d`	merge: `7ebf4e6`	diff	diff %
ydbd size	6 019 595 400 Bytes	6 019 602 840 Bytes	+7.3 KiB	+0.000%
ydbd stripped size	1 507 062 320 Bytes	1 507 067 440 Bytes	+5.0 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-05T03:09:50Z

⚪ 2024-09-05 03:09:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7ebf4e6 has started.
⚪ 2024-09-05 03:12:46 UTC ya make is running...
🟡 2024-09-05 04:26:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75567	61777	0	14	13753	23

⚪ 2024-09-05 04:33:21 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-05 04:45:01 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
48 (only retried tests)	29	0	0	1	18

🟢 2024-09-05 04:45:08 UTC Build successful.
🟢 2024-09-05 04:45:48 UTC ydbd size 8.4 GiB changed* by +8.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `7f64b2d`	merge: `7ebf4e6`	diff	diff %
ydbd size	9 001 080 624 Bytes	9 001 089 160 Bytes	+8.3 KiB	+0.000%
ydbd stripped size	486 651 624 Bytes	486 656 424 Bytes	+4.7 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-06T11:07:33Z

⚪ 2024-09-06 11:07:32 UTC Pre-commit check linux-x86_64-release-asan for 7ac36bf has started.
⚪ 2024-09-06 11:10:28 UTC ya make is running...
🔴 2024-09-06 12:48:29 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13360	13273	0	25	39	23

🟢 2024-09-06 12:49:50 UTC Build successful.
🟢 2024-09-06 12:50:20 UTC ydbd size 5.6 GiB changed* by +9.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `7770159`	merge: `7ac36bf`	diff	diff %
ydbd size	6 028 974 952 Bytes	6 028 984 336 Bytes	+9.2 KiB	+0.000%
ydbd stripped size	1 509 662 256 Bytes	1 509 664 560 Bytes	+2.2 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-06T11:08:25Z

⚪ 2024-09-06 11:08:24 UTC Pre-commit check linux-x86_64-release-clang14 for 7ac36bf has started.
⚪ 2024-09-06 11:13:03 UTC ya make is running...
🟢 2024-09-06 11:19:55 UTC Build successful.

github-actions · 2024-09-06T11:10:30Z

⚪ 2024-09-06 11:10:30 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7ac36bf has started.
⚪ 2024-09-06 11:14:07 UTC ya make is running...
🟡 2024-09-06 12:23:42 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75595	61717	0	84	13769	25

⚪ 2024-09-06 12:31:09 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-06 12:43:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
121 (only retried tests)	84	0	20	0	17

⚪ 2024-09-06 12:43:13 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-06 12:50:13 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40 (only retried tests)	20	0	3	0	17

🟢 2024-09-06 12:50:20 UTC Build successful.
🟢 2024-09-06 12:53:09 UTC ydbd size 8.4 GiB changed* by +4.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4782b8a`	merge: `7ac36bf`	diff	diff %
ydbd size	9 013 371 016 Bytes	9 013 375 456 Bytes	+4.3 KiB	+0.000%
ydbd stripped size	487 291 688 Bytes	487 292 392 Bytes	+704 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

igormunkin added the area/yql YQL query language issues label Sep 3, 2024

igormunkin self-assigned this Sep 3, 2024

github-actions bot added the bugfix label Sep 3, 2024

github-actions bot added bugfix and removed bugfix labels Sep 4, 2024

igormunkin marked this pull request as ready for review September 4, 2024 17:42

igormunkin requested a review from a team as a code owner September 4, 2024 17:42

igormunkin requested a review from vitstn September 4, 2024 17:42

igormunkin added the rebase-and-check Rebase PR with the current base branch and check label Sep 5, 2024

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Sep 5, 2024

vitstn previously approved these changes Sep 6, 2024

View reviewed changes

igormunkin added 5 commits September 6, 2024 11:05

Move assertion to the right place

dd44fe3

Introduce HasBlocks helper to handle output blocks the right way

953574a

Check whether output builders are not full prior to fetch

926d7c2

Add assertion in FillArrays

b43e6e5

Add tests

0481c47

igormunkin dismissed vitstn’s stale review via 0481c47 September 6, 2024 11:05

igormunkin force-pushed the YQL-18498-block-map-join-comp-node-fix-makeblocks branch from 8ba5d64 to 0481c47 Compare September 6, 2024 11:05

igormunkin requested a review from vitstn September 6, 2024 11:05

vitstn approved these changes Sep 6, 2024

View reviewed changes

igormunkin enabled auto-merge (squash) September 6, 2024 13:06

maximyurchuk disabled auto-merge September 6, 2024 13:57

maximyurchuk merged commit e10ef55 into ydb-platform:main Sep 6, 2024
7 of 12 checks passed

igormunkin deleted the YQL-18498-block-map-join-comp-node-fix-makeblocks branch September 6, 2024 17:11

niksaveliev mentioned this pull request Sep 9, 2024

Fix CLI max partitions default #8944

Merged

GrigoriyPA mentioned this pull request Sep 10, 2024

YQ-3653 added resource pools classifiers validations #9037

Merged

shnikd mentioned this pull request Sep 11, 2024

Long name domain test #9096

Closed

niksaveliev mentioned this pull request Sep 12, 2024

Fix reading without consumer from federation #9136

Merged

StekPerepolnen mentioned this pull request Sep 17, 2024

doc hc api images source #9376

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix MakeBlocks usage in BlockMapJoinCore computation node #8686

Fix MakeBlocks usage in BlockMapJoinCore computation node #8686

igormunkin commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

Fix MakeBlocks usage in BlockMapJoinCore computation node #8686

Fix MakeBlocks usage in BlockMapJoinCore computation node #8686

Conversation

igormunkin commented Sep 3, 2024 • edited Loading

Changelog category

github-actions bot commented Sep 3, 2024 • edited Loading

github-actions bot commented Sep 3, 2024 • edited Loading

github-actions bot commented Sep 3, 2024 • edited Loading

github-actions bot commented Sep 4, 2024 • edited Loading

github-actions bot commented Sep 4, 2024 • edited Loading

github-actions bot commented Sep 4, 2024 • edited Loading

github-actions bot commented Sep 5, 2024 • edited Loading

github-actions bot commented Sep 5, 2024 • edited Loading

github-actions bot commented Sep 5, 2024 • edited Loading

github-actions bot commented Sep 6, 2024 • edited Loading

github-actions bot commented Sep 6, 2024 • edited Loading

github-actions bot commented Sep 6, 2024 • edited Loading

igormunkin commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 4, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading