bloom filter for ngramms #12893

ivanmorozov333 · 2024-12-23T17:12:52Z

New feature
Improvement
Performance improvement

github-actions · 2024-12-23T17:16:33Z

⚪ 2024-12-23 17:16:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4623dc8 has started.
⚪ 2024-12-23 17:16:44 UTC Artifacts will be uploaded here
⚪ 2024-12-23 17:20:05 UTC ya make is running...
🟡 2024-12-23 18:49:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31032	28084	0	2	2831	115

⚪ 2024-12-23 18:52:56 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-23 19:04:53 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
166 (only retried tests)	51	0	1	1	113

⚪ 2024-12-23 19:05:01 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-23 19:16:57 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
162 (only retried tests)	49	0	1	0	112

🟢 2024-12-23 19:17:04 UTC Build successful.
🟡 2024-12-23 19:17:25 UTC ydbd size 2.1 GiB changed* by +689.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ab00ae5`	merge: `4623dc8`	diff	diff %
ydbd size	2 235 025 960 Bytes	2 235 731 824 Bytes	+689.3 KiB	+0.032%
ydbd stripped size	477 982 320 Bytes	478 076 592 Bytes	+92.1 KiB	+0.020%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-23T17:16:38Z

⚪ 2024-12-23 17:16:37 UTC Pre-commit check linux-x86_64-release-asan for 4623dc8 has started.
⚪ 2024-12-23 17:16:44 UTC Artifacts will be uploaded here
⚪ 2024-12-23 17:19:56 UTC ya make is running...
🟡 2024-12-23 18:54:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14060	13980	0	17	14	49

🟢 2024-12-23 18:55:29 UTC Build successful.
🟡 2024-12-23 18:55:51 UTC ydbd size 3.6 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ab00ae5`	merge: `4623dc8`	diff	diff %
ydbd size	3 865 642 504 Bytes	3 866 824 184 Bytes	+1.1 MiB	+0.031%
ydbd stripped size	1 350 675 856 Bytes	1 350 956 656 Bytes	+274.2 KiB	+0.021%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

zverevgeny · 2024-12-24T05:48:39Z

ydb/core/tx/columnshard/engines/scheme/indexes/abstract/program.h

@@ -3,13 +3,50 @@

 namespace NKikimr::NOlap::NIndexes::NRequest {

+class TLikeDescription {


Я б назвал TSqlLikeSplitter

zverevgeny · 2024-12-24T05:54:40Z

ydb/core/tx/columnshard/engines/storage/indexes/bloom_ngramm/checker.h

+    }
+
+    TFixStringBitsStorage(const ui32 sizeBits)
+        : Data(sizeBits / 8 + ((sizeBits % 8) ? 1 : 0), '\0') {


(sizeBits + 7) / 8

zverevgeny · 2024-12-24T06:02:41Z

ydb/core/tx/columnshard/engines/storage/indexes/bloom_ngramm/checker.h

+#include <ydb/core/tx/columnshard/engines/scheme/indexes/abstract/simple.h>
+namespace NKikimr::NOlap::NIndexes::NBloomNGramm {
+
+class TFixStringBitsStorage {


Почему не std::vector?

github-actions · 2024-12-24T08:19:22Z

⚪ 2024-12-24 08:19:22 UTC Pre-commit check linux-x86_64-release-asan for 8cc9341 has started.
⚪ 2024-12-24 08:19:33 UTC Artifacts will be uploaded here
⚪ 2024-12-24 08:22:46 UTC ya make is running...
🟡 2024-12-24 09:55:56 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14060	13966	0	39	8	47

🟢 2024-12-24 09:57:06 UTC Build successful.
🟡 2024-12-24 09:57:28 UTC ydbd size 3.6 GiB changed* by +1000.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `18ed18e`	merge: `8cc9341`	diff	diff %
ydbd size	3 865 652 864 Bytes	3 866 677 336 Bytes	+1000.5 KiB	+0.027%
ydbd stripped size	1 350 678 768 Bytes	1 350 924 112 Bytes	+239.6 KiB	+0.018%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T08:19:23Z

⚪ 2024-12-24 08:19:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8cc9341 has started.
⚪ 2024-12-24 08:19:34 UTC Artifacts will be uploaded here
⚪ 2024-12-24 08:22:50 UTC ya make is running...
🟡 2024-12-24 09:52:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31032	28083	0	3	2831	115

⚪ 2024-12-24 09:55:31 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-24 10:07:22 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
168 (only retried tests)	56	0	0	1	111

🟢 2024-12-24 10:07:29 UTC Build successful.
🟡 2024-12-24 10:07:47 UTC ydbd size 2.1 GiB changed* by +605.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `18ed18e`	merge: `8cc9341`	diff	diff %
ydbd size	2 235 030 744 Bytes	2 235 650 232 Bytes	+605.0 KiB	+0.028%
ydbd stripped size	477 983 088 Bytes	478 062 960 Bytes	+78.0 KiB	+0.017%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T11:27:37Z

⚪ 2024-12-24 11:27:36 UTC Pre-commit check linux-x86_64-release-asan for 200c413 has started.
⚪ 2024-12-24 11:27:48 UTC Artifacts will be uploaded here
⚪ 2024-12-24 11:31:03 UTC ya make is running...
🟡 2024-12-24 13:04:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14059	13976	0	25	8	50

🟢 2024-12-24 13:05:45 UTC Build successful.
🟡 2024-12-24 13:06:08 UTC ydbd size 3.6 GiB changed* by +1000.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7cd35f3`	merge: `200c413`	diff	diff %
ydbd size	3 865 652 520 Bytes	3 866 677 040 Bytes	+1000.5 KiB	+0.027%
ydbd stripped size	1 350 679 408 Bytes	1 350 924 816 Bytes	+239.7 KiB	+0.018%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T11:27:50Z

⚪ 2024-12-24 11:27:50 UTC Pre-commit check linux-x86_64-relwithdebinfo for 200c413 has started.
⚪ 2024-12-24 11:28:01 UTC Artifacts will be uploaded here
⚪ 2024-12-24 11:31:15 UTC ya make is running...
🟡 2024-12-24 12:59:09 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31032	28081	0	3	2832	116

⚪ 2024-12-24 13:02:11 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-24 13:13:58 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
167 (only retried tests)	56	0	0	0	111

🟢 2024-12-24 13:14:08 UTC Build successful.
🟡 2024-12-24 13:14:26 UTC ydbd size 2.1 GiB changed* by +605.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7cd35f3`	merge: `200c413`	diff	diff %
ydbd size	2 235 028 664 Bytes	2 235 648 160 Bytes	+605.0 KiB	+0.028%
ydbd stripped size	477 983 088 Bytes	478 062 960 Bytes	+78.0 KiB	+0.017%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

bloom filter for ngramms

0ca7ce6

github-actions bot added the new-feature label Dec 23, 2024

zverevgeny reviewed Dec 24, 2024

View reviewed changes

fixes and improvements

5709a70

improve tests

45189a5

ivanmorozov333 marked this pull request as ready for review December 24, 2024 15:44

ivanmorozov333 requested a review from a team as a code owner December 24, 2024 15:44

ivanmorozov333 enabled auto-merge (squash) December 24, 2024 15:58

zverevgeny approved these changes Dec 25, 2024

View reviewed changes

ivanmorozov333 merged commit d014966 into ydb-platform:main Dec 25, 2024
12 checks passed

ivanmorozov333 deleted the bloom_filter_ngramm branch December 25, 2024 07:39

This was referenced Dec 25, 2024

Fixed empty error from CMS create tenant request #12977

Merged

YQ-3949 fix pq source sensors names #13088

Merged

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 2, 2025

bloom filter for ngramms (ydb-platform#12893)

cd2143b

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 4, 2025

bloom filter for ngramms (ydb-platform#12893)

77f887b

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 5, 2025

bloom filter for ngramms (ydb-platform#12893)

debaf05

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 5, 2025

bloom filter for ngramms (ydb-platform#12893)

e2af1d5

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 8, 2025

bloom filter for ngramms (ydb-platform#12893)

40de793

zverevgeny mentioned this pull request Jan 9, 2025

Integrate changes from main #13202

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bloom filter for ngramms #12893

bloom filter for ngramms #12893

ivanmorozov333 commented Dec 23, 2024

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

zverevgeny Dec 24, 2024

zverevgeny Dec 24, 2024

zverevgeny Dec 24, 2024

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

		@@ -3,13 +3,50 @@

		namespace NKikimr::NOlap::NIndexes::NRequest {

		class TLikeDescription {

bloom filter for ngramms #12893

bloom filter for ngramms #12893

Conversation

ivanmorozov333 commented Dec 23, 2024

github-actions bot commented Dec 23, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 • edited Loading

zverevgeny Dec 24, 2024

Choose a reason for hiding this comment

zverevgeny Dec 24, 2024

Choose a reason for hiding this comment

zverevgeny Dec 24, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading