Enable COUNT in view queries

[jepett0]

#6394
KIKIMR-21731

• Add ModuleResolver from KQP host.

  It seems that COUNT and other aggregate functions are implemented as imported modules. See, for example, AST of the following query:

  SELECT COUNT(*) FROM `/Root/episodes`

  Note the following line:

  (import aggregate_module '"/lib/yql/aggregate.yql")

  Aggregate module count_traits_factory is implemented here.

• Add ExpandApplyForLambdas transformation step after RewriteReadFromView.

  The same is done to rewrite reads from views in YT. The necessity could be well understood by studying the differences in:

  • transformations of

    SELECT COUNT(*) FROM `/Root/episodes`

    (Note how Apply of a lambda disappears due to some graph transformation between the first and the second debug printout of the TExprNode graph of the pure SELECT.)

  • transformations of the same query written inside a view

Both (ModuleResolver addition and ExpandApplyForLambdas step repetition) are necessary to enable aggregate functions in views.

Copying default KQP host's translation settings is done to make view compilation more similar to a pure SELECT compilation. It is not a necessary part of the fix.

[dorooleg]

What do these tests check? Do you have a validation for the results of these queries?

[jepett0]

Yeah, these tests are a non-standard way to test views. They are functional tests, but written in C++. A test case is a single folder in ydb/core/kqp/ut/view/input/cases (like count_rows folder, or count_episodes folder). Each test case is executed in the following order:

1. execute create_view.sql
2. execute etalon_query.sql
3. execute select_from_view.sql
4. execute drop_view.sql

Results of step (2) and step (3) are compared as YSON strings here.

[jepett0]

This particular test (count_episodes) is a way to test COUNT inside a view, but with a bit more complex query than just SELECT COUNT(*) FROM some_table

[dorooleg]

It's an intresting approach. Thank you for exlanation :)

[github-actions]

⚪ 2024-08-05 13:36:59 UTC Pre-commit check for 85daa8e has started.
⚪ 2024-08-05 13:39:42 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-05 15:34:05 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
9758	9715	0	4	24	15

🟢 2024-08-05 15:35:14 UTC Build successful.
🟢 2024-08-05 15:35:42 UTC ydbd size 5.4 GiB changed* by -201.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: 6533a28	merge: 85daa8e	diff	diff %
ydbd size	5 826 345 728 Bytes	5 826 139 096 Bytes	-201.8 KiB	-0.004%
ydbd stripped size	1 463 285 456 Bytes	1 463 202 576 Bytes	-80.9 KiB	-0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2024-08-05 13:38:37 UTC Pre-commit check for 85daa8e has started.
⚪ 2024-08-05 13:48:45 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 13:55:09 UTC Build successful.

[github-actions]

⚪ 2024-08-05 13:39:32 UTC Pre-commit check for 85daa8e has started.
⚪ 2024-08-05 13:49:28 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 15:46:32 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
38497	33175	0	3	5307	12

⚪ 2024-08-05 15:51:00 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-05 15:58:35 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
18 (only retried tests)	9	0	0	1	8

🟢 2024-08-05 16:01:46 UTC Build successful.
🟢 2024-08-05 16:02:24 UTC ydbd size 8.1 GiB changed* by -258.6 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: 6533a28	merge: 85daa8e	diff	diff %
ydbd size	8 666 494 576 Bytes	8 666 229 768 Bytes	-258.6 KiB	-0.003%
ydbd stripped size	471 487 400 Bytes	471 471 720 Bytes	-15.3 KiB	-0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[jepett0]

GUCSettings are needed here. That is why we pass them in CreateKikimrDataSource.

We cannot, unfortunately, pass the KQP translation settings builder itself from the KQP host to the Kikimr DataSource, because it needs a query to be built.

[jepett0]

Here I remove unnecessary includes and add only the explicitly needed ones. It needs to be done to eliminate dependency loop between kqp/provider and kqp/host

[jepett0]

This test might be an overkill, but it uses SUM, AVG and COUNT aggregate functions as well as a WINDOW clause