ci: add ydb test cases

[ZeaLoVe]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Add ydb logic test cases, part of issue #5889

Fixes #issue

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
databend	✅ Ready (Inspect)	Visit Preview	Jul 20, 2022 at 5:10AM (UTC)

[Xuanwo]

Can we improve the output?

    results: [(<re.Match object; span=(0, 4), match='----'>, 215, '1680 222 1 112.0 674\n1826 222 1 122.0 738\n1902 222 4 127.0 760\n1985 333 4 132.0 793\n2046 222 2 137.0 827\n2202 444 2 147.0 880\n2281 333 1 152.0 905\n2432 444 1 162.0 966\n2501 333 1 167.0 1000\n2654 444 1 177.0 1057\n2728 333 3 182.0 1096\n3175 333 3 212.0 1277\n3331 222 1 222.0 1338\n3473 333 1 232.0 1391\n3553 444 1 237.0 1416\n3706 111 2 247.0 1484')],
    runs_on: {'clickhouse', 'mysql', 'http'},

I expect to read:

Expected:

xxx

Actual:

xxx

It could be better in diff-style.

[Xuanwo]

cc @leiysky for help addressing SQL logic tests.

[ZeaLoVe]

Can we improve the output?

    results: [(<re.Match object; span=(0, 4), match='----'>, 215, '1680 222 1 112.0 674\n1826 222 1 122.0 738\n1902 222 4 127.0 760\n1985 333 4 132.0 793\n2046 222 2 137.0 827\n2202 444 2 147.0 880\n2281 333 1 152.0 905\n2432 444 1 162.0 966\n2501 333 1 167.0 1000\n2654 444 1 177.0 1057\n2728 333 3 182.0 1096\n3175 333 3 212.0 1277\n3331 222 1 222.0 1338\n3473 333 1 232.0 1391\n3553 444 1 237.0 1416\n3706 111 2 247.0 1484')],
    runs_on: {'clickhouse', 'mysql', 'http'},

I expect to read:

Expected:

xxx

Actual:

xxx

It could be better in diff-style.

If test return something but not match expected, will show like that, but when get a error, will only show error msg and raw statement infomation.

[leiysky]

The CI failed https://github.com/datafuselabs/databend/runs/7402822498?check_suite_focus=true due to #6682.

[Xuanwo]

If test return something but not match expected, will show like that,

Returning mismatched content in expeteced and actual will improve the readability.

[ZeaLoVe]

If test return something but not match expected, will show like that,

Returning mismatched content in expeteced and actual will improve the readability.

I see, but in this case, the most important thing is about SQL error, different errors need to be distinguished by different messages.

[ZeaLoVe]

@mergify update

[mergify]

update

✅ Branch has been successfully updated

[ZeaLoVe]

@leiysky @Xuanwo seems fixed, PTAL

[Xuanwo]

The sqllotic test has been canceled: https://github.com/datafuselabs/databend/runs/7407024695?check_suite_focus=true

[BohuTANG]

@ZeaLoVe can you take the reviewers?

[Xuanwo]

SQL logic test is too slow that exceeds 10 minutes limitation.

We can extend this limitation to 20 or 30 minutes, but we should improve this in the future.

I expect SQL logic tests could be finished within 10 minutes (maybe we can execute them concurrently)?

[ZeaLoVe]

SQL logic test is too slow that exceeds 10 minutes limitation.

We can extend this limitation to 20 or 30 minutes, but we should improve this in the future.

I expect SQL logic tests could be finished within 10 minutes (maybe we can execute them concurrently)?

YDB suites include mass number of complecated select statements, maybe run only on mysql handler ? Can this be optimize? cc @leiysky

[Xuanwo]

YDB suites include mass number of complecated select statements, maybe run only on mysql handler ?

How many data are we using to do sqllogic tests?

2022-07-19T13:01:44.992184Z  INFO databend_query::servers::mysql::mysql_interactive_worker: Normal query: select b as col1,
       case a+1 when b then 111 when c then 222
        when d then 333  when e then 444 else 555 end as col2,
       b-c as col3,
       case when c>avg_c then a*2 else b*10 end as col4,
       a as col5,
       abs(a) as col6,
       d-e as col7
from (
 select a, b, c, d, e, avg_c
 from (
  select avg(c) as avg_c from t1
 ) as Q cross join (select a, b, c, d, e from t1) as R
)
order by col5,col4,col2,col7,col3,col6,col1;
2022-07-19T13:01:45.128712Z  WARN databend_query::pipelines::executor::pipeline_pulling_executor: Send finish event error "Disconnected(..)"

Seems query can finish the query in 1s

[ZeaLoVe]

YDB suites include mass number of complecated select statements, maybe run only on mysql handler ?

How many data are we using to do sqllogic tests?

2022-07-19T13:01:44.992184Z  INFO databend_query::servers::mysql::mysql_interactive_worker: Normal query: select b as col1,
       case a+1 when b then 111 when c then 222
        when d then 333  when e then 444 else 555 end as col2,
       b-c as col3,
       case when c>avg_c then a*2 else b*10 end as col4,
       a as col5,
       abs(a) as col6,
       d-e as col7
from (
 select a, b, c, d, e, avg_c
 from (
  select avg(c) as avg_c from t1
 ) as Q cross join (select a, b, c, d, e from t1) as R
)
order by col5,col4,col2,col7,col3,col6,col1;
2022-07-19T13:01:45.128712Z  WARN databend_query::pipelines::executor::pipeline_pulling_executor: Send finish event error "Disconnected(..)"

Seems query can finish the query in 1s

About a dozen, I think something cause a degration, the stateless test run a longer time either, but it does not contains these test suites.

[Xuanwo]

After increasing the logic test timeout to 20 minutes, we can finish in 14 minutes.

This PR is OK to merge now, but I'm still concerned about the speed of the SQL logic test.

• Can we run the SQL logic test concurrently? For example, running 10 queries at the same time
• Maybe we should not print logs to stdout? (Save as file and upgrade to artifacts)
• Split SQL logic test into different parts.

This PR adds 10W lines and contains around 2W SQLs. Every SQL handled in 0.1s will cause us 30 minutes to finish the tests. So I don't think it's related to performance regression.

[ZeaLoVe]

macos logic test failure is a flaky error, https://github.com/datafuselabs/databend/runs/7421392619?check_suite_focus=true. issue here：#6614

[ZeaLoVe]

After increasing the logic test timeout to 20 minutes, we can finish in 14 minutes.

This PR is OK to merge now, but I'm still concerned about the speed of the SQL logic test.

• Can we run the SQL logic test concurrently? For example, running 10 queries at the same time
• Maybe we should not print logs to stdout? (Save as file and upgrade to artifacts)
• Split SQL logic test into different parts.

This PR adds 10W lines and contains around 2W SQLs. Every SQL handled in 0.1s will cause us 30 minutes to finish the tests. So I don't think it's related to performance regression.

Maybe ydb test only run on mysql handler? Split logic test into three pipeline also a choice, but I worry about the little degration, logic test just expose problem.

[Xuanwo]

Maybe ydb test only run on mysql handler?

Negetive, this option is never considered. We do want all handlers to be tested.

SQL Logic Test is designed for this.

Split logic test into three pipeline also a choice, but I worry about the little degration, logic test just expose problem.

What's the degration you mentioned here?

---

Is there any block to doing concurrent SQL logic testing? I think databend-query is not in very high load and is OK to take at least 10 queries concurrently.

[ZeaLoVe]

Maybe ydb test only run on mysql handler?

Negetive, this option is never considered. We do want all handlers to be tested.

SQL Logic Test is designed for this.

Split logic test into three pipeline also a choice, but I worry about the little degration, logic test just expose problem.

What's the degration you mentioned here?

Is there any block to doing concurrent SQL logic testing? I think databend-query is not in very high load and is OK to take at least 10 queries concurrently.

Concurrent is not a good idea, need the table name and database name in test to be unique, otherwise will cause more flaky error.
The degration is now my feeling, I ran this test local using v0.7.133 does not need 14min. Maybe caculate time cost will helps #6694 .
Can we merge this pr, and solve problem one by one?

[flaneur2020]

Concurrent is not a good idea, need the table name and database name in test to be unique, otherwise will cause more flaky error.

i guess we can try split the ydb test cases in a standalone github action job, so we can get the parallelism at the test suites level, like:

• job:ydb-sqllogictest
• job:stateless-sqlogictest
• job:clickhouse-sqlogictest
• ...

or even make a matrix:

• job:ydb-sqllogictest:mysql-protocol
• job:ydb-sqllogictest:clickhouse-protocol
• job:ydb-sqllogictest:http-protocol

these jobs can be executed in a parallel manner

[flaneur2020]

I ran this test local using v0.7.133 does not need 14min

I feels the cpu hz in the github action is not as high as my local 🤔

[flaneur2020]

Can we merge this pr, and solve problem one by one?

agree, we can record the feedbacks in the tracking issue.

this PR adds a massive test cases, can helps increasing our coverage quite a lot!

optimizing CI speed is a worthy work, it plays as an productivity leverage to all the project members.

[Xuanwo]

All concern has been added in #5889

Let's move forward!

[Xuanwo]

cc @leiysky @everpcpc @PsiACE for reviewing

[PsiACE]

LGTM.

I would prefer that we allow test suites to be loaded externally, rather than always piling them into this repo.

[Xuanwo]

I would prefer that we allow test suites to be loaded externally, rather than always piling them into this repo.

Nice idea, can you fill an issue for this?