feature(format): refactor output format #5422

sundy-li · 2022-05-17T07:16:52Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR:

Make clickhouse handler support output format.
Make clickhouse handler support post SQL in body.
Introduce parquet/csv/ndjson output format

Changelog

New Feature

Related Issues

Fixes #5433

vercel · 2022-05-17T07:16:56Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		May 18, 2022 at 8:20AM (UTC)

mergify · 2022-05-17T07:17:25Z

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

BohuTANG

Nice job!
I think this is useful for the result cache cc @youngsofun @flaneur2020

BohuTANG · 2022-05-18T04:13:11Z

The unit-test is skipped because the disk space is full:

[test_unit](https://github.com/datafuselabs/databend/runs/6482164666?check_suite_focus=true)
System.IO.IOException: No space left on device : '/home/runner/runners/2.291.1/_diag/Worker_20220518-030049-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.291.1/_diag/Worker_20220518-030049-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.291.1/_diag/Worker_20220518-030049-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.Main(String[] args)

Re-run the job to check it.

BohuTANG · 2022-05-18T05:03:06Z

@sundy-li Seems some tests break the unit-test in this patch. There is no space left for the CI, so the unit-test is skipped every time.

BohuTANG · 2022-05-18T05:05:25Z

Oops, it's not related to this patch, another PR has the same issue too:
https://github.com/datafuselabs/databend/actions/runs/2343011703

Ping @Xuanwo @everpcpc for help

query/src/servers/http/clickhouse_handler.rs

youngsofun · 2022-05-18T05:14:07Z

common/datavalues/src/types/serializations/string.rs

+        let column: &StringColumn = Series::check_get(column)?;
+        let result: Vec<String> = column
+            .iter()
+            .map(|v| format!("{:?}", String::from_utf8_lossy(v)))


need escape, but we can do it later.

Xuanwo · 2022-05-18T05:22:46Z

Oops, it's not related to this patch, another PR has the same issue too: https://github.com/datafuselabs/databend/actions/runs/2343011703

Ping @Xuanwo @everpcpc for help

Maybe related to our rust cache, I have reset it to see whether it works.

youngsofun · 2022-05-18T05:46:05Z

common/ast/tests/it/parser.rs

@@ -186,6 +186,7 @@ fn test_query() {
    let mut mint = Mint::new("tests/it/testdata");
    let mut file = mint.new_goldenfile("query.txt").unwrap();
    let cases = &[
+        r#"select * from a limit 3 offset 4 format csv"#,


@sundy-li where supposed to use the format in this SQL?

At the end of the sql.

sundy-li · 2022-05-18T05:51:41Z

Now unit tests have some bugs, I'll fix them later.

Xuanwo · 2022-05-18T06:15:12Z

Is test_unit able to run without hitting no space left error?

…utput-format

sundy-li added 2 commits May 17, 2022 11:13

feature(format): refactor-output-format

d7aed31

feature(format): support parquet format

afa7ddc

databend-bot added the need-review label May 17, 2022

sundy-li changed the title ~~feature(format): refactor-output-format~~ feature(format): refactor output format May 17, 2022

mergify bot added the pr-feature this PR introduces a new feature to the codebase label May 17, 2022

sundy-li added 3 commits May 18, 2022 10:36

feature(format): support parquet/ndjson formats

70e4acd

Merge branch 'main' into output-format

231376a

feature(format): add tests

05dbbc3

sundy-li marked this pull request as ready for review May 18, 2022 03:55

sundy-li requested a review from BohuTANG as a code owner May 18, 2022 03:55

sundy-li requested a review from youngsofun May 18, 2022 03:56

BohuTANG approved these changes May 18, 2022

View reviewed changes

youngsofun reviewed May 18, 2022

View reviewed changes

query/src/servers/http/clickhouse_handler.rs Show resolved Hide resolved

youngsofun reviewed May 18, 2022

View reviewed changes

Try reset rust cache to avoid no space left

154270d

youngsofun reviewed May 18, 2022

View reviewed changes

youngsofun approved these changes May 18, 2022

View reviewed changes

Merge branch 'main' into output-format

9e2d0f5

feature(format): support values format

e5decc6

sundy-li and others added 6 commits May 18, 2022 14:37

feature(format): support values format

e9dcbb9

Merge branch 'output-format' of github.com:sundy-li/fuse-query into o…

df64e4c

…utput-format

Merge branch 'main' into output-format

86d2315

Merge branch 'main' into output-format

f3afa8b

Merge branch 'main' into output-format

967e068

Merge branch 'main' into output-format

9903a6d

mergify bot merged commit 3d5700c into databendlabs:main May 18, 2022

sundy-li added the pr-need-doc label May 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(format): refactor output format #5422

feature(format): refactor output format #5422

sundy-li commented May 17, 2022 •

edited

Loading

vercel bot commented May 17, 2022 •

edited

Loading

mergify bot commented May 17, 2022

BohuTANG left a comment •

edited

Loading

BohuTANG commented May 18, 2022 •

edited

Loading

BohuTANG commented May 18, 2022

BohuTANG commented May 18, 2022 •

edited

Loading

youngsofun May 18, 2022

Xuanwo commented May 18, 2022

youngsofun May 18, 2022

sundy-li May 18, 2022

sundy-li commented May 18, 2022

Xuanwo commented May 18, 2022

feature(format): refactor output format #5422

feature(format): refactor output format #5422

Conversation

sundy-li commented May 17, 2022 • edited Loading

Summary

Changelog

Related Issues

vercel bot commented May 17, 2022 • edited Loading

mergify bot commented May 17, 2022

BohuTANG left a comment • edited Loading

Choose a reason for hiding this comment

BohuTANG commented May 18, 2022 • edited Loading

BohuTANG commented May 18, 2022

BohuTANG commented May 18, 2022 • edited Loading

youngsofun May 18, 2022

Choose a reason for hiding this comment

Xuanwo commented May 18, 2022

youngsofun May 18, 2022

Choose a reason for hiding this comment

sundy-li May 18, 2022

Choose a reason for hiding this comment

sundy-li commented May 18, 2022

Xuanwo commented May 18, 2022

sundy-li commented May 17, 2022 •

edited

Loading

vercel bot commented May 17, 2022 •

edited

Loading

BohuTANG left a comment •

edited

Loading

BohuTANG commented May 18, 2022 •

edited

Loading

BohuTANG commented May 18, 2022 •

edited

Loading