bug: CSV import and data consistency issue or encoding issue

[alitrack]

Search before asking

• I had searched in the issues and found no similar issues.

Version

0.7.3

What's Wrong?

import from csv may be garbled .

What You Expected?

How to Reproduce?

• create table

create table books
(
    title VARCHAR(255),
    author VARCHAR(255),
    date VARCHAR(255)
);

• import data

echo curl -H \"insert_sql:insert into book_db.books format CSV\" -H \"skip_header:0\" -H \"field_delimiter:','\" -H \"record_delimiter:'\n'\" -F  \"upload=@./books_gbk.csv\" -XPUT http://127.0.0.1:8081/v1/streaming_load|bash

books.zip

Anything Else?

• need encoding check
• ontime has the same issue, because data of year 2001 and month 10 have no ascii
• had better support compressed csv, for example gzip, bzip2
• unaccent support

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[sundy-li]

1. You can use iconv -f GBK -t UTF-8 a_gbk.csv -o b_utf8.csv to convert into utf8 format, databend can't detect the character encoding.

2. Compressiongzip, bzip2 can be supported in the future.

[alitrack]

1. You can use iconv -f GBK -t UTF-8 a_gbk.csv -o b_utf8.csv to convert into utf8 format, databend can't detect the character encoding.
2. Compressiongzip, bzip2 can be supported in the future.

if only support UTF-8, had better mention in the manual, and the tutorial also should do the conversion, the ontime csv is iso-8859-1, has non ascii.

btw, books_gbk.csv is converted with iconv for testing.