Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] [File Connector] Supports writing column names when the output type is file (CSV) #5459

Merged
merged 51 commits into from
Sep 25, 2023

Conversation

zck573693104
Copy link
Contributor

@zck573693104 zck573693104 commented Sep 11, 2023

close #5443

Purpose of this pull request

Check list

@@ -133,15 +133,21 @@ private FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {
OutputStream out =
lzo.createOutputStream(fileSystemUtils.getOutputStream(filePath));
fsDataOutputStream = new FSDataOutputStream(out, null);
fsDataOutputStream.write(String.join(",", seaTunnelRowType.getFieldNames()).getBytes());
fsDataOutputStream.write(System.lineSeparator().getBytes());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use fsDataOutputStream.write(rowDelimiter.getBytes()); ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah,i will fix it

break;
case NONE:
fsDataOutputStream = fileSystemUtils.getOutputStream(filePath);
fsDataOutputStream.write(String.join(",", seaTunnelRowType.getFieldNames()).getBytes());
fsDataOutputStream.write(System.lineSeparator().getBytes());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@EricJoy2048
Copy link
Member

Please fix code style by run sh tools/spotless_check/pre-commit.sh

@zck573693104
Copy link
Contributor Author

Please fix code style by run sh tools/spotless_check/pre-commit.sh

i will fix it

@ruanwenjun ruanwenjun changed the title [Feature] [File Connector] Supports writing column names when the out… [Feature] [File Connector] Supports writing column names when the output type is file (CSV) Sep 14, 2023
@@ -166,6 +167,10 @@ When File Format is Excel,The maximum number of data items that can be cached in

Writer the sheet of the workbook

### enable_header_write [boolean]

false:dont write header,true:write header.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems you only implement this in text format, if so it's needed to mention this only support in text format.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah,i will fix it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

support text and csv

@ruanwenjun ruanwenjun added the feature New feature label Sep 14, 2023
@EricJoy2048
Copy link
Member

Please fix CI error.

@zck573693104
Copy link
Contributor Author

Please fix CI error.

Please fix CI error.

yeah. fix it ok.

docs/en/connector-v2/sink/LocalFile.md Outdated Show resolved Hide resolved
docs/en/connector-v2/sink/LocalFile.md Outdated Show resolved Hide resolved
Copy link
Contributor

@FlechazoW FlechazoW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job!

@liugddx liugddx merged commit f73b372 into apache:dev Sep 25, 2023
8 checks passed
gnehil pushed a commit to gnehil/seatunnel that referenced this pull request Oct 12, 2023
…put type is file (CSV) (apache#5459)

* [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443

* [Feature] [File Connector] fix code style and lineSeparator apache#5443

* [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443

* [Feature] [File Connector] fix code style apache#5443

* [Feature] [File Connector] add enable_header_write explain apache#5443

* [Feature] [File Connector]fix code style apache#5443

* Update docs/en/connector-v2/sink/LocalFile.md

* Update docs/en/connector-v2/sink/LocalFile.md

* [Feature] [File Connector]fix code style apache#5443

* [Feature] [File Connector]add junit test apache#5443

* [Feature] [File Connector]add license header: apache#5443

* [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443

* [Feature] [File Connector] fix code style and lineSeparator apache#5443

* [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443

* [Feature] [File Connector] fix code style apache#5443

* [Feature] [File Connector] add enable_header_write explain apache#5443

* [Feature] [File Connector]fix code style apache#5443

* Update docs/en/connector-v2/sink/LocalFile.md

* Update docs/en/connector-v2/sink/LocalFile.md

* [Feature] [File Connector]fix code style apache#5443

* [Feature] [File Connector]add junit test apache#5443

* [Feature] [File Connector]add license header: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]remove scala: apache#5443

* [Feature] [File Connector]modify md style: apache#5443

* [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443

* [Feature] [File Connector] fix code style and lineSeparator apache#5443

* [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443

* [Feature] [File Connector] fix code style apache#5443

* [Feature] [File Connector] add enable_header_write explain apache#5443

* [Feature] [File Connector]fix code style apache#5443

* Update docs/en/connector-v2/sink/LocalFile.md

* Update docs/en/connector-v2/sink/LocalFile.md

* [Feature] [File Connector]fix code style apache#5443

* [Feature] [File Connector]add junit test apache#5443

* [Feature] [File Connector]add license header: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]remove scala: apache#5443

* [Feature] [File Connector]modify md style: apache#5443

* [Feature] [File Connector]junit modify: apache#5443

---------

Co-authored-by: zck <573693104@qq.com>
Co-authored-by: Eric <gaojun2048@gmail.com>
ruanwenjun pushed a commit to ruanwenjun/incubator-seatunnel that referenced this pull request Nov 6, 2023
…put type is file (CSV) (apache#5459)

* [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443

* [Feature] [File Connector] fix code style and lineSeparator apache#5443

* [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443

* [Feature] [File Connector] fix code style apache#5443

* [Feature] [File Connector] add enable_header_write explain apache#5443

* [Feature] [File Connector]fix code style apache#5443

* Update docs/en/connector-v2/sink/LocalFile.md

* Update docs/en/connector-v2/sink/LocalFile.md

* [Feature] [File Connector]fix code style apache#5443

* [Feature] [File Connector]add junit test apache#5443

* [Feature] [File Connector]add license header: apache#5443

* [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443

* [Feature] [File Connector] fix code style and lineSeparator apache#5443

* [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443

* [Feature] [File Connector] fix code style apache#5443

* [Feature] [File Connector] add enable_header_write explain apache#5443

* [Feature] [File Connector]fix code style apache#5443

* Update docs/en/connector-v2/sink/LocalFile.md

* Update docs/en/connector-v2/sink/LocalFile.md

* [Feature] [File Connector]fix code style apache#5443

* [Feature] [File Connector]add junit test apache#5443

* [Feature] [File Connector]add license header: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]remove scala: apache#5443

* [Feature] [File Connector]modify md style: apache#5443

* [Feature] [File Connector] Supports writing column names when the output type is file (CSV) apache#5443

* [Feature] [File Connector] fix code style and lineSeparator apache#5443

* [Feature] [File Connector] add enable_header_write,false:dont write header,true:write header. apache#5443

* [Feature] [File Connector] fix code style apache#5443

* [Feature] [File Connector] add enable_header_write explain apache#5443

* [Feature] [File Connector]fix code style apache#5443

* Update docs/en/connector-v2/sink/LocalFile.md

* Update docs/en/connector-v2/sink/LocalFile.md

* [Feature] [File Connector]fix code style apache#5443

* [Feature] [File Connector]add junit test apache#5443

* [Feature] [File Connector]add license header: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]add junit: apache#5443

* [Feature] [File Connector]remove scala: apache#5443

* [Feature] [File Connector]modify md style: apache#5443

* [Feature] [File Connector]junit modify: apache#5443

---------

Co-authored-by: zck <573693104@qq.com>
Co-authored-by: Eric <gaojun2048@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][File Connector] Supports writing column names when the output type is file (CSV)
7 participants