[Connector-V2] [Clickhouse] Improve Clickhouse File Connector #3416

Hisoka-X · 2022-11-13T13:51:32Z

Purpose of this pull request

Close [Bug] [Clickhouse-File] clickhouse-file connector-v2 generated data part name conflict #3242
Adjust ClickhouseFile sink to adapt to agg committer
Save memory

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist

docs/en/connector-v2/sink/ClickhouseFile.md

Co-authored-by: Eric <gaojun2048@gmail.com>

...org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java

...rg/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkFactory.java

EricJoy2048 · 2022-11-28T12:19:51Z

...rg/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkFactory.java

@@ -44,6 +46,6 @@ public String factoryIdentifier() {
    @Override
    public OptionRule optionRule() {


Please add UT for optionRule().

# Conflicts: # seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSink.java # seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java

...org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java

Z1Wu · 2022-12-07T15:14:47Z

...org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java

+        String data = this.readerOption.getFields().stream().map(field -> row.getField(this.readerOption.getSeaTunnelRowType().indexOf(field)).toString())
+            .collect(Collectors.joining("\t")) + "\n";
+        MappedByteBuffer buffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, fileChannel.size(),
+            data.getBytes(StandardCharsets.UTF_8).length);


Putting every row into buffer inside mmap region of file channel may cause the executor consumes too much RAM then crushes because of OOM. In my opinion, Clickhouse File Connector is designed for huge data loading scenario where there will be multiples GB of data in every executor.And it is hard to put all these data in memory.
Maybe we can make a tradeoff between speed and memory capacity. We can dump some rows into disk when it accumulate to a certain threshold.

Hi, can you tell me more about mmap consume too much RAM, because we map only one row each time. Anything I didn't know？

I am not very sured about the behaviors of file channel.
But when I using fileChannel.map, spark executors always crash. I currently use the following code as workaround.

ByteBuffer buffer = ByteBuffer.allocate(data.size()); buffer.put(out.toByteArray()); // flip offset buffer.flip(); fileChannel.write(buffer);

Can you provide your crash log? Thanks!

...org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java

docs/en/connector-v2/sink/ClickhouseFile.md

Co-authored-by: hailin0 <hailin088@gmail.com>

Hisoka-X added 3 commits November 13, 2022 21:47

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector

2fd355d

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector

007d898

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector

eb67337

Hisoka-X added improve connectors-v2 labels Nov 13, 2022

EricJoy2048 reviewed Nov 14, 2022

View reviewed changes

docs/en/connector-v2/sink/ClickhouseFile.md Outdated Show resolved Hide resolved

Hisoka-X and others added 2 commits November 14, 2022 10:36

Update docs/en/connector-v2/sink/ClickhouseFile.md

97d36f4

Co-authored-by: Eric <gaojun2048@gmail.com>

[Clickhouse] Change Random number to Context index

8a0993b

Z1Wu reviewed Nov 15, 2022

View reviewed changes

...org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java Outdated Show resolved Hide resolved

Z1Wu reviewed Nov 15, 2022

View reviewed changes

...org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkWriter.java Outdated Show resolved Hide resolved

Hisoka-X self-assigned this Nov 15, 2022

Hisoka-X added the Waiting for code update label Nov 15, 2022

[Connector-V2] [Clickhouse] Improve Clickhouse add compatible mode

903839e

Hisoka-X added waiting for review and removed Waiting for code update labels Nov 22, 2022

EricJoy2048 reviewed Nov 24, 2022

View reviewed changes

...rg/apache/seatunnel/connectors/seatunnel/clickhouse/sink/file/ClickhouseFileSinkFactory.java Outdated Show resolved Hide resolved

Hisoka-X removed their assignment Nov 26, 2022

EricJoy2048 reviewed Nov 28, 2022

View reviewed changes

Hisoka-X self-assigned this Nov 29, 2022

Hisoka-X added Waiting for code update and removed waiting for review labels Nov 29, 2022

Hisoka-X added 2 commits December 5, 2022 17:05

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector

2709630

Hisoka-X added waiting for review and removed Waiting for code update labels Dec 5, 2022

EricJoy2048 previously approved these changes Dec 5, 2022

View reviewed changes

github-actions bot added approved reviewed labels Dec 5, 2022

Hisoka-X closed this Dec 5, 2022

Hisoka-X reopened this Dec 5, 2022

EricJoy2048 requested a review from Z1Wu December 7, 2022 14:31

Z1Wu reviewed Dec 7, 2022

View reviewed changes

EricJoy2048 added this to the 2.3.0 milestone Dec 8, 2022

[Improve][ClickhouseFile] Add ClickhouseFile config

6babe9a

Hisoka-X dismissed EricJoy2048’s stale review via 6babe9a December 8, 2022 08:52

github-actions bot removed approved reviewed labels Dec 8, 2022

[Improve][ClickhouseFile] Add ClickhouseFile config

63be4c8

hailin0 previously approved these changes Dec 9, 2022

View reviewed changes

docs/en/connector-v2/sink/ClickhouseFile.md Outdated Show resolved Hide resolved

github-actions bot added approved reviewed labels Dec 9, 2022

Update docs/en/connector-v2/sink/ClickhouseFile.md

436747d

Co-authored-by: hailin0 <hailin088@gmail.com>

Hisoka-X dismissed hailin0’s stale review via 436747d December 9, 2022 07:05

github-actions bot removed approved reviewed labels Dec 9, 2022

hailin0 approved these changes Dec 9, 2022

View reviewed changes

github-actions bot added approved reviewed labels Dec 9, 2022

EricJoy2048 approved these changes Dec 13, 2022

View reviewed changes

EricJoy2048 merged commit e07e9a7 into apache:dev Dec 13, 2022

Hisoka-X deleted the improve_clickhousefile branch December 13, 2022 05:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector #3416

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector #3416

Hisoka-X commented Nov 13, 2022

EricJoy2048 Nov 28, 2022

Z1Wu Dec 7, 2022

Hisoka-X Dec 8, 2022

Z1Wu Dec 9, 2022

Hisoka-X Dec 10, 2022

		@@ -44,6 +46,6 @@ public String factoryIdentifier() {
		@Override
		public OptionRule optionRule() {

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector #3416

[Connector-V2] [Clickhouse] Improve Clickhouse File Connector #3416

Conversation

Hisoka-X commented Nov 13, 2022

Purpose of this pull request

Check list

EricJoy2048 Nov 28, 2022

Choose a reason for hiding this comment

Z1Wu Dec 7, 2022

Choose a reason for hiding this comment

Hisoka-X Dec 8, 2022

Choose a reason for hiding this comment

Z1Wu Dec 9, 2022

Choose a reason for hiding this comment

Hisoka-X Dec 10, 2022

Choose a reason for hiding this comment