[Feature][Connector-V2] add tdengine source #2832

lhyundeadsoul · 2022-09-21T09:21:12Z

Purpose of this pull request

refer to #2671

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

Hisoka-X · 2022-09-26T12:27:15Z

Hi, please solve ci problem and conflict. Thanks!

hailin0

can you add e2e testcase?

new version e2e:
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e

OR

old version e2e:
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iotdb-flink-e2e
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-iotdb-spark-e2e

docs/en/connector-v2/source/TDengine.md

seatunnel-connectors-v2-dist/pom.xml

seatunnel-connectors-v2/connector-tdengine/pom.xml

...gine/src/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSink.java

...ain/java/org/apache/seatunnel/connectors/seatunnel/tdengine/source/TDengineSourceReader.java

...org/apache/seatunnel/connectors/seatunnel/tdengine/source/TDengineSourceSplitEnumerator.java

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java

lhyundeadsoul

pre-version is PIW version. Now It's completed.

EricJoy2048 · 2022-10-05T14:17:57Z

Please fix the code style.

docs/en/connector-v2/source/TDengine.md

seatunnel-connectors-v2-dist/pom.xml

docs/en/connector-v2/source/TDengine.md

seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml

...s/seatunnel-flink-connector-v2-example/src/main/resources/examples/tdengine_to_tdengine.conf

seatunnel-connectors-v2-dist/pom.xml

hailin0 · 2022-10-25T02:39:02Z

can you add e2e testcase?

new version e2e: https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e

OR

old version e2e: https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iotdb-flink-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-iotdb-spark-e2e

do you need help？

seatunnel-connectors-v2-dist/pom.xml

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java

EricJoy2048 · 2022-10-25T02:47:26Z

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java

+                + " ) VALUES ("
+                + StringUtils.join(convertDataType(metrics), ",")
+                + ");";
+            final int rowCount = statement.executeUpdate(sql);


Can you use batch flush for jdbc?

org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSinkWriter#write method has only one SeaTunnelRow param, so I think batch flush will be used when we have write(List<SeaTunnelRow>) api.

Whether to support write(List<SeaTunnelRow>) needs to be discussed. Currently ST only supports write (SeaTunnelRow), if you need better performance, you need to implement batch flush yourself.

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java

lhyundeadsoul · 2022-10-26T08:04:51Z

can you add e2e testcase?
new version e2e: https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e
OR
old version e2e: https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iotdb-flink-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-iotdb-spark-e2e

do you need help？

Thx for reminding me . I will do this later.

hailin0

add sink docs
revert example module changes
add e2e testcase

seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml

EricJoy2048 · 2022-11-01T07:10:57Z

Retry CI

lhyundeadsoul · 2022-12-29T08:44:10Z

@EricJoy2048 @hailin0 @Hisoka-X @TyrantLucifer @CalvinKirs
Hey guys, please help to give me a CR if you are free. Thx!

docs/en/connector-v2/sink/TDengine.md

CalvinKirs · 2022-12-30T08:38:45Z

I can't find the previous context, why can't we implement this in jdbc-connector?

lhyundeadsoul · 2022-12-30T14:00:36Z

I can't find the previous context, why can't we implement this in jdbc-connector?

The reason why we implement tdengine connector separately is that there are some distinctive concepts in TDengine that do not belong to JDBC, such as subtable, tags, etc.

hailin0

@lhyundeadsoul

seatunnel-api/src/main/java/org/apache/seatunnel/api/table/type/SqlType.java

seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml

seatunnel-connectors-v2-dist/pom.xml

...use-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/ClickhouseIT.java

seatunnel-connectors-v2/connector-tdengine/pom.xml

…in connecotr.jar

hailin0

LGTM

docs/en/connector-v2/sink/TDengine.md

docs/en/connector-v2/source/TDengine.md

ic4y · 2023-01-07T02:28:34Z

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java

+                + " ) VALUES ("
+                + StringUtils.join(convertDataType(metrics), ",")
+                + ");";
+            final int rowCount = statement.executeUpdate(sql);


Whether to support write(List<SeaTunnelRow>) needs to be discussed. Currently ST only supports write (SeaTunnelRow), if you need better performance, you need to implement batch flush yourself.

...ain/java/org/apache/seatunnel/connectors/seatunnel/tdengine/config/TDengineSourceConfig.java

...ain/java/org/apache/seatunnel/connectors/seatunnel/tdengine/source/TDengineSourceReader.java

ic4y

LGTM

...org/apache/seatunnel/connectors/seatunnel/tdengine/exception/TDengineConnectorErrorCode.java

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/source/TDengineSource.java

...ain/java/org/apache/seatunnel/connectors/seatunnel/tdengine/source/TDengineSourceReader.java

...org/apache/seatunnel/connectors/seatunnel/tdengine/source/TDengineSourceSplitEnumerator.java

...n/java/org/apache/seatunnel/connectors/seatunnel/tdengine/typemapper/TDengineTypeMapper.java

TyrantLucifer

+1 cc @hailin0 @Hisoka-X @ic4y

* [Feature][Connector-V2]add tdengine source and sink 1. add ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY to statement 2. add tdengine e2e module * SourceSplitEnumerator.Context Co-authored-by: bjyflihongyu <lihongyuinfo@jd.com> Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com>

EricJoy2048 changed the title ~~add tdengine source~~ [Feature][Connector-V2] add tdengine source Sep 23, 2022

EricJoy2048 added the connectors-v2 label Sep 23, 2022

hailin0 reviewed Sep 28, 2022

View reviewed changes

lhyundeadsoul force-pushed the td_engine_source branch 2 times, most recently from 081314e to 0048e00 Compare September 30, 2022 09:19

lhyundeadsoul commented Sep 30, 2022

View reviewed changes

hailin0 reviewed Oct 13, 2022

View reviewed changes

docs/en/connector-v2/source/TDengine.md Outdated Show resolved Hide resolved

docs/en/connector-v2/source/TDengine.md Outdated Show resolved Hide resolved

seatunnel-connectors-v2-dist/pom.xml Outdated Show resolved Hide resolved

lhyundeadsoul force-pushed the td_engine_source branch 2 times, most recently from 130e0f4 to f37d682 Compare October 21, 2022 10:31

hailin0 reviewed Oct 25, 2022

View reviewed changes

EricJoy2048 reviewed Oct 25, 2022

View reviewed changes

Hisoka-X reviewed Oct 25, 2022

View reviewed changes

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java Outdated Show resolved Hide resolved

lhyundeadsoul closed this Oct 26, 2022

lhyundeadsoul reopened this Oct 26, 2022

hailin0 reviewed Oct 28, 2022

View reviewed changes

seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml Outdated Show resolved Hide resolved

EricJoy2048 closed this Nov 1, 2022

EricJoy2048 reopened this Nov 1, 2022

lhyundeadsoul force-pushed the td_engine_source branch from 608e4be to 2e18941 Compare December 22, 2022 09:07

CalvinKirs reviewed Dec 30, 2022

View reviewed changes

docs/en/connector-v2/sink/TDengine.md Outdated Show resolved Hide resolved

hailin0 reviewed Jan 3, 2023

View reviewed changes

lhyundeadsoul force-pushed the td_engine_source branch from 18cfcdb to 12bf101 Compare January 3, 2023 10:02

lhyundeadsoul added 2 commits January 6, 2023 17:05

add license

215220f

%s not ?

a6a9355

hailin0 reviewed Jan 7, 2023

View reviewed changes

seatunnel-connectors-v2/connector-tdengine/pom.xml Show resolved Hide resolved

lhyundeadsoul added 2 commits January 7, 2023 10:38

add module to dist

638c6b0

remove taos-jdbc in bin-ci.xml. because it has already been packaged …

35768f1

…in connecotr.jar

hailin0 previously approved these changes Jan 7, 2023

View reviewed changes

ic4y reviewed Jan 7, 2023

View reviewed changes

guaranteed to be exactly-once.

294a1b4

lhyundeadsoul dismissed hailin0’s stale review via 294a1b4 January 7, 2023 03:11

Hisoka-X previously approved these changes Jan 8, 2023

View reviewed changes

ic4y previously approved these changes Jan 9, 2023

View reviewed changes

lhyundeadsoul requested a review from TyrantLucifer January 9, 2023 04:05

TyrantLucifer reviewed Jan 9, 2023

View reviewed changes

use common error code

80da544

lhyundeadsoul dismissed stale reviews from ic4y and Hisoka-X via 80da544 January 9, 2023 06:31

[Feature][Connector-V2][TDEngine] Remove useless code

4298658

TyrantLucifer previously approved these changes Jan 9, 2023

View reviewed changes

github-actions bot added approved reviewed labels Jan 9, 2023

SourceSplitEnumerator.Context

4826ce1

lhyundeadsoul dismissed TyrantLucifer’s stale review via 4826ce1 January 9, 2023 06:47

github-actions bot removed approved reviewed labels Jan 9, 2023

CalvinKirs approved these changes Jan 10, 2023

View reviewed changes

hailin0 approved these changes Jan 10, 2023

View reviewed changes

CalvinKirs merged commit acf4d5b into apache:dev Jan 10, 2023

lhyundeadsoul deleted the td_engine_source branch January 10, 2023 07:40

EricJoy2048 mentioned this pull request Feb 3, 2023

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2] add tdengine source #2832

[Feature][Connector-V2] add tdengine source #2832

lhyundeadsoul commented Sep 21, 2022

Hisoka-X commented Sep 26, 2022

hailin0 left a comment •

edited

Loading

lhyundeadsoul left a comment

EricJoy2048 commented Oct 5, 2022

hailin0 commented Oct 25, 2022

EricJoy2048 Oct 25, 2022

lhyundeadsoul Oct 26, 2022

ic4y Jan 7, 2023

lhyundeadsoul commented Oct 26, 2022

hailin0 left a comment

EricJoy2048 commented Nov 1, 2022

lhyundeadsoul commented Dec 29, 2022 •

edited

Loading

CalvinKirs commented Dec 30, 2022

lhyundeadsoul commented Dec 30, 2022

hailin0 left a comment

hailin0 left a comment

ic4y Jan 7, 2023

ic4y left a comment

TyrantLucifer left a comment

[Feature][Connector-V2] add tdengine source #2832

[Feature][Connector-V2] add tdengine source #2832

Conversation

lhyundeadsoul commented Sep 21, 2022

Purpose of this pull request

Check list

Hisoka-X commented Sep 26, 2022

hailin0 left a comment • edited Loading

Choose a reason for hiding this comment

lhyundeadsoul left a comment

Choose a reason for hiding this comment

EricJoy2048 commented Oct 5, 2022

hailin0 commented Oct 25, 2022

EricJoy2048 Oct 25, 2022

Choose a reason for hiding this comment

lhyundeadsoul Oct 26, 2022

Choose a reason for hiding this comment

ic4y Jan 7, 2023

Choose a reason for hiding this comment

lhyundeadsoul commented Oct 26, 2022

hailin0 left a comment

Choose a reason for hiding this comment

EricJoy2048 commented Nov 1, 2022

lhyundeadsoul commented Dec 29, 2022 • edited Loading

CalvinKirs commented Dec 30, 2022

lhyundeadsoul commented Dec 30, 2022

hailin0 left a comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

ic4y Jan 7, 2023

Choose a reason for hiding this comment

ic4y left a comment

Choose a reason for hiding this comment

TyrantLucifer left a comment

Choose a reason for hiding this comment

hailin0 left a comment •

edited

Loading

lhyundeadsoul commented Dec 29, 2022 •

edited

Loading