Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2] Add influxDB connector source #2697

Merged
merged 15 commits into from
Oct 14, 2022

Conversation

531651225
Copy link
Contributor

Purpose of this pull request

#1946
Add InfluxDB Connector Source
close #2595

Check list

@hailin0
Copy link
Member

hailin0 commented Sep 14, 2022

can you add e2e-testcase(spark)?

@531651225
Copy link
Contributor Author

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector.
At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@EricJoy2048
Copy link
Member

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

@CalvinKirs
Copy link
Member

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

@TyrantLucifer
Copy link
Member

TyrantLucifer commented Sep 19, 2022

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

Yes, aggre with @CalvinKirs , you can add maven-shade plugin in your pom to rename these jars. BTW, could you please use httpclient, not use okhttp, it's better to unify versions and dependencies for http dependencies. In subsequent iterations, we need to abstract this http-related functionality into public modules and replace them uniformly with the renamed httpclient. What do you think about? @531651225 @CalvinKirs

@531651225
Copy link
Contributor Author

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

Yes, aggre with @CalvinKirs , you can add maven-shade plugin in your pom to rename these jars. BTW, could you please use httpclient, not use okhttp, it's better to unify versions and dependencies for http dependencies. In subsequent iterations, we need to abstract this http-related functionality into public modules and replace them uniformly with the renamed httpclient. What do you think about? @531651225 @CalvinKirs

图片

It seems impossible to replace okhttp with httpclient. The okhttp jar is used internally by the influxdb-java.jar.
Influxdb-java is the only way officially provided for accessing influxdb.

@TyrantLucifer
Copy link
Member

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

Yes, aggre with @CalvinKirs , you can add maven-shade plugin in your pom to rename these jars. BTW, could you please use httpclient, not use okhttp, it's better to unify versions and dependencies for http dependencies. In subsequent iterations, we need to abstract this http-related functionality into public modules and replace them uniformly with the renamed httpclient. What do you think about? @531651225 @CalvinKirs

图片

It seems impossible to replace okhttp with httpclient. The okhttp jar is used internally by the influxdb-java.jar. Influxdb-java is the only way officially provided for accessing influxdb.

This is really bad news. But don't worry, we also have the ultimate weapon, the maven-shade plugin. We can use this plugin to rename packages of okhttp to avoid jar conflict.

@531651225
Copy link
Contributor Author

531651225 commented Sep 19, 2022

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

Yes, aggre with @CalvinKirs , you can add maven-shade plugin in your pom to rename these jars. BTW, could you please use httpclient, not use okhttp, it's better to unify versions and dependencies for http dependencies. In subsequent iterations, we need to abstract this http-related functionality into public modules and replace them uniformly with the renamed httpclient. What do you think about? @531651225 @CalvinKirs

图片 It seems impossible to replace okhttp with httpclient. The okhttp jar is used internally by the influxdb-java.jar. Influxdb-java is the only way officially provided for accessing influxdb.

This is really bad news. But don't worry, we also have the ultimate weapon, the maven-shade plugin. We can use this plugin to rename packages of okhttp to avoid jar conflict.

Thanks for your advice. I'll try. By the way, is there any other case where the shade plugin is used by other connectors

@TyrantLucifer
Copy link
Member

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

Yes, aggre with @CalvinKirs , you can add maven-shade plugin in your pom to rename these jars. BTW, could you please use httpclient, not use okhttp, it's better to unify versions and dependencies for http dependencies. In subsequent iterations, we need to abstract this http-related functionality into public modules and replace them uniformly with the renamed httpclient. What do you think about? @531651225 @CalvinKirs

图片 It seems impossible to replace okhttp with httpclient. The okhttp jar is used internally by the influxdb-java.jar. Influxdb-java is the only way officially provided for accessing influxdb.

This is really bad news. But don't worry, we also have the ultimate weapon, the maven-shade plugin. We can use this plugin to rename packages of okhttp to avoid jar conflict.

Thanks for your advice. I'll try. By the way, is there any other case where the shade plugin is used by other connectors

You can refer to https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html or see seatunnel-config module get the usage of shade plugin.

@531651225
Copy link
Contributor Author

can you add e2e-testcase(spark)?

there is a conflict of the okhttp jar between the Spark jar and the influxdb connector. At present, this problem is solved by specifying the replacement of conflicting packages in the spark jars directory in the document

@CalvinKirs PTAL

We'd better be able to rename these jars @TyrantLucifer WDYT?

Yes, aggre with @CalvinKirs , you can add maven-shade plugin in your pom to rename these jars. BTW, could you please use httpclient, not use okhttp, it's better to unify versions and dependencies for http dependencies. In subsequent iterations, we need to abstract this http-related functionality into public modules and replace them uniformly with the renamed httpclient. What do you think about? @531651225 @CalvinKirs

图片 It seems impossible to replace okhttp with httpclient. The okhttp jar is used internally by the influxdb-java.jar. Influxdb-java is the only way officially provided for accessing influxdb.

This is really bad news. But don't worry, we also have the ultimate weapon, the maven-shade plugin. We can use this plugin to rename packages of okhttp to avoid jar conflict.

Thanks for your advice. I'll try. By the way, is there any other case where the shade plugin is used by other connectors

You can refer to https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html or see seatunnel-config module get the usage of shade plugin.

thks, The conflict has been resolved.
have already added spark E2E, Please help review @hailin0 @CalvinKirs @EricJoy2048 @TyrantLucifer

@EricJoy2048
Copy link
Member

@531651225 does this pr can review again?
If it's ok,@hailin0 @TyrantLucifer @Hisoka-X please reviews this pr again.

@531651225
Copy link
Contributor Author

@531651225 does this pr can review again? If it's ok,@hailin0 @TyrantLucifer @Hisoka-X please reviews this pr again.

@531651225 does this pr can review again? If it's ok,@hailin0 @TyrantLucifer @Hisoka-X please reviews this pr again.

Okay, help me review, thks

Copy link
Member

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

docs/en/connector-v2/source/InfluxDB.md Outdated Show resolved Hide resolved

@Override
public List<InfluxDBSourceSplit> snapshotState(long checkpointId) throws Exception {
return null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@531651225 531651225 requested review from ashulin and removed request for hailin0 October 1, 2022 00:17
if (shouldEnumerate) {
Set<InfluxDBSourceSplit> newSplits = getInfluxDBSplit();

synchronized (stateLock) {
Copy link
Member

@EricJoy2048 EricJoy2048 Oct 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@ashulin ashulin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@EricJoy2048 EricJoy2048 merged commit 1d70ea3 into apache:dev Oct 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants