[Feature][Connector-V2] Support Druid Source & Sink #2937
Conversation
… Druid Source&Sink
|
|
||
| > Druid sink connector | ||
|
|
||
| ## Description |
There was a problem hiding this comment.
Please add the Key Features reference https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/sink/Clickhouse.md
| | timestamp_format | `String` | no | auto | | ||
| | timestamp_missing_value | `String` | no | - | | ||
|
|
||
| ### coordinator_url [`String`] |
There was a problem hiding this comment.
Please use the same format as other documents https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/sink/Clickhouse.md
|
|
||
| > Druid source connector | ||
|
|
||
| ## Description |
| @@ -198,6 +198,12 @@ | |||
| <snappy-java.version>1.1.8.3</snappy-java.version> | |||
| <checker.qual.version>3.10.0</checker.qual.version> | |||
| <awaitility.version>4.2.0</awaitility.version> | |||
| <httpclient.version>4.5.13</httpclient.version> | |||
| <fastjson.version>1.2.83</fastjson.version> | |||
There was a problem hiding this comment.
Please manager the jars which used by connector in connector pom.xml like other connector.
|
|
||
| <dependency> | ||
| <groupId>com.alibaba</groupId> | ||
| <artifactId>fastjson</artifactId> |
There was a problem hiding this comment.
Suggest use jackson replace fastjson.
| </dependency> | ||
| <dependency> | ||
| <groupId>com.alibaba</groupId> | ||
| <artifactId>fastjson</artifactId> |
| statement = connection.prepareStatement(quarySQL, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); | ||
| return statement.getMetaData(); | ||
| } catch (SQLException se) { | ||
| throw new SQLException("ResultSetMetaData() failed." + se.getMessage(), se); |
There was a problem hiding this comment.
Please use ExceptionUtils.getMessage() replace se.getMessage()
| resultSet = statement.executeQuery(); | ||
| hasNext = resultSet.next(); | ||
| } catch (SQLException se) { | ||
| throw new IllegalArgumentException("openInputFormat() failed." + se.getMessage(), se); |
|
|
||
| private SeaTunnelRowType initTableField() { | ||
| ArrayList<SeaTunnelDataType<?>> seaTunnelDataTypes = new ArrayList<>(); | ||
| ArrayList<String> fieldNames = new ArrayList<>(); |
| LOGGER.info("Druid write task has been sent, and the response is {}", response.toString()); | ||
| } | ||
| } catch (IOException e) { | ||
| e.printStackTrace(); |
There was a problem hiding this comment.
Please don't use e.printStackTrace() and use logger to output exception messages.
| private Integer parallelism; | ||
|
|
||
| public DruidSourceOptions(Config pluginConfig) { | ||
| this.url = pluginConfig.getString(DruidSourceConfig.URL); |
There was a problem hiding this comment.
Please check whether params exists before get it.
| private int parallelism; | ||
|
|
||
| public DruidSinkOptions(Config pluginConfig) { | ||
| this.coordinatorURL = pluginConfig.getString(DruidSinkConfig.COORDINATOR_URL); |
There was a problem hiding this comment.
Please check whether params exists before get it.
| druidInputFormat = new DruidInputFormat(druidSourceOptions); | ||
| this.rowTypeInfo = druidInputFormat.getRowTypeInfo(); | ||
| } catch (Exception e) { | ||
| throw new PrepareFailException("Druid", PluginType.SOURCE, e.toString()); |
There was a problem hiding this comment.
Replace e.toString() with ExceptionUtils.getMessage(e)
ashulin
changed the title
…eview Code problem
hi bro，development e2e testcase,There are some problems with my environment. this is message: |
…esolved conflicting files pom.xml
…esolved conflicting files
…esolved conflicting files
|
Hi, @guanboo Did you fix the problems? |
Yes,done |
|
@guanboo please fix CI error and resolve conflicts |
|
hi brother,did you continue this PR? |
|
@guanboo Will this pr go ahead? If not, I would like to take over and finish it |
yes,brother. the pr is going,the connector is complete，but e2e is not completely, I has finished the reader of e2e,and guanboo try to finished the writer of e2e, but he is so busy to do it.I not similar with the writer for druid. |
if you so similar with druid you can continue it or another pr，guanboo and I are so busy for our work. |
fix [Feature][Connector] New SeaTunnel API Connectors #1946 Add Druid Source & Sink