-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Connector-V2] [Amazondynamodb Connector]add amazondynamodb source & sink connnector #3166
Changes from 17 commits
a692d35
08908af
d3fa1b9
7ed060b
5b0ab48
bc881a5
2a6d350
4714319
9f2d18b
24071a3
05c05d9
b586ce9
c0a9751
91ab78b
38692ef
efebf11
cd1ec33
73bb6df
39414cb
964d580
4e1e17c
d1e4f1f
aaf3d1d
5a7ff67
b26981a
1e6e186
97d8026
fca481e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
```` | ||
# Amazondynamodb | ||
|
||
> Amazondynamodb sink connector | ||
|
||
## Description | ||
|
||
Write data to `Amazondynamodb` | ||
|
||
## Key features | ||
|
||
- [x] [batch](../../concept/connector-v2-features.md) | ||
- [ ] [exactly-once](../../concept/connector-v2-features.md) | ||
- [ ] [schema projection](../../concept/connector-v2-features.md) | ||
- [ ] [parallelism](../../concept/connector-v2-features.md) | ||
- [ ] [support user-defined split](../../concept/connector-v2-features.md) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ditto There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
|
||
## Options | ||
|
||
| name | type | required | default value | | ||
|--------------- | ------ |----------| ------------- | | ||
| url | string | yes | - | | ||
| region | string | yes | - | | ||
| accessKeyId | string | yes | - | | ||
| secretAccessKey| string | yes | - | | ||
| table | string | yes | - | | ||
| common-options | | no | - | | ||
|
||
### url [string] | ||
|
||
url to write to Amazondynamodb. | ||
|
||
### region [string] | ||
|
||
The region of Amazondynamodb. | ||
|
||
### accessKeyId [string] | ||
|
||
The access id of Amazondynamodb. | ||
|
||
### secretAccessKey [string] | ||
|
||
The access secret of Amazondynamodb. | ||
|
||
### table [string] | ||
|
||
The table of Amazondynamodb. | ||
|
||
### common options | ||
|
||
Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details. | ||
|
||
## Example | ||
|
||
```bash | ||
Amazondynamodb { | ||
url = "http://127.0.0.1:8000" | ||
region = "us-east-1" | ||
accessKeyId = "dummy-key" | ||
secretAccessKey = "dummy-secret" | ||
table = "TableName" | ||
} | ||
``` | ||
|
||
## Changelog | ||
|
||
### next version | ||
|
||
- Add Amazondynamodb Sink Connector | ||
|
||
```` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,108 @@ | ||
# Amazondynamodb | ||
|
||
> Amazondynamodb source connector | ||
|
||
## Description | ||
|
||
Read data from Amazondynamodb. | ||
|
||
## Key features | ||
|
||
- [x] [batch](../../concept/connector-v2-features.md) | ||
- [ ] [stream](../../concept/connector-v2-features.md) | ||
- [ ] [exactly-once](../../concept/connector-v2-features.md) | ||
- [x] [schema projection](../../concept/connector-v2-features.md) | ||
- [ ] [parallelism](../../concept/connector-v2-features.md) | ||
- [ ] [support user-defined split](../../concept/connector-v2-features.md) | ||
|
||
## Options | ||
|
||
| name | type | required | default value | | ||
| -------------- | ------ | -------- | ------------- | | ||
| url | string | yes | - | | ||
| region | string | yes | - | | ||
| accessKeyId | string | yes | - | | ||
| secretAccessKey| string | yes | - | | ||
| table | string | yes | - | | ||
| schema | object | yes | - | | ||
| common-options | | yes | - | | ||
|
||
### url [string] | ||
|
||
url to read to Amazondynamodb. | ||
|
||
### region [string] | ||
|
||
The region of Amazondynamodb. | ||
|
||
### accessKeyId [string] | ||
|
||
The access id of Amazondynamodb. | ||
|
||
### secretAccessKey [string] | ||
|
||
The access secret of Amazondynamodb. | ||
|
||
### table [string] | ||
|
||
The table of Amazondynamodb. | ||
|
||
### schema [object] | ||
|
||
#### fields [Config] | ||
|
||
Amazon Dynamodb is a NOSQL database service of support keys-value storage and document data structure,there is no way to get the data type.Therefore, we must configure schma. | ||
|
||
such as: | ||
|
||
``` | ||
schema { | ||
fields { | ||
id = int | ||
key_aa = string | ||
key_bb = string | ||
} | ||
} | ||
``` | ||
|
||
### common options | ||
|
||
Source Plugin common parameters, refer to [Source Plugin](common-options.md) for details | ||
|
||
## Example | ||
|
||
```bash | ||
Amazondynamodb { | ||
url = "http://127.0.0.1:8000" | ||
region = "us-east-1" | ||
accessKeyId = "dummy-key" | ||
secretAccessKey = "dummy-secret" | ||
table = "TableName" | ||
schema = { | ||
fields { | ||
artist = string | ||
c_map = "map<string, array<int>>" | ||
c_array = "array<int>" | ||
c_string = string | ||
c_boolean = boolean | ||
c_tinyint = tinyint | ||
c_smallint = smallint | ||
c_int = int | ||
c_bigint = bigint | ||
c_float = float | ||
c_double = double | ||
c_decimal = "decimal(30, 8)" | ||
c_null = "null" | ||
c_bytes = bytes | ||
c_date = date | ||
c_timestamp = timestamp | ||
} | ||
} | ||
} | ||
``` | ||
|
||
## Changelog | ||
|
||
### next version | ||
|
||
- Add Amazondynamodb Source Connector |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<!-- | ||
|
||
Licensed to the Apache Software Foundation (ASF) under one or more | ||
contributor license agreements. See the NOTICE file distributed with | ||
this work for additional information regarding copyright ownership. | ||
The ASF licenses this file to You under the Apache License, Version 2.0 | ||
(the "License"); you may not use this file except in compliance with | ||
the License. You may obtain a copy of the License at | ||
|
||
http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
Unless required by applicable law or agreed to in writing, software | ||
distributed under the License is distributed on an "AS IS" BASIS, | ||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
See the License for the specific language governing permissions and | ||
limitations under the License. | ||
|
||
--> | ||
<project xmlns="http://maven.apache.org/POM/4.0.0" | ||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
<parent> | ||
<artifactId>seatunnel-connectors-v2</artifactId> | ||
<groupId>org.apache.seatunnel</groupId> | ||
<version>${revision}</version> | ||
</parent> | ||
<modelVersion>4.0.0</modelVersion> | ||
|
||
<artifactId>connector-amazondynamodb</artifactId> | ||
|
||
<properties> | ||
<amazon.awssdk>2.18.1</amazon.awssdk> | ||
</properties> | ||
|
||
<dependencyManagement> | ||
<dependencies> | ||
<dependency> | ||
<groupId>software.amazon.awssdk</groupId> | ||
<artifactId>bom</artifactId> | ||
<version>${amazon.awssdk}</version> | ||
<type>pom</type> | ||
<scope>import</scope> | ||
</dependency> | ||
</dependencies> | ||
</dependencyManagement> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>org.apache.seatunnel</groupId> | ||
<artifactId>connector-common</artifactId> | ||
<version>${project.version}</version> | ||
</dependency> | ||
<dependency> | ||
<groupId>software.amazon.awssdk</groupId> | ||
<artifactId>dynamodb-enhanced</artifactId> | ||
</dependency> | ||
<dependency> | ||
<groupId>software.amazon.awssdk</groupId> | ||
<artifactId>dynamodb</artifactId> | ||
</dependency> | ||
</dependencies> | ||
|
||
</project> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.seatunnel.connectors.seatunnel.amazondynamodb.config; | ||
|
||
import java.io.Serializable; | ||
|
||
public class AmazondynamodbConfig implements Serializable { | ||
public static final String URL = "url"; | ||
public static final String REGION = "region"; | ||
public static final String ACCESS_KEY_ID = "accessKeyId"; | ||
public static final String SECRET_ACCESS_KEY = "secretAccessKey"; | ||
public static final String TABLE = "table"; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.seatunnel.connectors.seatunnel.amazondynamodb.config; | ||
|
||
import org.apache.seatunnel.connectors.seatunnel.common.config.CommonConfig; | ||
|
||
import org.apache.seatunnel.shade.com.typesafe.config.Config; | ||
|
||
import lombok.AllArgsConstructor; | ||
import lombok.Data; | ||
|
||
import java.io.Serializable; | ||
|
||
@Data | ||
@AllArgsConstructor | ||
public class AmazondynamodbSourceOptions implements Serializable { | ||
|
||
private String url; | ||
|
||
private String region; | ||
|
||
private String accessKeyId; | ||
|
||
private String secretAccessKey; | ||
|
||
private String table; | ||
|
||
private Config schema; | ||
|
||
public AmazondynamodbSourceOptions(Config config) { | ||
if (config.hasPath(AmazondynamodbConfig.URL)) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seem like these config don't have default value, maybe you should use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You already check it exist, No need to check again There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Done. |
||
this.url = config.getString(AmazondynamodbConfig.URL); | ||
} | ||
if (config.hasPath(AmazondynamodbConfig.REGION)) { | ||
this.region = config.getString(AmazondynamodbConfig.REGION); | ||
} | ||
if (config.hasPath(AmazondynamodbConfig.ACCESS_KEY_ID)) { | ||
this.accessKeyId = config.getString(AmazondynamodbConfig.ACCESS_KEY_ID); | ||
} | ||
if (config.hasPath(AmazondynamodbConfig.SECRET_ACCESS_KEY)) { | ||
this.secretAccessKey = config.getString(AmazondynamodbConfig.SECRET_ACCESS_KEY); | ||
} | ||
if (config.hasPath(AmazondynamodbConfig.TABLE)) { | ||
this.table = config.getString(AmazondynamodbConfig.TABLE); | ||
} | ||
if (config.hasPath(CommonConfig.SCHEMA)) { | ||
this.schema = config.getConfig(CommonConfig.SCHEMA); | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sink Connector Features only contain
exactly-once
andschema projection
.