Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Jdbc] Supporting more ways to configure connection parameters. #5388

Merged
merged 31 commits into from
Oct 18, 2023
Merged

[Feature][Jdbc] Supporting more ways to configure connection parameters. #5388

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
e4511f8
[Feature][Jdbc] Supporting more ways to configure connection parameters.
Aug 28, 2023
9a4c6f9
[Feature][JDBC] Default parameter
Sep 11, 2023
d6fae86
[Feature][Jdbc] Supporting more ways to configure connection parameters
Sep 13, 2023
46da741
[Feature][Jdbc] Supporting more ways to configure connection parameters
Sep 13, 2023
bef925c
Merge branch 'apache:dev' into jdbc-p
Carl-Zhou-CN Sep 15, 2023
4aea653
[Feature][Jdbc] Supporting more ways to configure connection parameters
Sep 18, 2023
ad81a87
[Feature][jdbc] UT
Sep 21, 2023
478c385
[Feature][jdbc] UT
Sep 21, 2023
7046537
[Feature][jdbc] UT
Sep 21, 2023
98eb00c
[Feature][jdbc] UT
Sep 21, 2023
38d164f
[Feature][jdbc] UT
Sep 22, 2023
c887084
Merge branch 'apache:dev' into jdbc-p
Carl-Zhou-CN Sep 22, 2023
d7e4703
[Feature][jdbc] UT
Sep 24, 2023
5dbe8b8
[Feature][jdbc] UT
Sep 24, 2023
16ef200
[Feature][jdbc] UT
Sep 25, 2023
6f2a781
[Feature][cdc] UT
Oct 10, 2023
4c78b1b
[Feature][cdc] UT
Oct 10, 2023
7d606da
[Feature][cdc] UT
Oct 10, 2023
bc6a98a
[Feature][cdc] UT
Oct 10, 2023
9169f2d
[Feature][cdc] UT
Oct 10, 2023
886a39a
[Feature][cdc] UT
Oct 10, 2023
0b0d9e1
[Feature][cdc] UT
Oct 11, 2023
e568fdf
Merge branch 'apache:dev' into jdbc-p
Carl-Zhou-CN Oct 11, 2023
e368b60
[Improve][test] UT
Oct 11, 2023
de876f0
Merge branch 'apache:dev' into jdbc-p
Carl-Zhou-CN Oct 11, 2023
304c3de
UT
Oct 12, 2023
0968ef4
UT
Oct 12, 2023
6b52905
fix
Oct 16, 2023
ac1211d
fix
Oct 17, 2023
53488a9
fix
Oct 17, 2023
2e69042
fix
Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/en/connector-v2/sink/DB2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ semantics (using XA transaction guarantee).
## Data Type Mapping

| DB2 Data type | SeaTunnel Data type |
|------------------------------------------------------------------------------------------------------|---------------------|---|
|------------------------------------------------------------------------------------------------------|---------------------|
| BOOLEAN | BOOLEAN |
| SMALLINT | SHORT |
| INT<br/>INTEGER<br/> | INTEGER |
Expand Down Expand Up @@ -72,6 +72,7 @@ semantics (using XA transaction guarantee).
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
5 changes: 5 additions & 0 deletions docs/en/connector-v2/sink/Jdbc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it.
| transaction_timeout_sec | Int | No | -1 |
| auto_commit | Boolean | No | true |
| field_ide | String | No | - |
| properties | Map | No | - |
| common-options | | no | - |

### driver [string]
Expand Down Expand Up @@ -143,6 +144,10 @@ The field "field_ide" is used to identify whether the field needs to be converte
synchronizing from the source to the sink. "ORIGINAL" indicates no conversion is needed, "UPPERCASE" indicates
conversion to uppercase, and "LOWERCASE" indicates conversion to lowercase.

### properties

additional connection configuration parameters

### common options

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/Mysql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ semantics (using XA transaction guarantee).
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| field_ide | String | No | - | Identify whether the field needs to be converted when synchronizing from the source to the sink. `ORIGINAL` indicates no conversion is needed;`UPPERCASE` indicates conversion to uppercase;`LOWERCASE` indicates conversion to lowercase. |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/OceanBase.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/Oracle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ semantics (using XA transaction guarantee).
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/PostgreSql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ semantics (using XA transaction guarantee).
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| field_ide | String | No | - | Identify whether the field needs to be converted when synchronizing from the source to the sink. `ORIGINAL` indicates no conversion is needed;`UPPERCASE` indicates conversion to uppercase;`LOWERCASE` indicates conversion to lowercase. |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
127 changes: 63 additions & 64 deletions docs/en/connector-v2/sink/Snowflake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Snowflake

> JDBC Snowflake Sink Connector
>
> ## Support those engines
>

## Support Those Engines

> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
>
## Key features

## Key features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [cdc](../../concept/connector-v2-features.md)
Expand All @@ -27,8 +27,8 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre

> Please download the support list corresponding to 'Maven' and copy it to the '$SEATNUNNEL_HOME/plugins/jdbc/lib/' working directory<br/>
> For example Snowflake datasource: cp snowflake-connector-java-xxx.jar $SEATNUNNEL_HOME/plugins/jdbc/lib/
>
## Data Type Mapping

## Data Type Mapping

| Snowflake Data type | SeaTunnel Data type |
|-----------------------------------------------------------------------------|---------------------|
Expand Down Expand Up @@ -65,6 +65,7 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

## tips
Expand All @@ -76,68 +77,66 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre
### simple:

> This example defines a SeaTunnel synchronization task that automatically generates data through FakeSource and sends it to JDBC Sink. FakeSource generates a total of 16 rows of data (row.num=16), with each row having two fields, name (string type) and age (int type). The final target table is test_table will also be 16 rows of data in the table. Before run this job, you need create database test and table test_table in your snowflake database. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in [Install SeaTunnel](../../start-v2/locally/deployment.md) to install and deploy SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel Engine](../../start-v2/locally/quick-start-seatunnel-engine.md) to run this job.
>
> ```
> # Defining the runtime environment
> env {
> # You can set flink configuration here
> execution.parallelism = 1
> job.mode = "BATCH"
> }
> source {
> # This is a example source plugin **only for test and demonstrate the feature source plugin**
> FakeSource {
> parallelism = 1
> result_table_name = "fake"
> row.num = 16
> schema = {
> fields {
> name = "string"
> age = "int"
> }
> }
> }
> # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
> # please go to https://seatunnel.apache.org/docs/category/source-v2
> }
> transform {
> # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
> # please go to https://seatunnel.apache.org/docs/category/transform-v2
> }
> sink {
> jdbc {
> url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
> driver = "net.snowflake.client.jdbc.SnowflakeDriver"
> user = "root"
> password = "123456"
> query = "insert into test_table(name,age) values(?,?)"
> }
> # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
> # please go to https://seatunnel.apache.org/docs/category/sink-v2
> }
> ```

```
# Defining the runtime environment
env {
# You can set flink configuration here
execution.parallelism = 1
job.mode = "BATCH"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
FakeSource {
parallelism = 1
result_table_name = "fake"
row.num = 16
schema = {
fields {
name = "string"
age = "int"
}
}
}
# If you would like to get more information about how to configure seatunnel and see full list of source plugins,
# please go to https://seatunnel.apache.org/docs/category/source-v2
}
transform {
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/category/transform-v2
}
sink {
jdbc {
url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
driver = "net.snowflake.client.jdbc.SnowflakeDriver"
user = "root"
password = "123456"
query = "insert into test_table(name,age) values(?,?)"
}
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
# please go to https://seatunnel.apache.org/docs/category/sink-v2
}
```

### CDC(Change data capture) event

> CDC change data is also supported by us In this case, you need config database, table and primary_keys.
>
> ```
> jdbc {
> url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
> driver = "net.snowflake.client.jdbc.SnowflakeDriver"
> user = "root"
> password = "123456"
>
> ```

# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]

}

```

sink {
jdbc {
url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
driver = "net.snowflake.client.jdbc.SnowflakeDriver"
user = "root"
password = "123456"
generate_sink_sql = true


# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]
}
}
```

1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/Vertica.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ semantics (using XA transaction guarantee).
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
Loading
Loading