Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Jdbc] Supporting more ways to configure connection parameters. #5388

Merged
merged 31 commits into from
Oct 18, 2023
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/en/connector-v2/sink/DB2.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ semantics (using XA transaction guarantee).
## Data Type Mapping

| DB2 Data type | SeaTunnel Data type |
|------------------------------------------------------------------------------------------------------|---------------------|---|
|------------------------------------------------------------------------------------------------------|---------------------|
| BOOLEAN | BOOLEAN |
| SMALLINT | SHORT |
| INT<br/>INTEGER<br/> | INTEGER |
Expand Down Expand Up @@ -72,6 +72,7 @@ semantics (using XA transaction guarantee).
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
5 changes: 5 additions & 0 deletions docs/en/connector-v2/sink/Jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it.
| transaction_timeout_sec | Int | No | -1 |
| auto_commit | Boolean | No | true |
| field_ide | String | No | - |
| properties | Map | No | - |
| common-options | | no | - |

### driver [string]
Expand Down Expand Up @@ -143,6 +144,10 @@ The field "field_ide" is used to identify whether the field needs to be converte
synchronizing from the source to the sink. "ORIGINAL" indicates no conversion is needed, "UPPERCASE" indicates
conversion to uppercase, and "LOWERCASE" indicates conversion to lowercase.

### properties

additional connection configuration parameters

### common options

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/Mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ semantics (using XA transaction guarantee).
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| field_ide | String | No | - | Identify whether the field needs to be converted when synchronizing from the source to the sink. `ORIGINAL` indicates no conversion is needed;`UPPERCASE` indicates conversion to uppercase;`LOWERCASE` indicates conversion to lowercase. |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/OceanBase.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/Oracle.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ semantics (using XA transaction guarantee).
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/PostgreSql.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ semantics (using XA transaction guarantee).
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| field_ide | String | No | - | Identify whether the field needs to be converted when synchronizing from the source to the sink. `ORIGINAL` indicates no conversion is needed;`UPPERCASE` indicates conversion to uppercase;`LOWERCASE` indicates conversion to lowercase. |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
127 changes: 63 additions & 64 deletions docs/en/connector-v2/sink/Snowflake.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Snowflake

> JDBC Snowflake Sink Connector
>
> ## Support those engines
>

## Support Those Engines

> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
>
## Key features

## Key features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [cdc](../../concept/connector-v2-features.md)
Expand All @@ -27,8 +27,8 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre

> Please download the support list corresponding to 'Maven' and copy it to the '$SEATNUNNEL_HOME/plugins/jdbc/lib/' working directory<br/>
> For example Snowflake datasource: cp snowflake-connector-java-xxx.jar $SEATNUNNEL_HOME/plugins/jdbc/lib/
>
## Data Type Mapping

## Data Type Mapping

| Snowflake Data type | SeaTunnel Data type |
|-----------------------------------------------------------------------------|---------------------|
Expand Down Expand Up @@ -65,6 +65,7 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

## tips
Expand All @@ -76,68 +77,66 @@ Write data through jdbc. Support Batch mode and Streaming mode, support concurre
### simple:

> This example defines a SeaTunnel synchronization task that automatically generates data through FakeSource and sends it to JDBC Sink. FakeSource generates a total of 16 rows of data (row.num=16), with each row having two fields, name (string type) and age (int type). The final target table is test_table will also be 16 rows of data in the table. Before run this job, you need create database test and table test_table in your snowflake database. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in [Install SeaTunnel](../../start-v2/locally/deployment.md) to install and deploy SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel Engine](../../start-v2/locally/quick-start-seatunnel-engine.md) to run this job.
>
> ```
> # Defining the runtime environment
> env {
> # You can set flink configuration here
> execution.parallelism = 1
> job.mode = "BATCH"
> }
> source {
> # This is a example source plugin **only for test and demonstrate the feature source plugin**
> FakeSource {
> parallelism = 1
> result_table_name = "fake"
> row.num = 16
> schema = {
> fields {
> name = "string"
> age = "int"
> }
> }
> }
> # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
> # please go to https://seatunnel.apache.org/docs/category/source-v2
> }
> transform {
> # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
> # please go to https://seatunnel.apache.org/docs/category/transform-v2
> }
> sink {
> jdbc {
> url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
> driver = "net.snowflake.client.jdbc.SnowflakeDriver"
> user = "root"
> password = "123456"
> query = "insert into test_table(name,age) values(?,?)"
> }
> # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
> # please go to https://seatunnel.apache.org/docs/category/sink-v2
> }
> ```

```
# Defining the runtime environment
env {
# You can set flink configuration here
execution.parallelism = 1
job.mode = "BATCH"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
FakeSource {
parallelism = 1
result_table_name = "fake"
row.num = 16
schema = {
fields {
name = "string"
age = "int"
}
}
}
# If you would like to get more information about how to configure seatunnel and see full list of source plugins,
# please go to https://seatunnel.apache.org/docs/category/source-v2
}
transform {
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/category/transform-v2
}
sink {
jdbc {
url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
driver = "net.snowflake.client.jdbc.SnowflakeDriver"
user = "root"
password = "123456"
query = "insert into test_table(name,age) values(?,?)"
}
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
# please go to https://seatunnel.apache.org/docs/category/sink-v2
}
```

### CDC(Change data capture) event

> CDC change data is also supported by us In this case, you need config database, table and primary_keys.
>
> ```
> jdbc {
> url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
> driver = "net.snowflake.client.jdbc.SnowflakeDriver"
> user = "root"
> password = "123456"
>
> ```

# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]

}

```

sink {
jdbc {
url = "jdbc:snowflake://<account_name>.snowflakecomputing.com"
driver = "net.snowflake.client.jdbc.SnowflakeDriver"
user = "root"
password = "123456"
generate_sink_sql = true


# You need to configure both database and table
database = test
table = sink_table
primary_keys = ["id","name"]
}
}
```

1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/Vertica.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ semantics (using XA transaction guarantee).
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| properties | Map | No | - | Additional connection configuration parameters |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips
Expand Down
Loading
Loading