Skip to content

Commit

Permalink
Merge branch 'apache:dev' into dev-reformat-mysql-cdc-docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@hailin0
hailin0 authored Aug 30, 2023
2 parents 710e176 + 3c13275 commit f2537e8
Show file tree
Hide file tree
Showing 172 changed files with 730 additions and 760 deletions.
42 changes: 0 additions & 42 deletions .github/workflows/code-analysys.yml
View file

This file was deleted.

2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Please follow this [document](docs/en/contribution/setup.md).

* Mail list: **dev@seatunnel.apache.org**. Mail to `dev-subscribe@seatunnel.apache.org`, follow the reply to subscribe
the mail list.
* Slack: https://the-asf.slack.com/archives/C053HND1D6X
* Slack: https://s.apache.org/seatunnel-slack
* Twitter: https://twitter.com/ASFSeaTunnel
* [Bilibili](https://space.bilibili.com/1542095008) (for Chinese users)

Expand Down
4 changes: 2 additions & 2 deletions bin/install-plugin.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
# get seatunnel home
SEATUNNEL_HOME=$(cd $(dirname $0);cd ../;pwd)

# connector default version is 2.3.1, you can also choose a custom version. eg: 2.1.2: sh install-plugin.sh 2.1.2
version=2.3.1
# connector default version is 2.3.3, you can also choose a custom version. eg: 2.1.2: sh install-plugin.sh 2.1.2
version=2.3.3

if [ -n "$1" ]; then
version="$1"
Expand Down
2 changes: 1 addition & 1 deletion docs/en/about.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<img src="https://seatunnel.apache.org/image/logo.png" alt="seatunnel logo" width="200px" height="200px" align="right" />

[![Slack](https://img.shields.io/badge/slack-%23seatunnel-4f8eba?logo=slack)](https://the-asf.slack.com/archives/C053HND1D6X)
[![Slack](https://img.shields.io/badge/slack-%23seatunnel-4f8eba?logo=slack)](https://s.apache.org/seatunnel-slack)
[![Twitter Follow](https://img.shields.io/twitter/follow/ASFSeaTunnel.svg?label=Follow&logo=twitter)](https://twitter.com/ASFSeaTunnel)

SeaTunnel is a very easy-to-use, ultra-high-performance, distributed data integration platform that supports real-time
Expand Down
1 change: 0 additions & 1 deletion docs/en/connector-v2/sink/AmazonDynamoDB.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ Write data to Amazon DynamoDB
| secret_access_key | string | yes | - |
| table | string | yes | - |
| batch_size | string | no | 25 |
| batch_interval_ms | string | no | 1000 |
| common-options | | no | - |

### url [string]
Expand Down
16 changes: 13 additions & 3 deletions docs/en/connector-v2/sink/Console.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,24 @@ Used to send data to Console. Both support streaming and batch mode.

## Options

| name | type | required | default value |
|----------------|------|----------|---------------|
| common-options | | no | - |
| name | type | required | default value |
|--------------------|---------|----------|---------------|
| common-options | | no | - |
| log.print.data | boolean | no | yes |
| log.print.delay.ms | int | no | 0 |

### common options

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details

### log.print.data

Flag to determine whether data should be printed in the logs. The default value is `true`.

### log.print.delay.ms

Delay in milliseconds between printing each data item to the logs. The default value is `0`.

## Example

simple:
Expand Down
66 changes: 40 additions & 26 deletions docs/en/connector-v2/sink/Feishu.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,41 +2,55 @@

> Feishu sink connector
## Description

Used to launch Feishu web hooks using data.

> For example, if the data from upstream is [`age: 12, name: tyrantlucifer`], the body content is the following: `{"age": 12, "name": "tyrantlucifer"}`
## Support Those Engines

**Tips: Feishu sink only support `post json` webhook and the data from source will be treated as body content in web hook.**
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
## Key features
## Key Features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [cdc](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|----------------|--------|----------|---------------|
| url | String | Yes | - |
| headers | Map | No | - |
| common-options | | no | - |

### url [string]

Feishu webhook url

### headers [Map]

Http request headers
## Description

### common options
Used to launch Feishu web hooks using data.

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
> For example, if the data from upstream is [`age: 12, name: tyrantlucifer`], the body content is the following: `{"age": 12, "name": "tyrantlucifer"}`
## Example
**Tips: Feishu sink only support `post json` webhook and the data from source will be treated as body content in web hook.**

simple:
## Data Type Mapping

| Seatunnel Data type | Feishu Data type |
|-----------------------------|------------------|
| ROW<br/>MAP | Json |
| NULL | null |
| BOOLEAN | boolean |
| TINYINT | byte |
| SMALLINT | short |
| INT | int |
| BIGINT | long |
| FLOAT | float |
| DOUBLE | double |
| DECIMAL | BigDecimal |
| BYTES | byte[] |
| STRING | String |
| TIME<br/>TIMESTAMP<br/>TIME | String |
| ARRAY | JsonArray |

## Sink Options

| Name | Type | Required | Default | Description |
|----------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------|
| url | String | Yes | - | Feishu webhook url |
| headers | Map | No | - | Http request headers |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

## Task Example

### Simple:

```hocon
Feishu {
Expand Down
7 changes: 1 addition & 6 deletions docs/en/connector-v2/sink/InfluxDB.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ Write data to InfluxDB.
| key_time | string | no | processing time |
| key_tags | array | no | exclude `field` & `key_time` |
| batch_size | int | no | 1024 |
| batch_interval_ms | int | no | - |
| max_retries | int | no | - |
| retry_backoff_multiplier_ms | int | no | - |
| connect_timeout_ms | long | no | 15000 |
Expand Down Expand Up @@ -63,11 +62,7 @@ If not specified, include all fields with `influxDB` measurement field

### batch_size [int]

For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the influxDB

### batch_interval_ms [int]

For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the influxDB
For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `checkpoint.interval`, the data will be flushed into the influxDB

### max_retries [int]

Expand Down
157 changes: 54 additions & 103 deletions docs/en/connector-v2/sink/IoTDB.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,121 +2,73 @@

> IoTDB sink connector
## Description

Used to write data to IoTDB.

:::tip

There is a conflict of thrift version between IoTDB and Spark.Therefore, you need to execute `rm -f $SPARK_HOME/jars/libthrift*` and `cp $IOTDB_HOME/lib/libthrift* $SPARK_HOME/jars/` to resolve it.
## Support Those Engines

:::
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
## Key features
## Key Features

- [x] [exactly-once](../../concept/connector-v2-features.md)

IoTDB supports the `exactly-once` feature through idempotent writing. If two pieces of data have
the same `key` and `timestamp`, the new data will overwrite the old one.

## Options

| name | type | required | default value |
|-----------------------------|---------|----------|--------------------------------|
| node_urls | list | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| key_device | string | yes | - |
| key_timestamp | string | no | processing time |
| key_measurement_fields | array | no | exclude `device` & `timestamp` |
| storage_group | string | no | - |
| batch_size | int | no | 1024 |
| batch_interval_ms | int | no | - |
| max_retries | int | no | - |
| retry_backoff_multiplier_ms | int | no | - |
| max_retry_backoff_ms | int | no | - |
| default_thrift_buffer_size | int | no | - |
| max_thrift_frame_size | int | no | - |
| zone_id | string | no | - |
| enable_rpc_compression | boolean | no | - |
| connection_timeout_in_ms | int | no | - |
| common-options | | no | - |

### node_urls [list]

`IoTDB` cluster address, the format is `["host:port", ...]`

### username [string]

`IoTDB` user username

### password [string]

`IoTDB` user password

### key_device [string]

Specify field name of the `IoTDB` deviceId in SeaTunnelRow

### key_timestamp [string]

Specify field-name of the `IoTDB` timestamp in SeaTunnelRow. If not specified, use processing-time as timestamp

### key_measurement_fields [array]

Specify field-name of the `IoTDB` measurement list in SeaTunnelRow. If not specified, include all fields but exclude `device` & `timestamp`

### storage_group [string]

Specify device storage group(path prefix)

example: deviceId = ${storage_group} + "." + ${key_device}

### batch_size [int]

For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the IoTDB

### batch_interval_ms [int]

For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the IoTDB

### max_retries [int]

The number of retries to flush failed

### retry_backoff_multiplier_ms [int]

Using as a multiplier for generating the next delay for backoff

### max_retry_backoff_ms [int]

The amount of time to wait before attempting to retry a request to `IoTDB`

### default_thrift_buffer_size [int]

Thrift init buffer size in `IoTDB` client

### max_thrift_frame_size [int]

Thrift max frame size in `IoTDB` client

### zone_id [string]

java.time.ZoneId in `IoTDB` client

### enable_rpc_compression [boolean]

Enable rpc compression in `IoTDB` client
## Description

### connection_timeout_in_ms [int]
Used to write data to IoTDB.

The maximum time (in ms) to wait when connecting to `IoTDB`
:::tip

### common options
There is a conflict of thrift version between IoTDB and Spark.Therefore, you need to execute `rm -f $SPARK_HOME/jars/libthrift*` and `cp $IOTDB_HOME/lib/libthrift* $SPARK_HOME/jars/` to resolve it.

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
:::

## Examples
## Supported DataSource Info

| Datasource | Supported Versions | Url |
|------------|--------------------|----------------|
| IoTDB | `>= 0.13.0` | localhost:6667 |

## Database Dependency

## Data Type Mapping

| IotDB Data type | SeaTunnel Data type |
|-----------------|---------------------|
| BOOLEAN | BOOLEAN |
| INT32 | TINYINT |
| INT32 | SMALLINT |
| INT32 | INT |
| INT64 | BIGINT |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| TEXT | STRING |

## Sink Options

| Name | Type | Required | Default | Description |
|-----------------------------|---------|----------|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| node_urls | Array | Yes | - | `IoTDB` cluster address, the format is `["host:port", ...]` |
| username | String | Yes | - | `IoTDB` user username |
| password | String | Yes | - | `IoTDB` user password |
| key_device | String | No | - | Specify field name of the `IoTDB` deviceId in SeaTunnelRow |
| key_timestamp | String | No | processing time | Specify field-name of the `IoTDB` timestamp in SeaTunnelRow. If not specified, use processing-time as timestamp |
| key_measurement_fields | Array | No | exclude `device` & `timestamp` | Specify field-name of the `IoTDB` measurement list in SeaTunnelRow. If not specified, include all fields but exclude `device` & `timestamp` |
| storage_group | Array | No | - | Specify device storage group(path prefix) <br/> example: deviceId = ${storage_group} + "." + ${key_device} |
| batch_size | Integer | No | 1024 | For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the IoTDB |
| max_retries | Integer | No | - | The number of retries to flush failed |
| retry_backoff_multiplier_ms | Integer | No | - | Using as a multiplier for generating the next delay for backoff |
| max_retry_backoff_ms | Integer | No | - | The amount of time to wait before attempting to retry a request to `IoTDB` |
| default_thrift_buffer_size | Integer | No | - | Thrift init buffer size in `IoTDB` client |
| max_thrift_frame_size | Integer | No | - | Thrift max frame size in `IoTDB` client |
| zone_id | string | No | - | java.time.ZoneId in `IoTDB` client |
| enable_rpc_compression | Boolean | No | - | Enable rpc compression in `IoTDB` client |
| connection_timeout_in_ms | Integer | No | - | The maximum time (in ms) to wait when connecting to `IoTDB` |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

## Task Example

### Case1

Expand All @@ -129,7 +81,6 @@ sink {
username = "root"
password = "root"
batch_size = 1024
batch_interval_ms = 1000
}
}
```
Expand Down
Loading

0 comments on commit f2537e8

Please sign in to comment.