Skip to content

Commit

Permalink
[Improve][Connector-V2] Redis support custom key and value (apache#7888)
Browse files Browse the repository at this point in the history 
Co-authored-by: limin <limin@wedobest.com.cn>
  • Loading branch information
@lm-ylj
lm-ylj and limin authored Oct 29, 2024
1 parent dde6f9f commit ef2c3c7
Show file tree
Hide file tree
Showing 13 changed files with 1,182 additions and 47 deletions.
134 changes: 115 additions & 19 deletions docs/en/connector-v2/sink/Redis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,25 @@ Used to write data to Redis.

## Options

| name | type | required | default value |
|----------------|--------|-----------------------|---------------|
| host | string | yes | - |
| port | int | yes | - |
| key | string | yes | - |
| data_type | string | yes | - |
| batch_size | int | no | 10 |
| user | string | no | - |
| auth | string | no | - |
| db_num | int | no | 0 |
| mode | string | no | single |
| nodes | list | yes when mode=cluster | - |
| format | string | no | json |
| expire | long | no | -1 |
| common-options | | no | - |
| name | type | required | default value |
|--------------------|---------|-----------------------|---------------|
| host | string | yes | - |
| port | int | yes | - |
| key | string | yes | - |
| data_type | string | yes | - |
| batch_size | int | no | 10 |
| user | string | no | - |
| auth | string | no | - |
| db_num | int | no | 0 |
| mode | string | no | single |
| nodes | list | yes when mode=cluster | - |
| format | string | no | json |
| expire | long | no | -1 |
| support_custom_key | boolean | no | false |
| value_field | string | no | - |
| hash_key_field | string | no | - |
| hash_value_field | string | no | - |
| common-options | | no | - |

### host [string]

Expand All @@ -50,12 +54,12 @@ Upstream data is the following:
| 500 | internal error | false |

If you assign field name to `code` and data_type to `key`, two data will be written to redis:
1. `200 -> {code: 200, message: true, data: get success}`
2. `500 -> {code: 500, message: false, data: internal error}`
1. `200 -> {code: 200, data: get success, success: true}`
2. `500 -> {code: 500, data: internal error, success: false}`

If you assign field name to `value` and data_type to `key`, only one data will be written to redis because `value` is not existed in upstream data's fields:

1. `value -> {code: 500, message: false, data: internal error}`
1. `value -> {code: 500, data: internal error, success: false}`

Please see the data_type section for specific writing rules.

Expand Down Expand Up @@ -85,7 +89,7 @@ Redis data types, support `key` `hash` `list` `set` `zset`

> Each data from upstream will be added to the configured zset key with a weight of 1. So the order of data in zset is based on the order of data consumption.
>
### batch_size [int]
### batch_size [int]

ensure the batch write size in single-machine mode; no guarantees in cluster mode.

Expand Down Expand Up @@ -135,6 +139,61 @@ Connector will generate data as the following and write it to redis:

Set redis expiration time, the unit is second. The default value is -1, keys do not automatically expire by default.

### support_custom_key [boolean]

if true, the key can be customized by the field value in the upstream data.

Upstream data is the following:

| code | data | success |
|------|----------------|---------|
| 200 | get success | true |
| 500 | internal error | false |

You can customize the Redis key using '{' and '}', and the field name in '{}' will be parsed and replaced by the field value in the upstream data. For example, If you assign field name to `{code}` and data_type to `key`, two data will be written to redis:
1. `200 -> {code: 200, data: get success, success: true}`
2. `500 -> {code: 500, data: internal error, success: false}`

Redis key can be composed of fixed and variable parts, connected by ':'. For example, If you assign field name to `code:{code}` and data_type to `key`, two data will be written to redis:
1. `code:200 -> {code: 200, data: get success, success: true}`
2. `code:500 -> {code: 500, data: internal error, success: false}`

### value_field [string]

The field of value you want to write to redis, `data_type` support `key` `list` `set` `zset`.

When you assign field name to `value` and value_field is `data` and data_type to `key`, for example:

Upstream data is the following:

| code | data | success |
|------|-------------|---------|
| 200 | get success | true |

The following data will be written to redis:
1. `value -> get success`

### hash_key_field [string]

The field of hash key you want to write to redis, `data_type` support `hash`

### hash_value_field [string]

The field of hash value you want to write to redis, `data_type` support `hash`

When you assign field name to `value` and hash_key_field is `data` and hash_value_field is `success` and data_type to `hash`, for example:

Upstream data is the following:

| code | data | success |
|------|-------------|---------|
| 200 | get success | true |

Connector will generate data as the following and write it to redis:

The following data will be written to redis:
1. `value -> get success | true`

### common options

Sink plugin common parameters, please refer to [Sink Common Options](../sink-common-options.md) for details
Expand All @@ -152,6 +211,43 @@ Redis {
}
```

custom key:

```hocon
Redis {
host = localhost
port = 6379
key = "name:{name}"
support_custom_key = true
data_type = key
}
```

custom value:

```hocon
Redis {
host = localhost
port = 6379
key = person
value_field = "name"
data_type = key
}
```

custom HashKey and HashValue:

```hocon
Redis {
host = localhost
port = 6379
key = person
hash_key_field = "name"
hash_value_field = "age"
data_type = hash
}
```

## Changelog

### 2.2.0-beta 2022-09-26
Expand Down
137 changes: 118 additions & 19 deletions docs/zh/connector-v2/sink/Redis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,25 @@

## 选项

| 名称 | 类型 | 是否必须 | 默认值 |
|----------------|--------|---------------------|--------|
| host | string | 是 | - |
| port | int | 是 | - |
| key | string | 是 | - |
| data_type | string | 是 | - |
| user | string | 否 | - |
| auth | string | 否 | - |
| db_num | int | 否 | 0 |
| mode | string | 否 | single |
| nodes | list | 当 mode=cluster 时为:是 | - |
| format | string | 否 | json |
| expire | long | 否 | -1 |
| common-options | | 否 | - |
| name | type | required | default value |
|--------------------|---------|-----------------------|---------------|
| host | string | yes | - |
| port | int | yes | - |
| key | string | yes | - |
| data_type | string | yes | - |
| batch_size | int | no | 10 |
| user | string | no | - |
| auth | string | no | - |
| db_num | int | no | 0 |
| mode | string | no | single |
| nodes | list | yes when mode=cluster | - |
| format | string | no | json |
| expire | long | no | -1 |
| support_custom_key | boolean | no | false |
| value_field | string | no | - |
| hash_key_field | string | no | - |
| hash_value_field | string | no | - |
| common-options | | no | - |

### host [string]

Expand All @@ -48,13 +53,17 @@ Redis 端口
| 200 | 获取成功 | true |
| 500 | 内部错误 | false |

如果将字段名称指定为 `code` 并将 data_type 设置为 `key`，将有两个数据写入 Redis：
1. `200 -> {code: 200, message: true, data: 获取成功}`
2. `500 -> {code: 500, message: false, data: 内部错误}`
可以使用`{`和`}`符号自定义Redis键名，`{}`中的字段名会被解析替换为上游数据中的某个字段值，例如：将字段名称指定为 `{code}` 并将 data_type 设置为 `key`，将有两个数据写入 Redis：
1. `200 -> {code: 200, data: 获取成功, success: true}`
2. `500 -> {code: 500, data: 内部错误, success: false}`

如果将字段名称指定为 `value` 并将 data_type 设置为 `key`，则由于上游数据的字段中没有 `value` 字段，将只有一个数据写入 Redis：
Redis键名可以由固定部分和变化部分组成，通过Redis分组符号:连接，例如：将字段名称指定为 `code:{code}` 并将 data_type 设置为 `key`，将有两个数据写入 Redis：
1. `code:200 -> {code: 200, data: 获取成功, success: true}`
2. `code:500 -> {code: 500, data: 内部错误, success: false}`

1. `value -> {code: 500, message: false, data: 内部错误}`
如果将Redis键名指定为 `value` 并将 data_type 设置为 `key`，则只有一个数据写入 Redis：

1. `value -> {code: 500, data: 内部错误, success: false}`

请参见 data_type 部分以了解具体的写入规则。

Expand Down Expand Up @@ -128,6 +137,59 @@ Redis 节点信息，在集群模式下使用，必须按如下格式：

设置 Redis 的过期时间，单位为秒。默认值为 -1，表示键不会自动过期。

### support_custom_key [boolean]

设置为true，表示启用自定义Key。

上游数据如下：

| code | data | success |
|------|------|---------|
| 200 | 获取成功 | true |
| 500 | 内部错误 | false |

可以使用`{`和`}`符号自定义Redis键名，`{}`中的字段名会被解析替换为上游数据中的某个字段值，例如：将字段名称指定为 `{code}` 并将 data_type 设置为 `key`，将有两个数据写入 Redis：
1. `200 -> {code: 200, data: 获取成功, success: true}`
2. `500 -> {code: 500, data: 内部错误, success: false}`

Redis键名可以由固定部分和变化部分组成，通过Redis分组符号:连接，例如：将字段名称指定为 `code:{code}` 并将 data_type 设置为 `key`，将有两个数据写入 Redis：
1. `code:200 -> {code: 200, data: 获取成功, success: true}`
2. `code:500 -> {code: 500, data: 内部错误, success: false}`

### value_field [string]

要写入Redis的值的字段， `data_type` 支持 `key` `list` `set` `zset`.

当你指定Redis键名字段`key`指定为 `value`，值字段`value_field`指定为`data`，并将`data_type`指定为`key`时,

上游数据如下：

| code | data | success |
|------|------|---------|
| 200 | 获取成功 | true |

如下的数据会被写入Redis:
1. `value -> 获取成功`

### hash_key_field [string]

要写入Redis的hash键字段, `data_type` 支持 `hash`

### hash_value_field [string]

要写入Redis的hash值字段, `data_type` 支持 `hash`

当你指定Redis键名字段`key`指定为 `value`，hash键字段`hash_key_field`指定为`data`，hash值字段`hash_value_field`指定为`success`，并将`data_type`指定为`hash`时,

上游数据如下：

| code | data | success |
|------|------|---------|
| 200 | 获取成功 | true |

如下的数据会被写入Redis:
1. `value -> 获取成功 | true`

### common options

Sink 插件通用参数，请参考 [Sink Common Options](../sink-common-options.md) 获取详情
Expand All @@ -145,6 +207,43 @@ Redis {
}
```

自定义Key示例：

```hocon
Redis {
host = localhost
port = 6379
key = "name:{name}"
support_custom_key = true
data_type = key
}
```

自定义Value示例：

```hocon
Redis {
host = localhost
port = 6379
key = person
value_field = "name"
data_type = key
}
```

自定义HashKey和HashValue示例：

```hocon
Redis {
host = localhost
port = 6379
key = person
hash_key_field = "name"
hash_value_field = "age"
data_type = hash
}
```

## 更新日志

### 2.2.0-beta 2022-09-26
Expand Down
26 changes: 26 additions & 0 deletions ...dis/src/main/java/org/apache/seatunnel/connectors/seatunnel/redis/config/RedisConfig.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,32 @@ public enum HashKeyParseMode {
"batch_size is used to control the size of a batch of data during read and write operations"
+ ",default 10");

public static final Option<Boolean> SUPPORT_CUSTOM_KEY =
Options.key("support_custom_key")
.booleanType()
.defaultValue(false)
.withDescription(
"if true, the key can be customized by the field value in the upstream data.");

public static final Option<String> VALUE_FIELD =
Options.key("value_field")
.stringType()
.noDefaultValue()
.withDescription(
"The field of value you want to write to redis, support string list set zset");

public static final Option<String> HASH_KEY_FIELD =
Options.key("hash_key_field")
.stringType()
.noDefaultValue()
.withDescription("The field of hash key you want to write to redis");

public static final Option<String> HASH_VALUE_FIELD =
Options.key("hash_value_field")
.stringType()
.noDefaultValue()
.withDescription("The field of hash value you want to write to redis");

public enum Format {
JSON,
// TEXT will be supported later
Expand Down
Loading

0 comments on commit ef2c3c7

Please sign in to comment.