From 80cc4c52b310df7ed1037935b89c40a8e546a07b Mon Sep 17 00:00:00 2001 From: zhilinli Date: Thu, 10 Aug 2023 00:10:20 +0800 Subject: [PATCH 1/7] [Feature][doc][Connector-V2][Fake] FakeSource doc connector --- docs/en/connector-v2/source/FakeSource.md | 372 ++++++---------------- 1 file changed, 103 insertions(+), 269 deletions(-) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index d393d8f5704..35d6ae798cc 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -2,10 +2,14 @@ > FakeSource connector +## Support Connector Version + +- All versions + ## Description -The FakeSource is a virtual data source, which randomly generates the number of rows according to the data structure of the user-defined schema, -just for some test cases such as type conversion or connector new feature testing +The FakeSource is a virtual data source, which randomly generates the number of rows according to the data structure of +the user-defined schema, just for some test cases such as type conversion or connector new feature testing ## Key features @@ -16,53 +20,53 @@ just for some test cases such as type conversion or connector new feature testin - [ ] [parallelism](../../concept/connector-v2-features.md) - [ ] [support user-defined split](../../concept/connector-v2-features.md) -## Options - -| name | type | required | default value | -|---------------------|----------|----------|-------------------------| -| schema | config | yes | - | -| rows | config | no | - | -| row.num | int | no | 5 | -| split.num | int | no | 1 | -| split.read-interval | long | no | 1 | -| map.size | int | no | 5 | -| array.size | int | no | 5 | -| bytes.length | int | no | 5 | -| string.length | int | no | 5 | -| string.fake.mode | string | no | range | -| tinyint.fake.mode | string | no | range | -| tinyint.min | tinyint | no | 0 | -| tinyint.max | tinyint | no | 127 | -| tinyint.template | list | no | - | -| smallint.fake.mode | string | no | range | -| smallint.min | smallint | no | 0 | -| smallint.max | smallint | no | 32767 | -| smallint.template | list | no | - | -| int.fake.template | string | no | range | -| int.min | int | no | 0 | -| int.max | int | no | 0x7fffffff | -| int.template | list | no | - | -| bigint.fake.mode | string | no | range | -| bigint.min | bigint | no | 0 | -| bigint.max | bigint | no | 0x7fffffffffffffff | -| bigint.template | list | no | - | -| float.fake.mode | string | no | range | -| float.min | float | no | 0 | -| float.max | float | no | 0x1.fffffeP+127 | -| float.template | list | no | - | -| double.fake.mode | string | no | range | -| double.min | double | no | 0 | -| double.max | double | no | 0x1.fffffffffffffP+1023 | -| double.template | list | no | - | -| common-options | | no | - | - -### schema [config] - -#### fields [Config] - -The schema of fake data that you want to generate - -#### Examples +## Source Options + +| Name | Type | Required | Default | Description | +|---------------------|----------|----------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| schema | config | yes | - | Define Schema information | +| fields | config | yes | - | Define Field information | +| rows | config | no | - | The row list of fake data output per degree of parallelism | +| row.num | int | no | 5 | The total number of data generated per degree of parallelism | +| split.num | int | no | 1 | the number of splits generated by the enumerator for each degree of parallelism | +| split.read-interval | long | no | 1 | The interval(mills) between two split reads in a reader | +| map.size | int | no | 5 | The size of `map` type that connector generated | +| array.size | int | no | 5 | The size of `array` type that connector generated | +| bytes.length | int | no | 5 | The length of `bytes` type that connector generated | +| string.length | int | no | 5 | The length of `string` type that connector generated | +| string.fake.mode | string | no | range | The fake mode of generating string data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `string.template` option | +| string.template | list | no | - | The template list of string type that connector generated, if user configured it, connector will randomly select an item from the template list | +| tinyint.fake.mode | string | no | range | The fake mode of generating tinyint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `tinyint.template` option | +| tinyint.min | tinyint | no | 0 | The min value of tinyint data that connector generated | +| tinyint.max | tinyint | no | 127 | The max value of tinyint data that connector generated | +| tinyint.template | list | no | - | The template list of tinyint type that connector generated, if user configured it, connector will randomly select an item from the template list | +| smallint.fake.mode | string | no | range | The fake mode of generating smallint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `smallint.template` option | +| smallint.min | smallint | no | 0 | The min value of smallint data that connector generated | +| smallint.max | smallint | no | 32767 | The max value of smallint data that connector generated | +| smallint.template | list | no | - | The template list of smallint type that connector generated, if user configured it, connector will randomly select an item from the template list | +| int.fake.template | string | no | range | The fake mode of generating int data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `int.template` option | +| int.min | int | no | 0 | The min value of int data that connector generated | +| int.max | int | no | 0x7fffffff | The max value of int data that connector generated | +| int.template | list | no | - | The template list of int type that connector generated, if user configured it, connector will randomly select an item from the template list | +| bigint.fake.mode | string | no | range | The fake mode of generating bigint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `bigint.template` option | +| bigint.min | bigint | no | 0 | The min value of bigint data that connector generated | +| bigint.max | bigint | no | 0x7fffffffffffffff | The max value of bigint data that connector generated | +| bigint.template | list | no | - | The template list of bigint type that connector generated, if user configured it, connector will randomly select an item from the template list | +| float.fake.mode | string | no | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `float.template` option | +| float.min | float | no | 0 | The min value of float data that connector generated | +| float.max | float | no | 0x1.fffffeP+127 | The max value of float data that connector generated | +| float.template | list | no | - | The template list of float type that connector generated, if user configured it, connector will randomly select an item from the template list | +| double.fake.mode | string | no | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `double.template` option | +| double.min | double | no | 0 | The min value of double data that connector generated | +| double.max | double | no | 0x1.fffffffffffffP+1023 | The max value of double data that connector generated | +| double.template | list | no | - | The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list | +| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | + +## Task Example + +### Simple: + +> This example Randomly generates data of a specified type ```hocon schema = { @@ -103,172 +107,58 @@ schema = { } ``` -### rows - -The row list of fake data output per degree of parallelism +### Customize the data content Simple: -example +> This is a self-defining data source information, defining whether each piece of data is an add or delete modification operation, and defining what each field stores ```hocon -rows = [ - { - kind = INSERT - fields = [1, "A", 100] - }, - { - kind = UPDATE_BEFORE - fields = [1, "A", 100] - }, - { - kind = UPDATE_AFTER - fields = [1, "A_1", 100] - }, - { - kind = DELETE - fields = [1, "A_1", 100] +source { + FakeSource { + schema = { + fields { + c_map = "map" + c_array = "array" + c_string = string + c_boolean = boolean + c_tinyint = tinyint + c_smallint = smallint + c_int = int + c_bigint = bigint + c_float = float + c_double = double + c_decimal = "decimal(30, 8)" + c_null = "null" + c_bytes = bytes + c_date = date + c_timestamp = timestamp + } + } + rows = [ + { + kind = INSERT + fields = [{"a": "b"}, [101], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"] + } + { + kind = UPDATE_BEFORE + fields = [{"a": "c"}, [102], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"] + } + { + kind = UPDATE_AFTER + fields = [{"a": "e"}, [103], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"] + } + { + kind = DELETE + fields = [{"a": "f"}, [104], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"] + } + ] } -] -``` - -### row.num - -The total number of data generated per degree of parallelism - -### split.num - -the number of splits generated by the enumerator for each degree of parallelism - -### split.read-interval - -The interval(mills) between two split reads in a reader - -### map.size - -The size of `map` type that connector generated - -### array.size - -The size of `array` type that connector generated - -### bytes.length - -The length of `bytes` type that connector generated - -### string.length - -The length of `string` type that connector generated - -### string.fake.mode - -The fake mode of generating string data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `string.template` option - -### string.template - -The template list of string type that connector generated, if user configured it, connector will randomly select an item from the template list - -### tinyint.fake.mode - -The fake mode of generating tinyint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `tinyint.template` option - -### tinyint.min - -The min value of tinyint data that connector generated - -### tinyint.max - -The max value of tinyint data that connector generated - -### tinyint.template - -The template list of tinyint type that connector generated, if user configured it, connector will randomly select an item from the template list - -### smallint.fake.mode - -The fake mode of generating smallint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `smallint.template` option - -### smallint.min - -The min value of smallint data that connector generated - -### smallint.max - -The max value of smallint data that connector generated - -### smallint.template - -The template list of smallint type that connector generated, if user configured it, connector will randomly select an item from the template list - -### int.fake.mode - -The fake mode of generating int data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `int.template` option - -### int.min - -The min value of int data that connector generated - -### int.max - -The max value of int data that connector generated - -### int.template - -The template list of int type that connector generated, if user configured it, connector will randomly select an item from the template list - -### bigint.fake.mode - -The fake mode of generating bigint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `bigint.template` option - -### bigint.min - -The min value of bigint data that connector generated - -### bigint.max - -The max value of bigint data that connector generated - -### bigint.template - -The template list of bigint type that connector generated, if user configured it, connector will randomly select an item from the template list - -### float.fake.mode - -The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `float.template` option - -### float.min - -The min value of float data that connector generated - -### float.max - -The max value of float data that connector generated - -### float.template - -The template list of float type that connector generated, if user configured it, connector will randomly select an item from the template list - -### double.fake.mode - -The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `double.template` option - -### double.min - -The min value of double data that connector generated - -### double.max - -The max value of double data that connector generated - -### double.template - -The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list - -### common options +} -Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details +``` -## Example +### Specified Data number Simple: -Auto generate data rows +> This case specifies the number of data generated and the length of the generated value ```hocon FakeSource { @@ -316,45 +206,9 @@ FakeSource { } ``` -Using fake data rows +### Template data Simple: -```hocon -FakeSource { - schema = { - fields { - pk_id = bigint - name = string - score = int - } - } - rows = [ - { - kind = INSERT - fields = [1, "A", 100] - }, - { - kind = INSERT - fields = [2, "B", 100] - }, - { - kind = INSERT - fields = [3, "C", 100] - }, - { - kind = UPDATE_BEFORE - fields = [1, "A", 100] - }, - { - kind = UPDATE_AFTER - fields = [1, "A_1", 100] - }, - { - kind = DELETE - fields = [2, "B", 100] - } - ] -} -``` +> Randomly generated according to the specified template Using template @@ -389,7 +243,9 @@ FakeSource { } ``` -Use range +### Range data Simple: + +> The specified data generation range is randomly generated ```hocon FakeSource { @@ -421,25 +277,3 @@ FakeSource { } ``` -## Changelog - -### 2.2.0-beta 2022-09-26 - -- Add FakeSource Source Connector - -### 2.3.0-beta 2022-10-20 - -- [Improve] Supports direct definition of data values(row) ([2839](https://github.com/apache/seatunnel/pull/2839)) -- [Improve] Improve fake source connector: ([2944](https://github.com/apache/seatunnel/pull/2944)) - - Support user-defined map size - - Support user-defined array size - - Support user-defined string length - - Support user-defined bytes length -- [Improve] Support multiple splits for fake source connector ([2974](https://github.com/apache/seatunnel/pull/2974)) -- [Improve] Supports setting the number of splits per parallelism and the reading interval between two splits ([3098](https://github.com/apache/seatunnel/pull/3098)) - -### next version - -- [Feature] Support config fake data rows [3865](https://github.com/apache/seatunnel/pull/3865) -- [Feature] Support config template or range for fake data [3932](https://github.com/apache/seatunnel/pull/3932) - From e883808b5c5971876d6456bb799396e46d61a3a0 Mon Sep 17 00:00:00 2001 From: zhilinli Date: Mon, 28 Aug 2023 10:22:57 +0800 Subject: [PATCH 2/7] add FakeSource.md --- docs/en/connector-v2/source/FakeSource.md | 33 +++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index 35d6ae798cc..1b4972702eb 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -107,6 +107,39 @@ schema = { } ``` +### Random Generation + +> 16 data matching the type are randomly generated + +```hocon +source { + # This is a example input plugin **only for test and demonstrate the feature input plugin** + FakeSource { + row.num = 16 + schema = { + fields { + c_map = "map" + c_array = "array" + c_string = string + c_boolean = boolean + c_tinyint = tinyint + c_smallint = smallint + c_int = int + c_bigint = bigint + c_float = float + c_double = double + c_decimal = "decimal(30, 8)" + c_null = "null" + c_bytes = bytes + c_date = date + c_timestamp = timestamp + } + } + result_table_name = "fake" + } +} +``` + ### Customize the data content Simple: > This is a self-defining data source information, defining whether each piece of data is an add or delete modification operation, and defining what each field stores From a313852cf973742cf3c60956bff358825dfc72ea Mon Sep 17 00:00:00 2001 From: zhilinli Date: Wed, 18 Oct 2023 11:11:35 +0800 Subject: [PATCH 3/7] update comment --- docs/en/connector-v2/source/FakeSource.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index c31a17e97cb..0dcf43ea214 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -56,6 +56,7 @@ just for some test cases such as type conversion or connector new feature testin | double.min | double | no | 0 | The min value of double data that connector generated | | double.max | double | no | 0x1.fffffffffffffP+1023 | The max value of double data that connector generated | | double.template | list | no | - | The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list | +| table-names | list | no | - | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example From 633fd5f519a85b832775d77cd498559efdb344a9 Mon Sep 17 00:00:00 2001 From: zhilinli Date: Wed, 18 Oct 2023 11:19:54 +0800 Subject: [PATCH 4/7] update --- docs/en/connector-v2/source/FakeSource.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index 0dcf43ea214..e3ac375020d 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -21,7 +21,6 @@ just for some test cases such as type conversion or connector new feature testin | Name | Type | Required | Default | Description | |---------------------|----------|----------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | schema | config | yes | - | Define Schema information | -| fields | config | yes | - | Define Field information | | rows | config | no | - | The row list of fake data output per degree of parallelism | | row.num | int | no | 5 | The total number of data generated per degree of parallelism | | split.num | int | no | 1 | the number of splits generated by the enumerator for each degree of parallelism | From 5b5aed4c7ed0b720c9e565a1f2d1294362ea8f36 Mon Sep 17 00:00:00 2001 From: zhilinli Date: Wed, 18 Oct 2023 13:44:32 +0800 Subject: [PATCH 5/7] update --- docs/en/connector-v2/source/FakeSource.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index e3ac375020d..be05e1b1316 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -55,7 +55,7 @@ just for some test cases such as type conversion or connector new feature testin | double.min | double | no | 0 | The min value of double data that connector generated | | double.max | double | no | 0x1.fffffffffffffP+1023 | The max value of double data that connector generated | | double.template | list | no | - | The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list | -| table-names | list | no | - | +| table-names | list | no | - | The Supports configuration of multiple data sources for example: ["test.table1", "test.table2"] | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example @@ -306,7 +306,9 @@ FakeSource { } ``` -### Use table-names +### Configure Multiple FakeSource + +> This is a case of generating a multi-data source test.table1 and test.table2 ```hocon FakeSource { From 0e4e841e107b721a73409f10f32ab8b7b7301f0c Mon Sep 17 00:00:00 2001 From: zhilinli Date: Wed, 18 Oct 2023 13:51:44 +0800 Subject: [PATCH 6/7] update --- docs/en/connector-v2/source/FakeSource.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index be05e1b1316..195ba5f6fe7 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -306,7 +306,7 @@ FakeSource { } ``` -### Configure Multiple FakeSource +### Generate Multiple tables > This is a case of generating a multi-data source test.table1 and test.table2 From 8b55709d7784f08a6a09a43565e601c27a33732d Mon Sep 17 00:00:00 2001 From: zhilinli Date: Thu, 19 Oct 2023 11:26:44 +0800 Subject: [PATCH 7/7] update --- docs/en/connector-v2/source/FakeSource.md | 115 ++++++++++++++-------- 1 file changed, 75 insertions(+), 40 deletions(-) diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index 195ba5f6fe7..af89dceadaf 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -18,45 +18,45 @@ just for some test cases such as type conversion or connector new feature testin ## Source Options -| Name | Type | Required | Default | Description | -|---------------------|----------|----------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| schema | config | yes | - | Define Schema information | -| rows | config | no | - | The row list of fake data output per degree of parallelism | -| row.num | int | no | 5 | The total number of data generated per degree of parallelism | -| split.num | int | no | 1 | the number of splits generated by the enumerator for each degree of parallelism | -| split.read-interval | long | no | 1 | The interval(mills) between two split reads in a reader | -| map.size | int | no | 5 | The size of `map` type that connector generated | -| array.size | int | no | 5 | The size of `array` type that connector generated | -| bytes.length | int | no | 5 | The length of `bytes` type that connector generated | -| string.length | int | no | 5 | The length of `string` type that connector generated | -| string.fake.mode | string | no | range | The fake mode of generating string data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `string.template` option | -| string.template | list | no | - | The template list of string type that connector generated, if user configured it, connector will randomly select an item from the template list | -| tinyint.fake.mode | string | no | range | The fake mode of generating tinyint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `tinyint.template` option | -| tinyint.min | tinyint | no | 0 | The min value of tinyint data that connector generated | -| tinyint.max | tinyint | no | 127 | The max value of tinyint data that connector generated | -| tinyint.template | list | no | - | The template list of tinyint type that connector generated, if user configured it, connector will randomly select an item from the template list | -| smallint.fake.mode | string | no | range | The fake mode of generating smallint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `smallint.template` option | -| smallint.min | smallint | no | 0 | The min value of smallint data that connector generated | -| smallint.max | smallint | no | 32767 | The max value of smallint data that connector generated | -| smallint.template | list | no | - | The template list of smallint type that connector generated, if user configured it, connector will randomly select an item from the template list | -| int.fake.template | string | no | range | The fake mode of generating int data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `int.template` option | -| int.min | int | no | 0 | The min value of int data that connector generated | -| int.max | int | no | 0x7fffffff | The max value of int data that connector generated | -| int.template | list | no | - | The template list of int type that connector generated, if user configured it, connector will randomly select an item from the template list | -| bigint.fake.mode | string | no | range | The fake mode of generating bigint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `bigint.template` option | -| bigint.min | bigint | no | 0 | The min value of bigint data that connector generated | -| bigint.max | bigint | no | 0x7fffffffffffffff | The max value of bigint data that connector generated | -| bigint.template | list | no | - | The template list of bigint type that connector generated, if user configured it, connector will randomly select an item from the template list | -| float.fake.mode | string | no | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `float.template` option | -| float.min | float | no | 0 | The min value of float data that connector generated | -| float.max | float | no | 0x1.fffffeP+127 | The max value of float data that connector generated | -| float.template | list | no | - | The template list of float type that connector generated, if user configured it, connector will randomly select an item from the template list | -| double.fake.mode | string | no | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `double.template` option | -| double.min | double | no | 0 | The min value of double data that connector generated | -| double.max | double | no | 0x1.fffffffffffffP+1023 | The max value of double data that connector generated | -| double.template | list | no | - | The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list | -| table-names | list | no | - | The Supports configuration of multiple data sources for example: ["test.table1", "test.table2"] | -| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | +| Name | Type | Required | Default | Description | +|---------------------|----------|----------|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| schema | config | yes | - | Define Schema information | +| rows | config | no | - | The row list of fake data output per degree of parallelism see title `Options rows Case`. | +| row.num | int | no | 5 | The total number of data generated per degree of parallelism | +| split.num | int | no | 1 | the number of splits generated by the enumerator for each degree of parallelism | +| split.read-interval | long | no | 1 | The interval(mills) between two split reads in a reader | +| map.size | int | no | 5 | The size of `map` type that connector generated | +| array.size | int | no | 5 | The size of `array` type that connector generated | +| bytes.length | int | no | 5 | The length of `bytes` type that connector generated | +| string.length | int | no | 5 | The length of `string` type that connector generated | +| string.fake.mode | string | no | range | The fake mode of generating string data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `string.template` option | +| string.template | list | no | - | The template list of string type that connector generated, if user configured it, connector will randomly select an item from the template list | +| tinyint.fake.mode | string | no | range | The fake mode of generating tinyint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `tinyint.template` option | +| tinyint.min | tinyint | no | 0 | The min value of tinyint data that connector generated | +| tinyint.max | tinyint | no | 127 | The max value of tinyint data that connector generated | +| tinyint.template | list | no | - | The template list of tinyint type that connector generated, if user configured it, connector will randomly select an item from the template list | +| smallint.fake.mode | string | no | range | The fake mode of generating smallint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `smallint.template` option | +| smallint.min | smallint | no | 0 | The min value of smallint data that connector generated | +| smallint.max | smallint | no | 32767 | The max value of smallint data that connector generated | +| smallint.template | list | no | - | The template list of smallint type that connector generated, if user configured it, connector will randomly select an item from the template list | +| int.fake.template | string | no | range | The fake mode of generating int data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `int.template` option | +| int.min | int | no | 0 | The min value of int data that connector generated | +| int.max | int | no | 0x7fffffff | The max value of int data that connector generated | +| int.template | list | no | - | The template list of int type that connector generated, if user configured it, connector will randomly select an item from the template list | +| bigint.fake.mode | string | no | range | The fake mode of generating bigint data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `bigint.template` option | +| bigint.min | bigint | no | 0 | The min value of bigint data that connector generated | +| bigint.max | bigint | no | 0x7fffffffffffffff | The max value of bigint data that connector generated | +| bigint.template | list | no | - | The template list of bigint type that connector generated, if user configured it, connector will randomly select an item from the template list | +| float.fake.mode | string | no | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `float.template` option | +| float.min | float | no | 0 | The min value of float data that connector generated | +| float.max | float | no | 0x1.fffffeP+127 | The max value of float data that connector generated | +| float.template | list | no | - | The template list of float type that connector generated, if user configured it, connector will randomly select an item from the template list | +| double.fake.mode | string | no | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `double.template` option | +| double.min | double | no | 0 | The min value of double data that connector generated | +| double.max | double | no | 0x1.fffffffffffffP+1023 | The max value of double data that connector generated | +| double.template | list | no | - | The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list | +| table-names | list | no | - | The table list that connector generated, used to simulate multi-table scenarios.
This option will override the `table` option in the `schema` option. For example, if you configure the `table-names` option as follows, the connector will generate data for the `test.table1` and `test.table2` tables, the `database.schema.table` will be dropFor details, see title `Options table-names Case`. | +| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example @@ -182,7 +182,6 @@ source { ] } } - ``` ### Specified Data number Simple: @@ -327,6 +326,42 @@ FakeSource { } ``` +### Options `rows` Case + +```hocon +rows = [ + { + kind = INSERT + fields = [1, "A", 100] + }, + { + kind = UPDATE_BEFORE + fields = [1, "A", 100] + }, + { + kind = UPDATE_AFTER + fields = [1, "A_1", 100] + }, + { + kind = DELETE + fields = [1, "A_1", 100] + } +] +``` + +### Options `table-names` Case + +```agsl +FakeSource { + table-names = ["test.table1", "test.table2"] + schema = { + table = "database.schema.table" + ... + } + ... +} +``` + ## Changelog ### 2.2.0-beta 2022-09-26