[Improve][DOC] Perfect the connector v2 doc #2800

liugddx · 2022-09-20T02:36:35Z

Purpose of this pull request

Perfect the connector v2 doc

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

seatunnel-connectors-v2/README.zh.md

Hisoka-X · 2022-09-20T11:23:26Z

How about modify English doc too?

Co-authored-by: Hisoka <fanjiaeminem@qq.com>

…prove-doc-connectorv2

liugddx · 2022-09-20T12:15:49Z

PTAL @Hisoka-X

Hisoka-X

LGTM

TyrantLucifer · 2022-09-20T15:52:57Z

seatunnel-connectors-v2/README.zh.md

@@ -1,11 +1,11 @@
 ## 目的
-Because SeaTunnel design new API for connectors, 所以通过这篇文章来介绍新的接口以及新的代码结构，方便开发者快速的帮助新API和翻译层完善，以及开发出新的Connecotor.
+因为SeaTunnel 为connectors设计了新的API，所以通过这篇文章来介绍新的接口以及新的代码结构，方便开发者快速的帮助新API和翻译层完善，以及开发出新的Connector.


SeaTunnel为与计算引擎进行解耦，设计了新的连接器API，通过这篇文章来介绍新的接口以及新的代码结构，方便开发者快速上手使用新版API开发连接器并理解新版API运行原理

Who not change it?

TyrantLucifer · 2022-09-20T15:56:43Z

seatunnel-connectors-v2/README.zh.md

 为了和老的代码分开，方便现阶段的并行开发，以及降低merge的难度。我们为新的执行流程定义了新的模块
 ### Example
 我们已经在`seatunnel-examples`中准备好了新版本的可本地执行Example程序，直接调用`seatunnel-flink-connector-v2-example`或`seatunnel-spark-connector-v2-example`中的`SeaTunnelApiExample`即可。这也是本地开发Connector经常会用到的调试方式。
-对应的配置文件保存在同模块的`resources/examples`文件夹下，和以前一样。
+对应的配置文件保存在同模块的`resources/examples`文件夹下，和以前一样。目前测试需要将connector依赖进来，如果有部分第三方包需要自行依赖请使用类加载器自行加载，如果有些 Scope为 Provided的依赖，需要在Intellij IDEAZ中单击 "Run"->"Edit configurations...",在


目前在Example模块下测试需要把对应连接器的依赖添加到pom文件中，如果有部分三方依赖请自行添加至模块classpath，如果连接器的某些依赖的scope为provided，则还需要在Intellij IDEA中设置Include dependencies with Provided Scope

TyrantLucifer · 2022-09-20T15:58:06Z

seatunnel-connectors-v2/README.zh.md

现阶段所有的连接器实现及可参考的示例都在seatunnel-connectors-v2下，用户可自行查阅参考

TyrantLucifer · 2022-09-20T16:00:51Z

seatunnel-connectors-v2/README.md


 ### **Example**
 We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example`` in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector. The corresponding configuration files are saved in the same module ``resources/examples`` folder as before.
+For now, the test needs to rely on the Connector,if your test must use a third-party jar and the Scope is  Provided,you need to check "Include dependencies with 'Provided Scope" and choose current Project in "Run"->"Edit configurations..."->"Use classpath of module" . 


Currently, you need to add the dependencies of the corresponding connectors to the pom file for testing under the seatunnel-examples module, if there are some three-party dependencies, please add them to the module classpath by yourself, if the scope of some dependencies of the connectors is provided, you also need to set Include dependencies with Provided Scope in Intellij IDEA.

TyrantLucifer · 2022-09-20T16:03:47Z

seatunnel-connectors-v2/README.md

@@ -2,10 +2,11 @@
 This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.


SeaTunnel has designed a new connector API to decouple from the compute engine. This article introduces the new interface and the new code structure to help developers quickly get started with the new API and understand the principles of the new API.

EricJoy2048 · 2022-09-21T04:03:29Z

That's great.

Thank you for your contribution.

liugddx · 2022-09-21T10:53:44Z

@TyrantLucifer @EricJoy2048

liugddx · 2022-09-21T10:53:56Z

@TyrantLucifer @EricJoy2048

PTAL

liugddx · 2022-09-24T05:30:18Z

#2873

TyrantLucifer · 2022-09-24T08:46:27Z

@TyrantLucifer @EricJoy2048

PTAL

Now the document still has some incoherent and unreasonable places, please modify according to the comments.

EricJoy2048 · 2022-09-24T08:59:25Z

seatunnel-connectors-v2/README.md

@@ -1,48 +1,101 @@
 # Purpose
-This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.
+
+This article introduces the new interface and the new code structure on account of the newly designed API for Connectors


Can you link the proposal #1608

Please link proposal here like the Chinese document.

EricJoy2048 · 2022-09-24T09:03:40Z

seatunnel-connectors-v2/README.md

-This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.
+
+This article introduces the new interface and the new code structure on account of the newly designed API for Connectors
+in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and


This helps developers quickly understand API and transformation layer improvements. On the other hand, it can guide contributors how to use the new API to develop new connectors. is better.

EricJoy2048 · 2022-09-24T09:20:35Z

seatunnel-connectors-v2/README.md


+We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
+directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``


which can be
directly called using seatunnel-flink-connector-v2-example or seatunnel-spark-connector-v2-example
in SeaTunnelApiExample

One is seatunnel-examples/seatunnel-flink-connector-v2-example/src/main/java/org/apache/seatunnel/example/flink/v2/SeaTunnelApiExample.java, it runs in the Flink engine. Another one is seatunnel-examples/seatunnel-spark-connector-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java, it runs in the Spark engine.

EricJoy2048 · 2022-09-24T09:22:48Z

seatunnel-connectors-v2/README.md


+We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be


We have prepared a new version of the locally executable example program in seatunnel-examples

We have prepared two new version of the locally executable example program in seatunnel-examples.

EricJoy2048 · 2022-09-24T09:32:35Z

seatunnel-connectors-v2/README.md

+We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
+directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
+in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector.
+The corresponding configuration files are saved in the same module ``resources/examples`` folder as before. Currently,


The corresponding configuration files are saved in the same module resources/examples folder as before

The configuration files used in example are saved in the "resources/examples" folder..

I think it is enough to clarify the location of the example configuration file. It is unnecessary to guide the user to understand the previous configuration.

EricJoy2048 · 2022-09-24T09:36:46Z

seatunnel-connectors-v2/README.md


+We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
+directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
+in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector.


This is also the debugging method that is often used in the local development of Connector.

This is also the debugging method that is often used in the local development of Connector. You can debug these examples, which will help you better understand the running logic of the program.

EricJoy2048 · 2022-09-24T09:56:36Z

seatunnel-connectors-v2/README.md

+directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
+in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector.
+The corresponding configuration files are saved in the same module ``resources/examples`` folder as before. Currently,
+you need to add the dependencies of the corresponding connectors to the pom file for testing under the


you need to add the dependencies of the corresponding connectors to the pom file for testing under the
seatunnel-examples module, if there are some three-party dependencies, please add them to the module classpath by
yourself, if the scope of some dependencies of the connectors is provided, you also need to set Include dependencies
with Provided Scope in Intellij IDEA.

I think these are guidelines for contributors on how to add examples to their connectors. This requires clearer steps.

If you want to add examples for your own connectors, you need to follow the steps below.

Add the groupId, artifactId and version of the connector to be tested to seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(or add it to seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml when you want to runs it in Spark engine) as a dependency.

Find the dependency in your connector pom file which scope is test or provided and then add them to seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(or add it to seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml file and modify the scope to compile.

Refer to the SeaTunnelApiExample class to develop your sample code.

EricJoy2048 · 2022-09-24T10:24:23Z

seatunnel-connectors-v2/README.md


 ### **Startup Class**
-Aside from the old startup class, we have created two new startup class projects, namely ``seatunnel-core/seatunnel-flink-starter`` and ``seatunnel-core/seatunnel-spark-starter``. You can find out how to parse the configuration file into an executable Flink/Spark process here.


Aside from the old startup class, we have created two new startup class projects

Aside from the old startup module, we have created two new startup modules.

close apache#2794

…prove-doc-connectorv2

liugddx · 2022-09-24T12:53:20Z

done.

TyrantLucifer · 2022-09-24T13:01:05Z

seatunnel-connectors-v2/README.zh.md

 ```java
-if (Boundedness.BOUNDED.equals(context.getBoundedness())) {
+if(Boundedness.BOUNDED.equals(context.getBoundedness())){


if (Boundedness.BOUNDED.equals(context.getBoundedness())) {

TyrantLucifer · 2022-09-24T13:02:58Z

seatunnel-connectors-v2/README.md

@@ -1,48 +1,101 @@
 # Purpose
-This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.
+
+This article introduces the new interface and the new code structure on account of the newly designed API for Connectors


Please link proposal here like the Chinese document.

liugddx · 2022-09-24T13:13:35Z

done

TyrantLucifer · 2022-09-24T15:17:51Z

done

Please change the content of the above screenshot, thanks.

liugddx · 2022-09-25T01:40:10Z

done

EricJoy2048 · 2022-09-26T03:46:26Z

seatunnel-connectors-v2/README.zh.md

+
+1. 在`seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml`添加connector依赖的groupId, artifactId 和
+   version.（或者当你想在spark引擎运行时在`seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml`添加依赖）
+2. 如果你的connector存在scope为test或者provided的依赖则在`seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml`(


如果你的connector存在scope为test或者provided的依赖则在seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(
或者在seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml)修改为compile。

如果你的connector中存在scope为test或provided的依赖，将这些依赖添加到seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(
或者在seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml)中，并且修改scope为compile.

liugddx · 2022-09-26T03:55:36Z

Done. @EricJoy2048

EricJoy2048

LGTM @TyrantLucifer PTAL

* seatunnel-metrics init commit * seatunnel-metrics config add * codeStyle update * codeStyle update again * codeStyle seatunnel-spark update * [Imporve][Connector-V2]Parameter verification for connector V2 kafka sink (#2866) * parameter verification * update * update * [Improve][DOC] Perfect the connector v2 doc (#2800) * [Improve][DOC] Perfect the connector v2 doc * Update seatunnel-connectors-v2/README.zh.md Co-authored-by: Hisoka <fanjiaeminem@qq.com> * [Improve][DOC] A little tinkering * [Improve][DOC] A little tinkering * [Doc][connector] add Console sink doc close #2794 * [Doc][connector] add Console sink doc close #2794 * fix some problem * fix some problem * fine tuning Co-authored-by: Hisoka <fanjiaeminem@qq.com> * add seatunnel-examples from gitignore (#2892) * [Improve][connector-jdbc] Calculate splits only once in JdbcSourceSplitEnumerator (#2900) * [Bug][Connector-V2] Fix wechat sink data serialization (#2856) * [Improve][Connector-V2] Improve orc write strategy to support all data types (#2860) * [Improve][Connector-V2] Improve orc write strategy to support all data types Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com> * [Bug][seatunnel-translation-base] Fix Source restore state NPE (#2878) * [Improve][Connector-v2-Fake]Supports direct definition of data values(row) (#2839) * [Improve][Connector-v2]Supports direct definition of data values(row) * seatunnel-prometheus update * seatunnel-prometheus update * seatunnel-prometheus update * 1. Seatunnel unified configuration naming 2. Use reflection to automate assembly 3. Modify the flink/spark startup function 4. Try packaging configuration (todo) Co-authored-by: TaoZex <45089228+TaoZex@users.noreply.github.com> Co-authored-by: liugddx <804167098@qq.com> Co-authored-by: Hisoka <fanjiaeminem@qq.com> Co-authored-by: Eric <gaojun2048@gmail.com> Co-authored-by: Xiao Zhao <zhaomin1423@163.com> Co-authored-by: hailin0 <wanghailin@apache.org> Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com> Co-authored-by: Laglangyue <35491928+laglangyue@users.noreply.github.com>

[Improve][DOC] Perfect the connector v2 doc

95bc9b0

liugddx force-pushed the improve-doc-connectorv2 branch from 5f31b73 to 95bc9b0 Compare September 20, 2022 03:02

Hisoka-X reviewed Sep 20, 2022

View reviewed changes

seatunnel-connectors-v2/README.zh.md Outdated Show resolved Hide resolved

liugddx and others added 3 commits September 20, 2022 19:45

Update seatunnel-connectors-v2/README.zh.md

e15c75f

Co-authored-by: Hisoka <fanjiaeminem@qq.com>

[Improve][DOC] A little tinkering

b8c52f1

Merge remote-tracking branch 'origin/improve-doc-connectorv2' into im…

b044748

…prove-doc-connectorv2

Hisoka-X previously approved these changes Sep 20, 2022

View reviewed changes

TyrantLucifer requested changes Sep 20, 2022

View reviewed changes

liugddx dismissed Hisoka-X’s stale review via 604aa59 September 21, 2022 02:54

[Improve][DOC] A little tinkering

604aa59

liugddx mentioned this pull request Sep 24, 2022

[Bug] [seatunnel-examples] run jdbc task will error in examples #2873

Closed

3 tasks

EricJoy2048 reviewed Sep 24, 2022

View reviewed changes

liugddx added 3 commits September 24, 2022 20:45

[Doc][connector] add Console sink doc

62dfa3a

close apache#2794

[Doc][connector] add Console sink doc

e925341

close apache#2794

Merge remote-tracking branch 'origin/improve-doc-connectorv2' into im…

ef9361b

…prove-doc-connectorv2

TyrantLucifer reviewed Sep 24, 2022

View reviewed changes

fix some problem

1884054

fix some problem

b8396bc

EricJoy2048 reviewed Sep 26, 2022

View reviewed changes

fine tuning

6054abb

Hisoka-X linked an issue Sep 26, 2022 that may be closed by this pull request

[Bug] [seatunnel-examples] run jdbc task will error in examples #2873

Closed

3 tasks

EricJoy2048 approved these changes Sep 26, 2022

View reviewed changes

TyrantLucifer approved these changes Sep 26, 2022

View reviewed changes

TyrantLucifer merged commit 2af310a into apache:dev Sep 26, 2022

liugddx deleted the improve-doc-connectorv2 branch September 26, 2022 11:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improve][DOC] Perfect the connector v2 doc #2800

[Improve][DOC] Perfect the connector v2 doc #2800

liugddx commented Sep 20, 2022 •

edited

Loading

Hisoka-X commented Sep 20, 2022 •

edited

Loading

liugddx commented Sep 20, 2022

Hisoka-X left a comment

TyrantLucifer Sep 20, 2022

TyrantLucifer Sep 24, 2022

TyrantLucifer Sep 20, 2022

TyrantLucifer Sep 20, 2022

TyrantLucifer Sep 20, 2022

TyrantLucifer Sep 20, 2022

EricJoy2048 commented Sep 21, 2022

liugddx commented Sep 21, 2022

liugddx commented Sep 21, 2022

liugddx commented Sep 24, 2022

TyrantLucifer commented Sep 24, 2022

EricJoy2048 Sep 24, 2022

TyrantLucifer Sep 24, 2022

EricJoy2048 Sep 24, 2022

EricJoy2048 Sep 24, 2022

EricJoy2048 Sep 24, 2022

EricJoy2048 Sep 24, 2022

EricJoy2048 Sep 24, 2022

EricJoy2048 Sep 24, 2022 •

edited

Loading

EricJoy2048 Sep 24, 2022

liugddx commented Sep 24, 2022

TyrantLucifer Sep 24, 2022

TyrantLucifer Sep 24, 2022

liugddx commented Sep 24, 2022

TyrantLucifer commented Sep 24, 2022

liugddx commented Sep 25, 2022

EricJoy2048 Sep 26, 2022

liugddx commented Sep 26, 2022

EricJoy2048 left a comment

		@@ -2,10 +2,11 @@
		This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.


		We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
		directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``


		### Startup Class
		Aside from the old startup class, we have created two new startup class projects, namely ``seatunnel-core/seatunnel-flink-starter`` and ``seatunnel-core/seatunnel-spark-starter``. You can find out how to parse the configuration file into an executable Flink/Spark process here.

[Improve][DOC] Perfect the connector v2 doc #2800

[Improve][DOC] Perfect the connector v2 doc #2800

Conversation

liugddx commented Sep 20, 2022 • edited Loading

Purpose of this pull request

Check list

Hisoka-X commented Sep 20, 2022 • edited Loading

liugddx commented Sep 20, 2022

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricJoy2048 commented Sep 21, 2022

liugddx commented Sep 21, 2022

liugddx commented Sep 21, 2022

liugddx commented Sep 24, 2022

TyrantLucifer commented Sep 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricJoy2048 Sep 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liugddx commented Sep 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liugddx commented Sep 24, 2022

TyrantLucifer commented Sep 24, 2022

liugddx commented Sep 25, 2022

Choose a reason for hiding this comment

liugddx commented Sep 26, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

liugddx commented Sep 20, 2022 •

edited

Loading

Hisoka-X commented Sep 20, 2022 •

edited

Loading

EricJoy2048 Sep 24, 2022 •

edited

Loading