Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][DOC] Perfect the connector v2 doc #2800

Merged
merged 11 commits into from
Sep 26, 2022

Conversation

liugddx
Copy link
Member

@liugddx liugddx commented Sep 20, 2022

Purpose of this pull request

Perfect the connector v2 doc

Check list

@Hisoka-X
Copy link
Member

Hisoka-X commented Sep 20, 2022

How about modify English doc too?

@liugddx
Copy link
Member Author

liugddx commented Sep 20, 2022

PTAL @Hisoka-X

Hisoka-X
Hisoka-X previously approved these changes Sep 20, 2022
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -1,11 +1,11 @@
## 目的
Because SeaTunnel design new API for connectors, 所以通过这篇文章来介绍新的接口以及新的代码结构,方便开发者快速的帮助新API和翻译层完善,以及开发出新的Connecotor.
因为SeaTunnel 为connectors设计了新的API,所以通过这篇文章来介绍新的接口以及新的代码结构,方便开发者快速的帮助新API和翻译层完善,以及开发出新的Connector.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SeaTunnel为与计算引擎进行解耦,设计了新的连接器API,通过这篇文章来介绍新的接口以及新的代码结构,方便开发者快速上手使用新版API开发连接器并理解新版API运行原理

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who not change it?

为了和老的代码分开,方便现阶段的并行开发,以及降低merge的难度。我们为新的执行流程定义了新的模块
### Example
我们已经在`seatunnel-examples`中准备好了新版本的可本地执行Example程序,直接调用`seatunnel-flink-connector-v2-example`或`seatunnel-spark-connector-v2-example`中的`SeaTunnelApiExample`即可。这也是本地开发Connector经常会用到的调试方式。
对应的配置文件保存在同模块的`resources/examples`文件夹下,和以前一样。
对应的配置文件保存在同模块的`resources/examples`文件夹下,和以前一样。目前测试需要将connector依赖进来,如果有部分第三方包需要自行依赖请使用类加载器自行加载,如果有些 Scope为 Provided的依赖,需要在Intellij IDEAZ中单击 "Run"->"Edit configurations...",在
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

目前在Example模块下测试需要把对应连接器的依赖添加到pom文件中,如果有部分三方依赖请自行添加至模块classpath,如果连接器的某些依赖的scopeprovided,则还需要在Intellij IDEA中设置Include dependencies with Provided Scope

所有的Connector实现都应该在`seatunnel-connectors-v2`模块下,现阶段可参考的示例均在此模块下。
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现阶段所有的连接器实现及可参考的示例都在seatunnel-connectors-v2下,用户可自行查阅参考


### **Example**
We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example`` in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector. The corresponding configuration files are saved in the same module ``resources/examples`` folder as before.
For now, the test needs to rely on the Connector,if your test must use a third-party jar and the Scope is Provided,you need to check "Include dependencies with 'Provided Scope" and choose current Project in "Run"->"Edit configurations..."->"Use classpath of module" .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, you need to add the dependencies of the corresponding connectors to the pom file for testing under the seatunnel-examples module, if there are some three-party dependencies, please add them to the module classpath by yourself, if the scope of some dependencies of the connectors is provided, you also need to set Include dependencies with Provided Scope in Intellij IDEA.

@@ -2,10 +2,11 @@
This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SeaTunnel has designed a new connector API to decouple from the compute engine. This article introduces the new interface and the new code structure to help developers quickly get started with the new API and understand the principles of the new API.

@EricJoy2048
Copy link
Member

That's great.

Thank you for your contribution.

@liugddx
Copy link
Member Author

liugddx commented Sep 21, 2022

@liugddx
Copy link
Member Author

liugddx commented Sep 21, 2022

@TyrantLucifer @EricJoy2048

PTAL

@liugddx
Copy link
Member Author

liugddx commented Sep 24, 2022

#2873

@TyrantLucifer
Copy link
Member

@TyrantLucifer @EricJoy2048

PTAL

Now the document still has some incoherent and unreasonable places, please modify according to the comments.

@@ -1,48 +1,101 @@
# Purpose
This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.

This article introduces the new interface and the new code structure on account of the newly designed API for Connectors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you link the proposal #1608

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please link proposal here like the Chinese document.

This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.

This article introduces the new interface and the new code structure on account of the newly designed API for Connectors
in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helps developers quickly understand API and transformation layer improvements. On the other hand, it can guide contributors how to use the new API to develop new connectors. is better.


We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which can be
directly called using seatunnel-flink-connector-v2-example or seatunnel-spark-connector-v2-example
in SeaTunnelApiExample

One is seatunnel-examples/seatunnel-flink-connector-v2-example/src/main/java/org/apache/seatunnel/example/flink/v2/SeaTunnelApiExample.java, it runs in the Flink engine. Another one is seatunnel-examples/seatunnel-spark-connector-v2-example/src/main/java/org/apache/seatunnel/example/spark/v2/SeaTunnelApiExample.java, it runs in the Spark engine.


We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have prepared a new version of the locally executable example program in seatunnel-examples

We have prepared two new version of the locally executable example program in seatunnel-examples.

We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector.
The corresponding configuration files are saved in the same module ``resources/examples`` folder as before. Currently,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The corresponding configuration files are saved in the same module resources/examples folder as before

The configuration files used in example are saved in the "resources/examples" folder..

I think it is enough to clarify the location of the example configuration file. It is unnecessary to guide the user to understand the previous configuration.


We have prepared a new version of the locally executable example program in ``seatunnel-examples``, which can be
directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also the debugging method that is often used in the local development of Connector.

This is also the debugging method that is often used in the local development of Connector. You can debug these examples, which will help you better understand the running logic of the program.

directly called using ``seatunnel-flink-connector-v2-example`` or ``seatunnel-spark-connector-v2-example``
in ``SeaTunnelApiExample``. This is also the debugging method that is often used in the local development of Connector.
The corresponding configuration files are saved in the same module ``resources/examples`` folder as before. Currently,
you need to add the dependencies of the corresponding connectors to the pom file for testing under the
Copy link
Member

@EricJoy2048 EricJoy2048 Sep 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to add the dependencies of the corresponding connectors to the pom file for testing under the
seatunnel-examples module, if there are some three-party dependencies, please add them to the module classpath by
yourself, if the scope of some dependencies of the connectors is provided, you also need to set Include dependencies
with Provided Scope in Intellij IDEA.

I think these are guidelines for contributors on how to add examples to their connectors. This requires clearer steps.

If you want to add examples for your own connectors, you need to follow the steps below.

  1. Add the groupId, artifactId and version of the connector to be tested to seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(or add it to seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml when you want to runs it in Spark engine) as a dependency.
  2. Find the dependency in your connector pom file which scope is test or provided and then add them to seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(or add it to seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml file and modify the scope to compile.
  3. Refer to the SeaTunnelApiExample class to develop your sample code.


### **Startup Class**
Aside from the old startup class, we have created two new startup class projects, namely ``seatunnel-core/seatunnel-flink-starter`` and ``seatunnel-core/seatunnel-spark-starter``. You can find out how to parse the configuration file into an executable Flink/Spark process here.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside from the old startup class, we have created two new startup class projects

Aside from the old startup module, we have created two new startup modules.

@liugddx
Copy link
Member Author

liugddx commented Sep 24, 2022

done.

```java
if (Boundedness.BOUNDED.equals(context.getBoundedness())) {
if(Boundedness.BOUNDED.equals(context.getBoundedness())){
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (Boundedness.BOUNDED.equals(context.getBoundedness())) {

@@ -1,48 +1,101 @@
# Purpose
This article introduces the new interface and the new code structure on account of the newly designed API for Connectors in Apache SeaTunnel. This helps developers with quick overview regarding API, translation layer improvement, and development of new Connector.

This article introduces the new interface and the new code structure on account of the newly designed API for Connectors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please link proposal here like the Chinese document.

@liugddx
Copy link
Member Author

liugddx commented Sep 24, 2022

done

@TyrantLucifer
Copy link
Member

done

image

image

Please change the content of the above screenshot, thanks.

@liugddx
Copy link
Member Author

liugddx commented Sep 25, 2022

done


1. 在`seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml`添加connector依赖的groupId, artifactId 和
version.(或者当你想在spark引擎运行时在`seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml`添加依赖)
2. 如果你的connector存在scope为test或者provided的依赖则在`seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml`(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果你的connector存在scope为test或者provided的依赖则在seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(
或者在seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml)修改为compile。

如果你的connector中存在scope为test或provided的依赖,将这些依赖添加到seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml(
或者在seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml)中,并且修改scope为compile.

@liugddx
Copy link
Member Author

liugddx commented Sep 26, 2022

Done. @EricJoy2048

@Hisoka-X Hisoka-X linked an issue Sep 26, 2022 that may be closed by this pull request
3 tasks
Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @TyrantLucifer PTAL

@TyrantLucifer TyrantLucifer merged commit 2af310a into apache:dev Sep 26, 2022
@liugddx liugddx deleted the improve-doc-connectorv2 branch September 26, 2022 11:53
Hisoka-X added a commit that referenced this pull request Oct 27, 2022
* seatunnel-metrics init commit

* seatunnel-metrics config add

* codeStyle update

* codeStyle update again

* codeStyle seatunnel-spark update

* [Imporve][Connector-V2]Parameter verification for connector V2 kafka sink (#2866)

* parameter verification

* update

* update

* [Improve][DOC] Perfect the connector v2 doc (#2800)

* [Improve][DOC] Perfect the connector v2 doc

* Update seatunnel-connectors-v2/README.zh.md

Co-authored-by: Hisoka <fanjiaeminem@qq.com>

* [Improve][DOC] A little tinkering

* [Improve][DOC] A little tinkering

* [Doc][connector] add Console sink doc

close #2794

* [Doc][connector] add Console sink doc

close #2794

* fix some problem

* fix some problem

* fine tuning

Co-authored-by: Hisoka <fanjiaeminem@qq.com>

* add seatunnel-examples from gitignore (#2892)

* [Improve][connector-jdbc] Calculate splits only once in JdbcSourceSplitEnumerator (#2900)

* [Bug][Connector-V2] Fix wechat sink data serialization (#2856)

* [Improve][Connector-V2] Improve orc write strategy to support all data types (#2860)

* [Improve][Connector-V2] Improve orc write strategy to support all data types

Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com>

* [Bug][seatunnel-translation-base] Fix Source restore state NPE (#2878)

* [Improve][Connector-v2-Fake]Supports direct definition of data values(row) (#2839)

* [Improve][Connector-v2]Supports direct definition of data values(row)

* seatunnel-prometheus update

* seatunnel-prometheus update

* seatunnel-prometheus update

* 1. Seatunnel unified configuration naming
2. Use reflection to automate assembly
3. Modify the flink/spark startup function
4. Try packaging configuration (todo)

Co-authored-by: TaoZex <45089228+TaoZex@users.noreply.github.com>
Co-authored-by: liugddx <804167098@qq.com>
Co-authored-by: Hisoka <fanjiaeminem@qq.com>
Co-authored-by: Eric <gaojun2048@gmail.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Co-authored-by: hailin0 <wanghailin@apache.org>
Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com>
Co-authored-by: Laglangyue <35491928+laglangyue@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] [seatunnel-examples] run jdbc task will error in examples
4 participants