Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Imporve][Fake-Connector-V2]support user-defined-schmea and random data for fake-table #2406

Merged
merged 23 commits into from
Aug 23, 2022

Conversation

laglangyue
Copy link
Contributor

@laglangyue laglangyue commented Aug 13, 2022

Purpose of this pull request

the connector-v2-fake did't not support enough data type, and not quickly to modify,more time we need the fake to test the dataType Mapper,
so I do these works:
(1) optimization connector-fake-v2 so that we can easy to add column name,type and data
(2) change the spark example for connector-fake-v2,and fix some dead Http link in this example conf file
(3) I find the method don't contain the condition SqlType=NULL: org.apache.seatunnel.translation.serialization.RowConverter#validate,I don't sure it OK。and the SqlType of VOID_TYPE is NULL
(4) the error message is bad in org.apache.seatunnel.translation.serialization.RowConverter(line 60), i fix it

Check list

@laglangyue laglangyue changed the title [Connector-V2][JDBC-connector] optimization fake [Connector-V2][fake-connector] optimization fake Aug 13, 2022
@laglangyue
Copy link
Contributor Author

I find SeatunnelSchema that it not implement Serializable, it Example-task will error.
I add it.

@laglangyue
Copy link
Contributor Author

the PR has completed. I add Junit5-test and test random data correctly.
For some complex types, example-v2-spark may occur some exceptions, but they are not the reason for Fake-V2, and during the spark type conversion process. I will test and try to repair them in another PR, because there are too many in this PR

Copy link
Member

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add the connectors-v2/FakeSource docs?

@laglangyue laglangyue changed the title [Connector-V2][fake-connector] optimization fake [Imporve][Fake-Connector-V2]support user-defined-schmea and random data for fake-table Aug 21, 2022
@hailin0
Copy link
Member

hailin0 commented Aug 21, 2022

Can you add the connectors-v2/FakeSource docs?

here https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2/source

@laglangyue
Copy link
Contributor Author

Can you add the connectors-v2/FakeSource docs?

yes , I update the config to keep the schema parameters consistent such as LocalFile and HDFS source.
Thank you for your guidance. This is my first contribution in Github.

@hailin0
Copy link
Member

hailin0 commented Aug 23, 2022

LGTM
@CalvinKirs

@CalvinKirs CalvinKirs merged commit a544752 into apache:dev Aug 23, 2022
@laglangyue laglangyue deleted the fake_data branch September 5, 2022 12:51
TyrantLucifer pushed a commit to TyrantLucifer/incubator-seatunnel that referenced this pull request Sep 18, 2022
…ta for fake-table (apache#2406)

* [Connector-V2][JDBC-connector] optimization fake
Co-authored-by: tangjiafu <tangjiafu@corp.netease.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants