Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-v2]add raw format with KafkaSource #4844

Closed
wants to merge 16 commits into from

Conversation

lightzhao
Copy link
Contributor

Purpose of this pull request

Check list

@lightzhao
Copy link
Contributor Author

@Hisoka-X @EricJoy2048 @TyrantLucifer please approve ci.

Copy link
Contributor

@TaoZex TaoZex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add UT for Raw format.

@lightzhao
Copy link
Contributor Author

Please add UT for Raw format.
e2e has been added, please approve ci.

@ic4y
Copy link
Contributor

ic4y commented Aug 5, 2023

@lightzhao Please merge the latest dev branch and resolve the conflicts.

@lightzhao
Copy link
Contributor Author

@lightzhao Please merge the latest dev branch and resolve the conflicts.

fixed.

lightzhao and others added 4 commits August 14, 2023 15:05
# Conflicts:
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/config/MessageFormat.java
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaSource.java
# Conflicts:
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/config/MessageFormat.java
@lightzhao
Copy link
Contributor Author

@TaoZex @ic4y @Hisoka-X PTAL.

@lightzhao
Copy link
Contributor Author

image Why is there such an exception? It has nothing to do with the module I submitted, and the latest code has also been merged.

@ic4y ic4y closed this Sep 6, 2023
@ic4y ic4y reopened this Sep 6, 2023
@lightzhao
Copy link
Contributor Author

@TaoZex @ic4y @EricJoy2048 @Hisoka-X PTAL.

yx91490
yx91490 previously approved these changes Oct 2, 2023
Copy link
Member

@TyrantLucifer TyrantLucifer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we specify format as text and do not configure schema information, the record will be output as is, so I believe this modification has no meaning. cc @EricJoy2048 @hailin0

@hailin0
Copy link
Member

hailin0 commented Oct 7, 2023

When we specify format as text and do not configure schema information, the record will be output as is, so I believe this modification has no meaning. cc @EricJoy2048 @hailin0

+1

This is a duplicate feature

@lightzhao
Copy link
Contributor Author

When we specify format as text and do not configure schema information, the record will be output as is, so I believe this modification has no meaning.

If schema information is not set, how should field information be obtained in the transform and sink phases?

@EricJoy2048
Copy link
Member

When we specify format as text and do not configure schema information, the record will be output as is, so I believe this modification has no meaning.

If schema information is not set, how should field information be obtained in the transform and sink phases?

Schema is required when read message from kafka.

@EricJoy2048
Copy link
Member

When we specify format as text and do not configure schema information, the record will be output as is, so I believe this modification has no meaning. cc @EricJoy2048 @hailin0

+1

@lightzhao
Copy link
Contributor Author

When we specify format as text and do not configure schema information, the record will be output as is, so I believe this modification has no meaning. cc @EricJoy2048 @hailin0

+1

I don’t understand the previous and later replies. You need to set the schema, but the alternative method is not to set it. What does it mean?

lightzhao added 5 commits November 7, 2023 10:20
# Conflicts:
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/config/MessageFormat.java
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/serialize/DefaultSeaTunnelRowSerializer.java
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaSource.java
…nnel into add-raw-format

� Conflicts:
�	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/config/MessageFormat.java
�	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaSource.java
# Conflicts:
#	seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/serialize/DefaultSeaTunnelRowSerializer.java
#	seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaIT.java
@lightzhao lightzhao closed this Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants