Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark: Add column comment when create clickhouse table #176

Merged
merged 1 commit into from
Aug 8, 2022

Conversation

sketchmind
Copy link
Contributor

close #174

@pan3793
Copy link
Collaborator

pan3793 commented Aug 4, 2022

Thanks, LGTM, would you please make this change in spark-3.3 as well? And better to add a unit test for verification(encouraged but not mandatory)

@sketchmind
Copy link
Contributor Author

Thanks, LGTM, would you please make this change in spark-3.3 as well? And better to add a unit test for verification(encouraged but not mandatory)

I am now using spark-clickhouse-connector with pyspark, throw java.lang.ClassNotFoundException: xenon.clickhouse.Logging error, which makes me unable to test temporarily

@sketchmind
Copy link
Contributor Author

Thanks, LGTM, would you please make this change in spark-3.3 as well? And better to add a unit test for verification(encouraged but not mandatory)

ok,I forgot, I will add it right away

@pan3793
Copy link
Collaborator

pan3793 commented Aug 4, 2022

I am now using spark-clickhouse-connector with pyspark, throw java.lang.ClassNotFoundException: xenon.clickhouse.Logging error, which makes me unable to test temporarily

Please use spark-3.2/clickhouse-spark-runtime/build/libs/clickhouse-spark-runtime-3.2_2.12-0.5.0-SNAPSHOT.jar for testing.

@pan3793
Copy link
Collaborator

pan3793 commented Aug 4, 2022

Please take attention, I just merged 5dbc2fd, which changed the build command from ./gradlew build -x test -Dspark_binary_versions=3.2 to ./gradlew build -x test -Dspark_binary_version=3.2

@sketchmind sketchmind force-pushed the Feature-174 branch 2 times, most recently from 30b115b to 5246ee3 Compare August 4, 2022 08:45
@sketchmind
Copy link
Contributor Author

I am now using spark-clickhouse-connector with pyspark, throw java.lang.ClassNotFoundException: xenon.clickhouse.Logging error, which makes me unable to test temporarily

Please use spark-3.2/clickhouse-spark-runtime/build/libs/clickhouse-spark-runtime-3.2_2.12-0.5.0-SNAPSHOT.jar for testing.

I'm using this file from the beginning, I'll try again

@pan3793
Copy link
Collaborator

pan3793 commented Aug 4, 2022

LGTM, please click button "Ready for review" when you passed your local testing.

@pan3793 pan3793 changed the title add column comment when create clickhouse table Spark: Add column comment when create clickhouse table Aug 4, 2022
@sketchmind
Copy link
Contributor Author

LGTM, please click button "Ready for review" when you passed your local testing.

My pyspark has another problem with jackson during clickhouse initialization. After I test it locally, I will click the ready button

@sketchmind
Copy link
Contributor Author

sketchmind commented Aug 4, 2022

LGTM, please click button "Ready for review" when you passed your local testing.

My pyspark has another problem with jackson during clickhouse initialization. After I test it locally, I will click the ready button

It seems that my JDBC version is too low cause throw an error

@pan3793
Copy link
Collaborator

pan3793 commented Aug 4, 2022

@sketchmind
Copy link
Contributor Author

The master branch code depends on the latest clickhouse client, you can choose to use one of the following jars

I don't know why it can't be tested locally, including using scala. I click ready first, you need to do a local test of the feature

@sketchmind sketchmind marked this pull request as ready for review August 8, 2022 07:33
@pan3793
Copy link
Collaborator

pan3793 commented Aug 8, 2022

OK, let me merge it first and do manual test then.

@pan3793 pan3793 merged commit b7cbe97 into ClickHouse:master Aug 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[feature] add column comment based dataframe scheme when create table
2 participants