-
Notifications
You must be signed in to change notification settings - Fork 349
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Custom JDBC column types #220
Conversation
Seems reasonable to me; do you mind adding a simple test which exercises this code path? |
Signed-off-by: Marc-André Tremblay <marcandre.tr@gmail.com>
Sorry for the delay. I based the test on the existing "maxlength" feature test. |
case _ => throw new IllegalArgumentException(s"Don't know how to save $field to JDBC") | ||
val typ: String = if (field.metadata.contains("redshift_type")) { | ||
field.metadata.getString("redshift_type") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor style nit: place the else
on the same line as this brace.
LGTM. I'll try to fix the style issue and merge conflicts myself. |
Author: Marc-André Tremblay <marcandre.tr@gmail.com> This patch had conflicts when merged, resolved by Committer: Josh Rosen <joshrosen@databricks.com> Closes databricks#220 from marctrem/custom-jdbc-column-types.
@JoshRosen We are facing the same issue in the general JDBC data source of Spark. What do you think if we do it in Spark too? |
@gatorsmile, I wouldn't be opposed. |
Thanks! |
@JoshRosen We submitted two solutions:
I am wondering which solution you prefer? It sounds like the first one is more user friendly. val mdb = new MetadataBuilder()
mdb.putString("name", "VARCHAR(128)”)
mdb.putString("comments”, “CLOB(20K)”)
val createTableColTypes = mdb.build().json
df.write.option("createTableColumnTypes", createTableColTypes).jdbc(url, "TEST.DBCOLTYPETEST", properties) |
This patch allows us to set custom column types.
Please tell me if you want some edits on it.
Thank you,
Marc