-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OLTP spark query pipeline draft on DataSourceV2 spark3 #17774
OLTP spark query pipeline draft on DataSourceV2 spark3 #17774
Conversation
.master("local") | ||
.getOrCreate() | ||
|
||
val df = spark.read.format("cosmos.items").options(cfg).load() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should discuss the format name - "cosmos.items" feels a little off to me.. can we go with the style we have in the unified Spark connector in Synapse
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's discuss in the scrum.
sdk/cosmos/azure-cosmos-spark_3-0_2-12/src/main/scala/com/azure/cosmos/spark/CosmosScan.scala
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Mo!
This PR adds the support for spark query DataSourceV2 pipeline
TODO: we need to discuss on the following items who does what (to be done after this PR):
TestReadE2EMain
work. I have a separate PR supporting filter translation: spark filter to cosmos db pushdown query #17789TestReadE2EMain
work.