Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

basic support for user provided schema in the query path #18031

Merged
merged 2 commits into from
Dec 9, 2020
Merged

basic support for user provided schema in the query path #18031

merged 2 commits into from
Dec 9, 2020

Conversation

moderakh
Copy link
Contributor

@moderakh moderakh commented Dec 9, 2020 •
edited
Loading

Cosmos Spark3 DataSourceV2 basic support for user provided schema in the query path

  • basic support for user provided schema in the query path
  • integration e2e test for query with user provided schema see SparkE2EQuerySpec for e2e test.

sample usage:

val customSchema = StructType(Array(
  StructField("id", StringType),
  StructField("name", StringType),
  StructField("type", StringType),
  StructField("age", IntegerType),
  StructField("isAlive", BooleanType)
))

val df = spark.read.schema(customSchema).format("cosmos.items").options(cfg).load()

For now we only support user provided schema in the query path.
TODO:

  • schema inference support

@ghost ghost added the Cosmos label Dec 9, 2020
@moderakh moderakh added the cosmos:spark3 Cosmos DB Spark3 OLTP Connector label Dec 9, 2020
FabianMeiswinkel
FabianMeiswinkel approved these changes Dec 9, 2020
View reviewed changes
Copy link
Member

@FabianMeiswinkel FabianMeiswinkel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@moderakh moderakh merged commit 4a5459a into Azure:feature/cosmos/spark30 Dec 9, 2020
@moderakh moderakh deleted the users/moderakh/user-provided-schema-on-read-path branch December 9, 2020 17:58
@moderakh moderakh linked an issue Dec 18, 2020 that may be closed by this pull request
basic support for user provided schema in the query path #18239
Closed
@moderakh moderakh mentioned this pull request Dec 18, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FabianMeiswinkel FabianMeiswinkel FabianMeiswinkel approved these changes

@kirankumarkolli kirankumarkolli Awaiting requested review from kirankumarkolli

@kushagraThapar kushagraThapar Awaiting requested review from kushagraThapar

@mbhaskar mbhaskar Awaiting requested review from mbhaskar

@milismsft milismsft Awaiting requested review from milismsft

@simplynaveen20 simplynaveen20 Awaiting requested review from simplynaveen20

@xinlian12 xinlian12 Awaiting requested review from xinlian12

Assignees
No one assigned
Labels
cosmos:spark3 Cosmos DB Spark3 OLTP Connector Cosmos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

basic support for user provided schema in the query path
2 participants
@moderakh @FabianMeiswinkel