Cosmos Spark End to End Integration Test against Cosmos Emulator runs in CI #17952
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
the
moderakh
changed the title
moderakh
changed the title
moderakh
requested review from
danieljurek,
FabianMeiswinkel,
JimSuplizio,
kirankumarkolli,
kushagraThapar,
mbhaskar,
milismsft,
mitchdenny,
simplynaveen20,
weshaggard and
xinlian12
as code owners
| override def pruneColumns(requiredSchema: StructType): Unit = { | ||
| // TODO moderakh add projection to the query | ||
| // TODO moderakh: we need to decide whether do a push down or not on the projection |
There was a problem hiding this comment.
Good point - I think it might be useful to see whether we can make that decision based on "avg." document size? Like < 1 KB don't push down pruning - but for larger documents do it?
There was a problem hiding this comment.
Not blocking of course...
There was a problem hiding this comment.
Thanks for the suggestion. good idea. I will look into this.
moderakh
added a commit
that referenced
this pull request
Dec 8, 2020
This PR adds support for spark3 DataSourceV2 Catalog API: NOTE: this PR is the same as this PR (moderakh#15) targeting Azure repo. The original PR is already reviewed and signed off by reviewers. ```scala spark.conf.set(s"spark.sql.catalog.cosmoscatalog", "com.azure.cosmos.spark.CosmosCatalog") spark.conf.set(s"spark.sql.catalog.cosmoscatalog.spark.cosmos.accountEndpoint", cosmosEndpoint) spark.conf.set(s"spark.sql.catalog.cosmoscatalog.spark.cosmos.accountKey", cosmosMasterKey) spark.sql(s"CREATE DATABASE cosmoscatalog.mydb;") spark.sql(s"CREATE TABLE cosmoscatalog.mydb.myContainer (word STRING, number INT) using cosmos.items TBLPROPERTIES(partitionKeyPath = '/mypk', manualThroughput = '1100')") ``` Please see `CosmosCatalogSpec` for end to end integration tests. The integration testings will work once this earlier PR merges: #17952 getting merged. TODO: - There are some TODO in the code, (e.g., add support for table alter) - the integration tests resource management needs to be figured out. - This PR adds support for catalog metadata operation, we should also validate data operation through catalog api.
openapi-sdkautomation bot
pushed a commit
to AzureSDKAutomation/azure-sdk-for-java
that referenced
this pull request
Feb 23, 2022
Change SubscriptionIdParameter to client instead of method (Azure#17952)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Now we have gated CI for end to end test spark <-> cosmos db emulator.
Cosmos Spark End to End Test against Cosmos Emulator runs in CI.
RequiresCosmosEndpointwill get included in sparkE2E test group and executed by the newly added CI, Spark_Integration_Tests_Java8 against cosmos db emulator.SparkE2EWriteSpec.scalafor end to end spark sample test. It writes data from spark to cosmos db emulator. Emulator CIWe have two spark test groups running in the CI:
how to run locally in your dev machine:
mvn -e -Dgpg.skip -Dmaven.javadoc.skip=true -Dspotbugs.skip=true -Dcheckstyle.skip=true -Drevapi.skip=true -pl ,azure-cosmos-spark_3-0_2-12 -am clean testhow to run locally in your dev machine:
mvn -e -Dgpg.skip -Dmaven.javadoc.skip=true -Dspotbugs.skip=true -Dcheckstyle.skip=true -Drevapi.skip=true -pl ,azure-cosmos-spark_3-0_2-12 -am -PsparkE2E clean testTODO:
-- proper resource (Database, Container) cleaning
-- proper shutdown of CosmosClient and Spark session
-- possible sharing of the CosmosClient and spark session between tests
-- which scala test style should be used?