Support for merging spark application config with dataframe config.

[moderakh]

This PR adds support for merging spark application config with dataframe options.
Cosmos Spark3 DataSourceV2.

example:

val sparkConfig = new SparkConf()
sparkConfig.set("spark.cosmos.accountEndpoint", cosmosEndpoint)
sparkConfig.set("spark.cosmos.accountKey", cosmosMasterKey)

val spark = SparkSession.builder()
  .appName("spark connector sample")
  .master("local")
  .config(sparkConfig)
  .getOrCreate()

val options = Map(
  "spark.cosmos.database" -> cosmosDatabase,
  "spark.cosmos.container" -> cosmosContainer
)
val df = spark.read.format("cosmos.items").options(options).load()

see SparkE2EConfigResolutionSpec e2e test for more examples.

TODO:

• we should investigate how to merge spark sql config.
• we should investigate how to integrate this with catalog api (if possible)

[kirankumarkolli]

What are the OLAP configs?

One more alternative is spark.azure.cosmos.accountEndpoint

[FabianMeiswinkel]

Nice one!

[moderakh]

What are the OLAP configs?

One more alternative is spark.azure.cosmos.accountEndpoint

@kirankumarkolli we are using the same config name prefixes in OLAP and synapse unified spark connector.

Before we GA the spark3 oltp connector, similar to sdk we should go over all the configs and review together.