Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-26794][SQL]SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists #23709

Closed
wants to merge 15 commits into from
Closed

[SPARK-26794][SQL]SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists #23709

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
275e998
fix #SPARK-26794 SparkSession enableHiveSupport does not point to hiv…
yaooqinn Jan 31, 2019
e1eafe7
fix compile
yaooqinn Jan 31, 2019
4a1a18b
add ut
yaooqinn Jan 31, 2019
62d2aa3
clone conf and apply all initiate options to them
yaooqinn Feb 1, 2019
8004c19
typo & fix ut
yaooqinn Feb 1, 2019
10ebff7
not import map
yaooqinn Feb 1, 2019
f5c15e0
naming
yaooqinn Feb 1, 2019
d57226a
set conf to lazy val
yaooqinn Feb 1, 2019
0c07ff1
fix ut
yaooqinn Feb 1, 2019
35b3f17
add some comments
yaooqinn Feb 1, 2019
0cac5e6
add more ut
yaooqinn Feb 1, 2019
92fec4d
fix ut
yaooqinn Feb 2, 2019
391e14c
mv confs behind warehousePath
yaooqinn Feb 12, 2019
eff569b
meaningful titles for uts
yaooqinn Feb 12, 2019
748821c
rm one ut
yaooqinn Feb 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ class SparkSession private(
@Unstable
@transient
lazy val sharedState: SharedState = {
existingSharedState.getOrElse(new SharedState(sparkContext))
existingSharedState.getOrElse(new SharedState(sparkContext, initialSessionOptions.toMap))
HyukjinKwon marked this conversation as resolved.
Show resolved Hide resolved
}

/**
Expand Down
12 changes: 8 additions & 4 deletions sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ import org.apache.spark.util.{MutableURLClassLoader, Utils}
/**
* A class that holds all state shared across sessions in a given [[SQLContext]].
*/
private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {
private[sql] class SharedState(val sparkContext: SparkContext, initConfig: Map[String, String])
HyukjinKwon marked this conversation as resolved.
Show resolved Hide resolved
extends Logging {

// Load hive-site.xml into hadoopConf and determine the warehouse path we want to use, based on
// the config from both hive and Spark SQL. Finally set the warehouse config value to sparkConf.
Expand Down Expand Up @@ -101,7 +102,7 @@ private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {
*/
lazy val externalCatalog: ExternalCatalogWithListener = {
val externalCatalog = SharedState.reflect[ExternalCatalog, SparkConf, Configuration](
SharedState.externalCatalogClassName(sparkContext.conf),
SharedState.externalCatalogClassName(sparkContext.conf, initConfig),
sparkContext.conf,
sparkContext.hadoopConfiguration)

Expand Down Expand Up @@ -165,8 +166,11 @@ object SharedState extends Logging {

private val HIVE_EXTERNAL_CATALOG_CLASS_NAME = "org.apache.spark.sql.hive.HiveExternalCatalog"

private def externalCatalogClassName(conf: SparkConf): String = {
conf.get(CATALOG_IMPLEMENTATION) match {
private def externalCatalogClassName(
conf: SparkConf,
initSessionConfig: Map[String, String]): String = {
HyukjinKwon marked this conversation as resolved.
Show resolved Hide resolved
initSessionConfig
.getOrElse(CATALOG_IMPLEMENTATION.key, conf.get(CATALOG_IMPLEMENTATION)) match {
case "hive" => HIVE_EXTERNAL_CATALOG_CLASS_NAME
case "in-memory" => classOf[InMemoryCatalog].getCanonicalName
}
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ private[hive] class TestHiveExternalCatalog(
private[hive] class TestHiveSharedState(
sc: SparkContext,
hiveClient: Option[HiveClient] = None)
extends SharedState(sc) {
extends SharedState(sc, Map.empty) {
yaooqinn marked this conversation as resolved.
Show resolved Hide resolved

override lazy val externalCatalog: ExternalCatalogWithListener = {
new ExternalCatalogWithListener(new TestHiveExternalCatalog(
Expand Down