[SPARK-33432][SQL] SQL parser should use active SQLConf

[luluorta]

What changes were proposed in this pull request?

This PR makes SQL parser using active SQLConf instead of the one in ctor-parameters.

Why are the changes needed?

In ANSI mode, schema string parsing should fail if the schema uses ANSI reserved keyword as attribute name:

spark.conf.set("spark.sql.ansi.enabled", "true")
spark.sql("""select from_json('{"time":"26/10/2015"}', 'time Timestamp', map('timestampFormat',  'dd/MM/yyyy'));""").show

output:

Cannot parse the data type:
no viable alternative at input 'time'(line 1, pos 0)

== SQL ==
time Timestamp
^^^

But this query may accidentally succeed in certain cases cause the DataType parser sticks to the configs of the first created session in the current thread:

DataType.fromDDL("time Timestamp")
val newSpark = spark.newSession()
newSpark.conf.set("spark.sql.ansi.enabled", "true")
newSpark.sql("""select from_json('{"time":"26/10/2015"}', 'time Timestamp', map('timestampFormat', 'dd/MM/yyyy'));""").show

output:

+--------------------------------+
|from_json({"time":"26/10/2015"})|
+--------------------------------+
| {2015-10-26 00:00...|
+--------------------------------+

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Newly and updated UTs

[SparkQA]

Test build #131011 has finished for PR 30357 at commit a166c40.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with Logging
• abstract class AbstractSqlParser extends ParserInterface with Logging
• class CatalystSqlParser extends AbstractSqlParser
• class SparkSqlParser extends AbstractSqlParser
• class SparkSqlAstBuilder extends AstBuilder
• class VariableSubstitution

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35637/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35637/

[HyukjinKwon]

cc @cloud-fan

[SparkQA]

Test build #131031 has finished for PR 30357 at commit 6d02b6c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with Logging
• abstract class AbstractSqlParser extends ParserInterface with Logging
• class CatalystSqlParser extends AbstractSqlParser
• class SparkSqlParser extends AbstractSqlParser
• class SparkSqlAstBuilder extends AstBuilder
• class VariableSubstitution

[cloud-fan]

LGTM except one comment

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35653/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35653/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35658/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35658/

[SparkQA]

Test build #131047 has finished for PR 30357 at commit 8464da0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #131049 has finished for PR 30357 at commit 4139f6d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #131053 has finished for PR 30357 at commit acb19f2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

This style changes look like another benefit. Thanks.

[dongjoon-hyun]

+1, LGTM. Thank you , @luluorta and @cloud-fan .
I also verified in spark-shell interactively.

Merged to master for Apache Spark 3.1.

[dongjoon-hyun]

If this is required at branch-3.0 as described in JIRA, please make a backporting PR, @luluorta .