Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-33703][SQL] Migrate MSCK REPAIR TABLE to use UnresolvedTable to resolve the identifier #30664

Closed
wants to merge 2 commits into from

Conversation

imback82
Copy link
Contributor

@imback82 imback82 commented Dec 8, 2020

What changes were proposed in this pull request?

This PR proposes to migrate MSCK REPAIR TABLE to use UnresolvedTable to resolve the table identifier. This allows consistent resolution rules (temp view first, etc.) to be applied for both v1/v2 commands. More info about the consistent resolution rule proposal can be found in JIRA or proposal doc.

Note that MSCK REPAIR TABLE is not supported for v2 tables.

Why are the changes needed?

The PR makes the resolution consistent behavior consistent. For example,

sql("CREATE DATABASE test")
sql("CREATE TABLE spark_catalog.test.t (id bigint, val string) USING csv PARTITIONED BY (id)")
sql("CREATE TEMPORARY VIEW t AS SELECT 2")
sql("USE spark_catalog.test")
sql("MSCK REPAIR TABLE t") // works fine

, but after this PR:

sql("MSCK REPAIR TABLE t")
org.apache.spark.sql.AnalysisException: t is a temp view. 'MSCK REPAIR TABLE' expects a table; line 1 pos 0

, which is the consistent behavior with other commands.

Does this PR introduce any user-facing change?

After this PR, MSCK REPAIR TABLE t in the above example is resolved to a temp view t first instead of spark_catalog.test.t.

How was this patch tested?

Updated existing tests.

@github-actions github-actions bot added the SQL label Dec 8, 2020
@SparkQA
Copy link

SparkQA commented Dec 8, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37011/

@SparkQA
Copy link

SparkQA commented Dec 8, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37011/

@SparkQA
Copy link

SparkQA commented Dec 8, 2020

Test build #132411 has finished for PR 30664 at commit 3e8aa99.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class RepairTable(child: LogicalPlan) extends Command

@imback82
Copy link
Contributor Author

imback82 commented Dec 8, 2020

cc @cloud-fan

@SparkQA
Copy link

SparkQA commented Dec 8, 2020

Test build #132442 has finished for PR 30664 at commit f6b776e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • trait CheckAnalysis extends PredicateHelper with LookupCatalog
  • case class UnresolvedView(
  • case class NoopCommand(
  • case class ShowTableExtended(
  • case class DropView(

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in 29fed23 Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants