Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-24252][SQL] Add TableCatalog API #24246

Closed
wants to merge 14 commits into from
Closed
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@

package org.apache.spark.sql.catalog.v2;

import com.google.common.base.Preconditions;
import org.apache.spark.annotation.Experimental;

import java.util.Arrays;
import java.util.Objects;

/**
* An {@link Identifier} implementation.
*/
Expand All @@ -29,6 +33,8 @@ class IdentifierImpl implements Identifier {
private String name;

IdentifierImpl(String[] namespace, String name) {
Preconditions.checkNotNull(namespace, "Identifier namespace cannot be null");
Preconditions.checkNotNull(name, "Identifier name cannot be null");
this.namespace = namespace;
this.name = name;
}
Expand All @@ -42,4 +48,23 @@ public String[] namespace() {
public String name() {
return name;
}

@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}

if (o == null || getClass() != o.getClass()) {
return false;
}

IdentifierImpl that = (IdentifierImpl) o;
return Arrays.equals(namespace, that.namespace) && name.equals(that.name);
}

@Override
public int hashCode() {
return Objects.hash(Arrays.hashCode(namespace), name);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.sql.catalog.v2;

import org.apache.spark.sql.catalog.v2.expressions.Transform;
import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
import org.apache.spark.sql.sources.v2.Table;
import org.apache.spark.sql.types.StructType;

import java.util.Map;

/**
* Catalog methods for working with Tables.
* <p>
* TableCatalog implementations may be case sensitive or case insensitive. Spark will pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may worth to add a method to report this information. We can think about it later.

* {@link Identifier table identifiers} without modification. Field names passed to
* {@link #alterTable(Identifier, TableChange...)} will be normalized to match the case used in the
* table schema when updating, renaming, or dropping existing columns when catalyst analysis is case
* insensitive.
*/
public interface TableCatalog extends CatalogPlugin {
Copy link
Member

@gatorsmile gatorsmile May 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still not sure whether we should have a dedicated ViewCatalog for view management, like what the proposal suggests. For the catalog API developers, metadata management of views and tables are very similar. Do they need to implement two catalogs? cc @rxin @marmbrus

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only similarity is that the two typically share the same namespace. The implementation is very different and will use different code paths. I think that it over-complicates the API to return the two using the same load method, when the alternative is to have two resolution rules instead.

That said, it would be easier to understand what you're asking for if you wrote up a proposed API. I can add this topic to the next v2 sync if you plan to propose an alternative.

/**
* List the tables in a namespace from the catalog.
* <p>
* If the catalog supports views, this must return identifiers for only tables and not views.
*
* @param namespace a multi-part namespace
* @return an array of Identifiers for tables
* @throws NoSuchNamespaceException If the namespace does not exist (optional).
*/
Identifier[] listTables(String[] namespace) throws NoSuchNamespaceException;

/**
* Load table metadata by {@link Identifier identifier} from the catalog.
* <p>
* If the catalog supports views and contains a view for the identifier and not a table, this
* must throw {@link NoSuchTableException}.
*
* @param ident a table identifier
* @return the table's metadata
* @throws NoSuchTableException If the table doesn't exist or is a view
*/
Table loadTable(Identifier ident) throws NoSuchTableException;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ExternalCatalog, we have getTable and loadTable. It's a little weird to see loadTable here which is actually the same as ExternalCatalog.getTable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the word "load" implies that, the table will be loaded to somewhere. Will we cache the table metadata somewhere inside Spark?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table is loaded so Spark can use it. This usage not unusual for the word "load".

The name getTable is no appropriate here because in the Java world, "get" signals that the operation is merely returning something that already exists, like an instance field. This is expected to make a remote call to fetch information from another store, so it is not an appropriate use of "get".

I think that the fact that this takes an identifier and returns a Table makes the operation clear. If you have a better verb for this (that isn't "get"), I'd consider it. I can't think of anything that works as well as loadTable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will we support the Hive LOAD TABLE in the future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan, LOAD TABLE is a SQL operation, not a catalog operation.

LOAD TABLE can be implemented as a write to some Table. If we wanted to support a path to load compatible data files directly into a table as an optimization, then we would need to build an API for that. Such an API would be part of a source API and not a catalog API.


/**
* Invalidate cached table metadata for an {@link Identifier identifier}.
* <p>
* If the table is already loaded or cached, drop cached data. If the table does not exist or is
* not cached, do nothing. Calling this method should not query remote services.
*
* @param ident a table identifier
*/
default void invalidateTable(Identifier ident) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the use case of this method? I can only think of one use case: when end users run REFRESH TABLE. Is there any more use cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be called by REFRESH TABLE to invalidate cached data. This was previously named refreshTable, but you preferred a void return type. In that case, invalidateTable describes what the method does better than refresh.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about invalidateTableCache? I think that would be more straightforward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a misleading name. This invalidates a single table's cached data, not the entire cache.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method accepts one parameter of Identifier type, which should not be misleading.
Just suggestion, overall I don't have a strong opinion on this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One quick question. This sounds like only to invalidate cached table metadata, and not touch query cache. But Catalog.refreshTable that is used v1 REFRESH TABLE command also drops InMemoryRelation if the table is cached in Spark query cache.

Is the difference intentional?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table cache is managed by Spark, so the API here can not affect table cache inside Spark.

Copy link
Member

@sunchao sunchao Oct 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan I think even though the table cache is managed by Spark, currently the behavior is different between v1 and v2 (which doesn't invalidate the table cache). When a cached temp view is created such as using CACHE TABLE ... AS SELECT from a v2 table, and later on REFRESH TABLE command is called on the v2 table, the cache which won't get invalidated. Is this something we should resolve?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should add this kind of missing actions into v2 commands as well. I think DROP TABLE has the same issue.

}

/**
* Test whether a table exists using an {@link Identifier identifier} from the catalog.
* <p>
* If the catalog supports views and contains a view for the identifier and not a table, this
* must return false.
*
* @param ident a table identifier
* @return true if the table exists, false otherwise
*/
default boolean tableExists(Identifier ident) {
try {
return loadTable(ident) != null;
} catch (NoSuchTableException e) {
return false;
}
}

/**
* Create a table in the catalog.
*
* @param ident a table identifier
* @param schema the schema of the new table, as a struct type
* @param partitions transforms to use for partitioning data in the table
* @param properties a string map of table properties
* @return metadata for the new table
* @throws TableAlreadyExistsException If a table or view already exists for the identifier
* @throws UnsupportedOperationException If a requested partition transform is not supported
* @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
*/
Table createTable(
Identifier ident,
StructType schema,
Transform[] partitions,
Map<String, String> properties) throws TableAlreadyExistsException, NoSuchNamespaceException;

/**
* Apply a set of {@link TableChange changes} to a table in the catalog.
* <p>
* Implementations may reject the requested changes. If any change is rejected, none of the
* changes should be applied to the table.
* <p>
* If the catalog supports views and contains a view for the identifier and not a table, this
* must throw {@link NoSuchTableException}.
*
* @param ident a table identifier
* @param changes changes to apply to the table
* @return updated metadata for the table
* @throws NoSuchTableException If the table doesn't exist or is a view
* @throws IllegalArgumentException If any change is rejected by the implementation.
*/
Table alterTable(
Identifier ident,
TableChange... changes) throws NoSuchTableException;

/**
* Drop a table in the catalog.
Copy link
Member

@gatorsmile gatorsmile May 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to clearly define the behavior when the input is a view or a temp view. If we do not expect users to specify view names here, we need to document it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the table metadata APIs are facing the same issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add a comment for how to handle views if they share the same namespace.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the javadoc for all of the methods to include how to handle views in the backing catalog.

* <p>
* If the catalog supports views and contains a view for the identifier and not a table, this
* must not drop the view and must return false.
*
* @param ident a table identifier
* @return true if a table was deleted, false if no table exists for the identifier
*/
boolean dropTable(Identifier ident);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @rdblue dropTable() here does not support to specify purge or not. Would you please share your idea about why it is designed like that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Purge is an option specific to some sources, like Hive. If you drop a table in most JDBC databases, you don't have an option to keep the data around somewhere. If an option can't be satisfied by sources, then it isn't a good candidate to be in the API.

Even in Hive, purge is optional because managed tables will drop data automatically, but external tables will not. Using table configuration for sources that support "external" data is a better option, I think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for now maybe we can fail the analysis if PURGE is specified but we are dropping a v2 table. @waterlx would you like to send a PR to do it?

}
Loading