Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-24252][SQL] Add TableCatalog API #24246

Closed
wants to merge 14 commits into from
Closed
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@

package org.apache.spark.sql.catalog.v2;

import com.google.common.base.Preconditions;
import org.apache.spark.annotation.Experimental;

import java.util.Arrays;
import java.util.Objects;

/**
* An {@link Identifier} implementation.
*/
Expand All @@ -29,6 +33,8 @@ class IdentifierImpl implements Identifier {
private String name;

IdentifierImpl(String[] namespace, String name) {
Preconditions.checkNotNull(namespace, "Identifier namespace cannot be null");
Preconditions.checkNotNull(name, "Identifier name cannot be null");
this.namespace = namespace;
this.name = name;
}
Expand All @@ -42,4 +48,23 @@ public String[] namespace() {
public String name() {
return name;
}

@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}

if (o == null || getClass() != o.getClass()) {
return false;
}

IdentifierImpl that = (IdentifierImpl) o;
return Arrays.equals(namespace, that.namespace) && name.equals(that.name);
}

@Override
public int hashCode() {
return Objects.hash(Arrays.hashCode(namespace), name);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.sql.catalog.v2;

import org.apache.spark.sql.catalog.v2.expressions.Transform;
import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
import org.apache.spark.sql.sources.v2.Table;
import org.apache.spark.sql.types.StructType;

import java.util.Collections;
import java.util.Map;

/**
* Catalog methods for working with Tables.
*/
public interface TableCatalog extends CatalogPlugin {
Copy link
Member

@gatorsmile gatorsmile May 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still not sure whether we should have a dedicated ViewCatalog for view management, like what the proposal suggests. For the catalog API developers, metadata management of views and tables are very similar. Do they need to implement two catalogs? cc @rxin @marmbrus

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only similarity is that the two typically share the same namespace. The implementation is very different and will use different code paths. I think that it over-complicates the API to return the two using the same load method, when the alternative is to have two resolution rules instead.

That said, it would be easier to understand what you're asking for if you wrote up a proposed API. I can add this topic to the next v2 sync if you plan to propose an alternative.

/**
* List the tables in a namespace from the catalog.
*
* @param namespace a multi-part namespace
* @return an array of Identifiers for tables
* @throws NoSuchNamespaceException If the namespace does not exist (optional).
*/
Identifier[] listTables(String[] namespace) throws NoSuchNamespaceException;

/**
* Load table metadata by {@link Identifier identifier} from the catalog.
*
* @param ident a table identifier
* @return the table's metadata
* @throws NoSuchTableException If the table doesn't exist.
*/
Table loadTable(Identifier ident) throws NoSuchTableException;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ExternalCatalog, we have getTable and loadTable. It's a little weird to see loadTable here which is actually the same as ExternalCatalog.getTable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the word "load" implies that, the table will be loaded to somewhere. Will we cache the table metadata somewhere inside Spark?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table is loaded so Spark can use it. This usage not unusual for the word "load".

The name getTable is no appropriate here because in the Java world, "get" signals that the operation is merely returning something that already exists, like an instance field. This is expected to make a remote call to fetch information from another store, so it is not an appropriate use of "get".

I think that the fact that this takes an identifier and returns a Table makes the operation clear. If you have a better verb for this (that isn't "get"), I'd consider it. I can't think of anything that works as well as loadTable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will we support the Hive LOAD TABLE in the future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan, LOAD TABLE is a SQL operation, not a catalog operation.

LOAD TABLE can be implemented as a write to some Table. If we wanted to support a path to load compatible data files directly into a table as an optimization, then we would need to build an API for that. Such an API would be part of a source API and not a catalog API.


/**
* Refresh table metadata for {@link Identifier identifier}.
* <p>
* If the table is already loaded or cached, drop cached data and reload it. If the table is not
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is in the scope of this PR, but I was wondering if we could make the metadata/schema for a table un cacheable by default. For example, in the current V1 code if I reference a relation in the Hive catalog the session will cache the metadata associated with the table. This means that I can't update a user's view of the in catalog table without a manual user refresh.

I was hoping we can signal via the catalog api that a table's metadata is dynamic and should not be cached.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, we've implemented caching in our catalogs. Spark always calls load and uses the version that is returned instead of caching. Our catalog then uses a weak reference in the cache so that the version is periodically updated when no more queries are running against that version.

I'm not sure whether Spark should have some caching above the catalog or if the catalog should be responsible for this. Let's plan on talking about this at the next sync.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

* already loaded, load it and update caches with the new version.
*
* @param ident a table identifier
* @return the table's current metadata
* @throws NoSuchTableException If the table doesn't exist.
*/
default Table refreshTable(Identifier ident) throws NoSuchTableException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API design looks a bit weird, I think a more common approach is: refreshTable is just an action:

Table loadTable(Identifier ident)
default void refreshTable(Identifier ident) {}

When Spark want to get a fresh table, just call refreshTable first, then call loadTable. Is there any specific reason to not go with this design?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refresh table doesn't just require invalidating caches, it requires loading the table data at the time that refresh is called. Otherwise, it would be invalidateTable. If the refresh loads the table data, then there are good reasons to return it here. First, it ensures that the implementation does load the table because it must be returned. Second, it avoids a second call to load the table, which may cause a second load operation in catalogs that don't cache.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API behavior also seems a bit weird to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a precedent for how this is handled in other DBMS standards?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine renaming this to invalidateTable and not returning the metadata, if that's easier.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think this API refreshTable should not return a metadata.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you're saying that this should be invalidateTable and not refreshTable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's what invalidateTable would look like:

  /**
   * Invalidate cached table metadata for an {@link Identifier identifier}.
   * <p>
   * If the table is already loaded or cached, drop cached data. If the table does not exist or is
   * not cached, do nothing. Calling this method should not query remote services.
   *
   * @param ident a table identifier
   */
  default void invalidateTable(Identifier ident) {
  }

Would you prefer those requirements over the semantics of refreshTable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the DSv2 sync, no one had a strong opinion on this, but since since @cloud-fan, @gengliangwang, and @gatorsmile have all pointed this out, I went ahead and made this change. It is now invalidateTable instead of refreshTable.

Note that invalidateTable should not make a remote call and should only invalidate cached metadata, not refresh it.

return loadTable(ident);
}

/**
* Test whether a table exists using an {@link Identifier identifier} from the catalog.
*
* @param ident a table identifier
* @return true if the table exists, false otherwise
*/
default boolean tableExists(Identifier ident) {
try {
return loadTable(ident) != null;
} catch (NoSuchTableException e) {
return false;
}
}

/**
* Create a table in the catalog.
*
* @param ident a table identifier
* @param schema the schema of the new table, as a struct type
* @param partitions transforms to use for partitioning data in the table
* @param properties a string map of table properties
* @return metadata for the new table
* @throws TableAlreadyExistsException If a table already exists for the identifier
* @throws UnsupportedOperationException If a requested partition transform is not supported
* @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
*/
Table createTable(
Identifier ident,
StructType schema,
Transform[] partitions,
Map<String, String> properties) throws TableAlreadyExistsException, NoSuchNamespaceException;

/**
* Apply a set of {@link TableChange changes} to a table in the catalog.
* <p>
* Implementations may reject the requested changes. If any change is rejected, none of the
* changes should be applied to the table.
*
* @param ident a table identifier
* @param changes changes to apply to the table
* @return updated metadata for the table
* @throws NoSuchTableException If the table doesn't exist.
* @throws IllegalArgumentException If any change is rejected by the implementation.
*/
Table alterTable(
Identifier ident,
TableChange... changes) throws NoSuchTableException;

/**
* Drop a table in the catalog.
Copy link
Member

@gatorsmile gatorsmile May 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to clearly define the behavior when the input is a view or a temp view. If we do not expect users to specify view names here, we need to document it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the table metadata APIs are facing the same issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add a comment for how to handle views if they share the same namespace.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the javadoc for all of the methods to include how to handle views in the backing catalog.

*
* @param ident a table identifier
* @return true if a table was deleted, false if no table exists for the identifier
*/
boolean dropTable(Identifier ident);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @rdblue dropTable() here does not support to specify purge or not. Would you please share your idea about why it is designed like that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Purge is an option specific to some sources, like Hive. If you drop a table in most JDBC databases, you don't have an option to keep the data around somewhere. If an option can't be satisfied by sources, then it isn't a good candidate to be in the API.

Even in Hive, purge is optional because managed tables will drop data automatically, but external tables will not. Using table configuration for sources that support "external" data is a better option, I think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for now maybe we can fail the analysis if PURGE is specified but we are dropping a v2 table. @waterlx would you like to send a PR to do it?

}
Loading