-
Notifications
You must be signed in to change notification settings - Fork 28.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-25530][SQL] data source v2 API refactor (batch write)
## What changes were proposed in this pull request? Adjust the batch write API to match the read API refactor after #23086 The doc with high-level ideas: https://docs.google.com/document/d/1vI26UEuDpVuOjWw4WPoH2T6y8WAekwtI7qoowhOFnI4/edit?usp=sharing Basically it renames `BatchWriteSupportProvider` to `SupportsBatchWrite`, and make it extend `Table`. Renames `WriteSupport` to `Write`. It also cleans up some code as batch API is completed. This PR also removes the test from #22688 . Now data source must return a table for read/write. A few notes about future changes: 1. We will create `SupportsStreamingWrite` later for streaming APIs 2. We will create `SupportsBatchReplaceWhere`, `SupportsBatchAppend`, etc. for the new end-user write APIs. I think streaming APIs would remain to use `OutputMode`, and new end-user write APIs will apply to batch only, at least in the near future. 3. We will remove `SaveMode` from data source API: https://issues.apache.org/jira/browse/SPARK-26356 ## How was this patch tested? existing tests Closes #23208 from cloud-fan/refactor-batch. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
- Loading branch information
1 parent
1b75f3b
commit 954ef96
Showing
21 changed files
with
330 additions
and
245 deletions.
There are no files selected for viewing
59 changes: 0 additions & 59 deletions
59
sql/core/src/main/java/org/apache/spark/sql/sources/v2/BatchWriteSupportProvider.java
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
sql/core/src/main/java/org/apache/spark/sql/sources/v2/SupportsBatchWrite.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.spark.sql.sources.v2; | ||
|
||
import org.apache.spark.annotation.Evolving; | ||
import org.apache.spark.sql.sources.v2.writer.WriteBuilder; | ||
|
||
/** | ||
* An empty mix-in interface for {@link Table}, to indicate this table supports batch write. | ||
* <p> | ||
* If a {@link Table} implements this interface, the | ||
* {@link SupportsWrite#newWriteBuilder(DataSourceOptions)} must return a {@link WriteBuilder} | ||
* with {@link WriteBuilder#buildForBatch()} implemented. | ||
* </p> | ||
*/ | ||
@Evolving | ||
public interface SupportsBatchWrite extends SupportsWrite {} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
35 changes: 35 additions & 0 deletions
35
sql/core/src/main/java/org/apache/spark/sql/sources/v2/SupportsWrite.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.spark.sql.sources.v2; | ||
|
||
import org.apache.spark.sql.sources.v2.writer.BatchWrite; | ||
import org.apache.spark.sql.sources.v2.writer.WriteBuilder; | ||
|
||
/** | ||
* An internal base interface of mix-in interfaces for writable {@link Table}. This adds | ||
* {@link #newWriteBuilder(DataSourceOptions)} that is used to create a write | ||
* for batch or streaming. | ||
*/ | ||
interface SupportsWrite extends Table { | ||
|
||
/** | ||
* Returns a {@link WriteBuilder} which can be used to create {@link BatchWrite}. Spark will call | ||
* this method to configure each data source write. | ||
*/ | ||
WriteBuilder newWriteBuilder(DataSourceOptions options); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
26 changes: 26 additions & 0 deletions
26
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/SupportsSaveMode.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.spark.sql.sources.v2.writer; | ||
|
||
import org.apache.spark.sql.SaveMode; | ||
|
||
// A temporary mixin trait for `WriteBuilder` to support `SaveMode`. Will be removed before | ||
// Spark 3.0 when all the new write operators are finished. See SPARK-26356 for more details. | ||
public interface SupportsSaveMode extends WriteBuilder { | ||
WriteBuilder mode(SaveMode mode); | ||
} |
69 changes: 69 additions & 0 deletions
69
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/WriteBuilder.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to You under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.spark.sql.sources.v2.writer; | ||
|
||
import org.apache.spark.annotation.Evolving; | ||
import org.apache.spark.sql.sources.v2.SupportsBatchWrite; | ||
import org.apache.spark.sql.sources.v2.Table; | ||
import org.apache.spark.sql.types.StructType; | ||
|
||
/** | ||
* An interface for building the {@link BatchWrite}. Implementations can mix in some interfaces to | ||
* support different ways to write data to data sources. | ||
* | ||
* Unless modified by a mixin interface, the {@link BatchWrite} configured by this builder is to | ||
* append data without affecting existing data. | ||
*/ | ||
@Evolving | ||
public interface WriteBuilder { | ||
|
||
/** | ||
* Passes the `queryId` from Spark to data source. `queryId` is a unique string of the query. It's | ||
* possible that there are many queries running at the same time, or a query is restarted and | ||
* resumed. {@link BatchWrite} can use this id to identify the query. | ||
* | ||
* @return a new builder with the `queryId`. By default it returns `this`, which means the given | ||
* `queryId` is ignored. Please override this method to take the `queryId`. | ||
*/ | ||
default WriteBuilder withQueryId(String queryId) { | ||
return this; | ||
} | ||
|
||
/** | ||
* Passes the schema of the input data from Spark to data source. | ||
* | ||
* @return a new builder with the `schema`. By default it returns `this`, which means the given | ||
* `schema` is ignored. Please override this method to take the `schema`. | ||
*/ | ||
default WriteBuilder withInputDataSchema(StructType schema) { | ||
return this; | ||
} | ||
|
||
/** | ||
* Returns a {@link BatchWrite} to write data to batch source. By default this method throws | ||
* exception, data sources must overwrite this method to provide an implementation, if the | ||
* {@link Table} that creates this scan implements {@link SupportsBatchWrite}. | ||
* | ||
* Note that, the returned {@link BatchWrite} can be null if the implementation supports SaveMode, | ||
* to indicate that no writing is needed. We can clean it up after removing | ||
* {@link SupportsSaveMode}. | ||
*/ | ||
default BatchWrite buildForBatch() { | ||
throw new UnsupportedOperationException("Batch scans are not supported"); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.