Skip to content

Commit

Permalink
Merge pull request #1657 from apache/master
Browse files Browse the repository at this point in the history
Create a new pull request by comparing changes across two branches
  • Loading branch information
GulajavaMinistudio authored Jun 13, 2024
2 parents f467e87 + bdcb79f commit ad19577
Show file tree
Hide file tree
Showing 35 changed files with 526 additions and 93 deletions.
24 changes: 24 additions & 0 deletions common/utils/src/main/resources/error/error-conditions.json
Original file line number Diff line number Diff line change
Expand Up @@ -3735,6 +3735,18 @@
],
"sqlState" : "42K06"
},
"STATE_STORE_KEY_ROW_FORMAT_VALIDATION_FAILURE" : {
"message" : [
"The streaming query failed to validate written state for key row.",
"The following reasons may cause this:",
"1. An old Spark version wrote the checkpoint that is incompatible with the current one",
"2. Corrupt checkpoint files",
"3. The query changed in an incompatible way between restarts",
"For the first case, use a new checkpoint directory or use the original Spark version",
"to process the streaming state. Retrieved error_message=<errorMsg>"
],
"sqlState" : "XX000"
},
"STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE" : {
"message" : [
"Provided key schema does not match existing state key schema.",
Expand Down Expand Up @@ -3769,6 +3781,18 @@
],
"sqlState" : "42802"
},
"STATE_STORE_VALUE_ROW_FORMAT_VALIDATION_FAILURE" : {
"message" : [
"The streaming query failed to validate written state for value row.",
"The following reasons may cause this:",
"1. An old Spark version wrote the checkpoint that is incompatible with the current one",
"2. Corrupt checkpoint files",
"3. The query changed in an incompatible way between restarts",
"For the first case, use a new checkpoint directory or use the original Spark version",
"to process the streaming state. Retrieved error_message=<errorMsg>"
],
"sqlState" : "XX000"
},
"STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE" : {
"message" : [
"Provided value schema does not match existing state value schema.",
Expand Down
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ scala-library/2.13.14//scala-library-2.13.14.jar
scala-parallel-collections_2.13/1.0.4//scala-parallel-collections_2.13-1.0.4.jar
scala-parser-combinators_2.13/2.4.0//scala-parser-combinators_2.13-2.4.0.jar
scala-reflect/2.13.14//scala-reflect-2.13.14.jar
scala-xml_2.13/2.2.0//scala-xml_2.13-2.2.0.jar
scala-xml_2.13/2.3.0//scala-xml_2.13-2.3.0.jar
slf4j-api/2.0.13//slf4j-api-2.0.13.jar
snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar
snakeyaml/2.2//snakeyaml-2.2.jar
Expand Down
2 changes: 2 additions & 0 deletions docs/_data/menu-sql.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@
url: sql-performance-tuning.html#optimizing-the-join-strategy
- text: Adaptive Query Execution
url: sql-performance-tuning.html#adaptive-query-execution
- text: Storage Partition Join
url: sql-performance-tuning.html#storage-partition-join
- text: Distributed SQL Engine
url: sql-distributed-sql-engine.html
subitems:
Expand Down
119 changes: 119 additions & 0 deletions docs/sql-performance-tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -428,3 +428,122 @@ You can control the details of how AQE works by providing your own cost evaluato
<td>3.2.0</td>
</tr>
</table>

## Storage Partition Join

Storage Partition Join (SPJ) is an optimization technique in Spark SQL that makes use the existing storage layout to avoid the shuffle phase.

This is a generalization of the concept of Bucket Joins, which is only applicable for [bucketed](sql-data-sources-load-save-functions.html#bucketing-sorting-and-partitioning) tables, to tables partitioned by functions registered in FunctionCatalog. Storage Partition Joins are currently supported for compatible V2 DataSources.

The following SQL properties enable Storage Partition Join in different join queries with various optimizations.

<table class="spark-config">
<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr></thead>
<tr>
<td><code>spark.sql.sources.v2.bucketing.enabled</code></td>
<td>false</td>
<td>
When true, try to eliminate shuffle by using the partitioning reported by a compatible V2 data source.
</td>
<td>3.3.0</td>
</tr>
<tr>
<td><code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code></td>
<td>true</td>
<td>
When enabled, try to eliminate shuffle if one side of the join has missing partition values from the other side. This config requires <code>spark.sql.sources.v2.bucketing.enabled</code> to be true.
</td>
<td>3.4.0</td>
</tr>
<tr>
<td><code>spark.sql.requireAllClusterKeysForCoPartition</code></td>
<td>true</td>
<td>
When true, require the join or MERGE keys to be same and in the same order as the partition keys to eliminate shuffle. Hence, set to <b>false</b> in this situation to eliminate shuffle.
</td>
<td>3.4.0</td>
</tr>
<tr>
<td><code>spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled</code></td>
<td>false</td>
<td>
When true, and when the join is not a full outer join, enable skew optimizations to handle partitions with large amounts of data when avoiding shuffle. One side will be chosen as the big table based on table statistics, and the splits on this side will be partially-clustered. The splits of the other side will be grouped and replicated to match. This config requires both <code>spark.sql.sources.v2.bucketing.enabled</code> and <code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code> to be true.
</td>
<td>3.4.0</td>
</tr>
<tr>
<td><code>spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled</code></td>
<td>false</td>
<td>
When enabled, try to avoid shuffle if join or MERGE condition does not include all partition columns. This config requires both <code>spark.sql.sources.v2.bucketing.enabled</code> and <code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code> to be true, and <code>spark.sql.requireAllClusterKeysForCoPartition</code> to be false.
</td>
<td>4.0.0</td>
</tr>
<tr>
<td><code>spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled</code></td>
<td>false</td>
<td>
When enabled, try to avoid shuffle if partition transforms are compatible but not identical. This config requires both <code>spark.sql.sources.v2.bucketing.enabled</code> and <code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code> to be true.
</td>
<td>4.0.0</td>
</tr>
<tr>
<td><code>spark.sql.sources.v2.bucketing.shuffle.enabled</code></td>
<td>false</td>
<td>
When enabled, try to avoid shuffle on one side of the join, by recognizing the partitioning reported by a V2 data source on the other side.
</td>
<td>4.0.0</td>
</tr>
</table>

If Storage Partition Join is performed, the query plan will not contain Exchange nodes prior to the join.

The following example uses Iceberg ([https://iceberg.apache.org/docs/latest/spark-getting-started/](https://iceberg.apache.org/docs/latest/spark-getting-started/)), a Spark V2 DataSource that supports Storage Partition Join.
```sql
CREATE TABLE prod.db.target (id INT, salary INT, dep STRING)
USING iceberg
PARTITIONED BY (dep, bucket(8, id))

CREATE TABLE prod.db.source (id INT, salary INT, dep STRING)
USING iceberg
PARTITIONED BY (dep, bucket(8, id))

EXPLAIN SELECT * FROM target t INNER JOIN source s
ON t.dep = s.dep AND t.id = s.id

-- Plan without Storage Partition Join
== Physical Plan ==
* Project (12)
+- * SortMergeJoin Inner (11)
:- * Sort (5)
: +- Exchange (4) // DATA SHUFFLE
: +- * Filter (3)
: +- * ColumnarToRow (2)
: +- BatchScan (1)
+- * Sort (10)
+- Exchange (9) // DATA SHUFFLE
+- * Filter (8)
+- * ColumnarToRow (7)
+- BatchScan (6)


SET 'spark.sql.sources.v2.bucketing.enabled' 'true'
SET 'spark.sql.iceberg.planning.preserve-data-grouping' 'true'
SET 'spark.sql.sources.v2.bucketing.pushPartValues.enabled' 'true'
SET 'spark.sql.requireAllClusterKeysForCoPartition' 'false'
SET 'spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled' 'true'

-- Plan with Storage Partition Join
== Physical Plan ==
* Project (10)
+- * SortMergeJoin Inner (9)
:- * Sort (4)
: +- * Filter (3)
: +- * ColumnarToRow (2)
: +- BatchScan (1)
+- * Sort (8)
+- * Filter (7)
+- * ColumnarToRow (6)
+- BatchScan (5)
```
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1120,7 +1120,7 @@
<dependency>
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-xml_${scala.binary.version}</artifactId>
<version>2.2.0</version>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
Expand Down
45 changes: 20 additions & 25 deletions python/pyspark/sql/connect/column.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,15 @@
Any,
Union,
Optional,
cast,
)

from pyspark.sql.column import Column as ParentColumn
from pyspark.errors import PySparkTypeError, PySparkAttributeError, PySparkValueError
from pyspark.sql.types import DataType

import pyspark.sql.connect.proto as proto
from pyspark.sql.connect.functions import builtin as F
from pyspark.sql.connect.expressions import (
Expression,
UnresolvedFunction,
Expand Down Expand Up @@ -308,14 +310,12 @@ def when(self, condition: ParentColumn, value: Any) -> ParentColumn:
message_parameters={},
)

if isinstance(value, Column):
_value = value._expr
else:
_value = LiteralExpression._from_value(value)

_branches = self._expr._branches + [(condition._expr, _value)]

return Column(CaseWhen(branches=_branches, else_value=None))
return Column(
CaseWhen(
branches=self._expr._branches + [(condition._expr, F.lit(value)._expr)],
else_value=None,
)
)

def otherwise(self, value: Any) -> ParentColumn:
if not isinstance(self._expr, CaseWhen):
Expand All @@ -328,12 +328,12 @@ def otherwise(self, value: Any) -> ParentColumn:
"otherwise() can only be applied once on a Column previously generated by when()"
)

if isinstance(value, Column):
_value = value._expr
else:
_value = LiteralExpression._from_value(value)

return Column(CaseWhen(branches=self._expr._branches, else_value=_value))
return Column(
CaseWhen(
branches=self._expr._branches,
else_value=cast(Expression, F.lit(value)._expr),
)
)

def like(self: ParentColumn, other: str) -> ParentColumn:
return _bin_op("like", self, other)
Expand Down Expand Up @@ -457,14 +457,11 @@ def isin(self, *cols: Any) -> ParentColumn:
else:
_cols = list(cols)

_exprs = [self._expr]
for c in _cols:
if isinstance(c, Column):
_exprs.append(c._expr)
else:
_exprs.append(LiteralExpression._from_value(c))

return Column(UnresolvedFunction("in", _exprs))
return Column(
UnresolvedFunction(
"in", [self._expr] + [cast(Expression, F.lit(c)._expr) for c in _cols]
)
)

def between(
self,
Expand Down Expand Up @@ -554,10 +551,8 @@ def __getitem__(self, k: Any) -> ParentColumn:
message_parameters={},
)
return self.substr(k.start, k.stop)
elif isinstance(k, Column):
return Column(UnresolvedExtractValue(self._expr, k._expr))
else:
return Column(UnresolvedExtractValue(self._expr, LiteralExpression._from_value(k)))
return Column(UnresolvedExtractValue(self._expr, cast(Expression, F.lit(k)._expr)))

def __iter__(self) -> None:
raise PySparkTypeError(
Expand Down
Loading

0 comments on commit ad19577

Please sign in to comment.