Skip to content

Commit

Permalink
[SPARK-46550][BUILD][SQL] Upgrade datasketches-java to 5.0.1
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This pr aims to upgarde `datasketches-java` from 3.3.0 to 5.0.1.
Additionally, this PR adds handling for `SketchesArgumentException` to ensure backward compatibility for exceptional test scenarios in `hll.sql`.

### Why are the changes needed?
The new brings some bug fix and improvements, like

- apache/datasketches-java#437
- apache/datasketches-java#482

The full release notes as follows:
- https://github.com/apache/datasketches-java/releases/tag/4.0.0
- https://github.com/apache/datasketches-java/releases/tag/4.1.0
- https://github.com/apache/datasketches-java/releases/tag/4.2.0
- https://github.com/apache/datasketches-java/releases/tag/5.0.0
- https://github.com/apache/datasketches-java/releases/tag/5.0.1

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#44547 from LuciferYang/datasketches-java-5.

Lead-authored-by: yangjie01 <yangjie01@baidu.com>
Co-authored-by: YangJie <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Jan 3, 2024
1 parent 605fecd commit 06f9e74
Show file tree
Hide file tree
Showing 4 changed files with 9 additions and 7 deletions.
4 changes: 2 additions & 2 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ curator-recipes/5.2.0//curator-recipes-5.2.0.jar
datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar
datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
datasketches-java/3.3.0//datasketches-java-3.3.0.jar
datasketches-memory/2.1.0//datasketches-memory-2.1.0.jar
datasketches-java/5.0.1//datasketches-java-5.0.1.jar
datasketches-memory/2.2.0//datasketches-memory-2.2.0.jar
derby/10.16.1.1//derby-10.16.1.1.jar
derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@
<commons-cli.version>1.6.0</commons-cli.version>
<bouncycastle.version>1.77</bouncycastle.version>
<tink.version>1.9.0</tink.version>
<datasketches.version>3.3.0</datasketches.version>
<datasketches.version>5.0.1</datasketches.version>
<netty.version>4.1.100.Final</netty.version>
<netty-tcnative.version>2.0.61.Final</netty-tcnative.version>
<!--
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

package org.apache.spark.sql.catalyst.expressions.aggregate

import org.apache.datasketches.common.SketchesArgumentException
import org.apache.datasketches.hll.{HllSketch, TgtHllType, Union}
import org.apache.datasketches.memory.Memory

Expand Down Expand Up @@ -317,7 +318,7 @@ case class HllUnionAgg(
union.update(sketch)
Some(union)
} catch {
case _: java.lang.Error =>
case _: SketchesArgumentException | _: java.lang.Error =>
throw QueryExecutionErrors.hllInvalidInputSketchBuffer(prettyName)
}
case _ =>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

package org.apache.spark.sql.catalyst.expressions

import org.apache.datasketches.common.SketchesArgumentException
import org.apache.datasketches.hll.{HllSketch, TgtHllType, Union}
import org.apache.datasketches.memory.Memory

Expand Down Expand Up @@ -55,7 +56,7 @@ case class HllSketchEstimate(child: Expression)
try {
Math.round(HllSketch.heapify(Memory.wrap(buffer)).getEstimate)
} catch {
case _: java.lang.Error =>
case _: SketchesArgumentException | _: java.lang.Error =>
throw QueryExecutionErrors.hllInvalidInputSketchBuffer(prettyName)
}
}
Expand Down Expand Up @@ -107,13 +108,13 @@ case class HllUnion(first: Expression, second: Expression, third: Expression)
val sketch1 = try {
HllSketch.heapify(Memory.wrap(value1.asInstanceOf[Array[Byte]]))
} catch {
case _: java.lang.Error =>
case _: SketchesArgumentException | _: java.lang.Error =>
throw QueryExecutionErrors.hllInvalidInputSketchBuffer(prettyName)
}
val sketch2 = try {
HllSketch.heapify(Memory.wrap(value2.asInstanceOf[Array[Byte]]))
} catch {
case _: java.lang.Error =>
case _: SketchesArgumentException | _: java.lang.Error =>
throw QueryExecutionErrors.hllInvalidInputSketchBuffer(prettyName)
}
val allowDifferentLgConfigK = value3.asInstanceOf[Boolean]
Expand Down

0 comments on commit 06f9e74

Please sign in to comment.