Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump aws-hudi to 1.0.0-beta2 #71

Merged
merged 1 commit into from
Jul 22, 2024
Merged

Bump aws-hudi to 1.0.0-beta2 #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions ...re/src/test/scala/com.snowplowanalytics.snowplow.lakes/processing/AbstractSparkSpec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,7 @@ abstract class AbstractSparkSpec extends Specification with CatsEffect {

override val Timeout = 60.seconds

// TODO: After Hudi 1.0.0 is released, remove `skipAll` to re-enable these tests

def is = skipAll ^ sequential ^ s2"""
def is = sequential ^ s2"""
The lake loader should:
Write a single window of events into a lake table $e1
Create unstruct_* column for unstructured events with valid schemas $e2
Expand Down
3 changes: 3 additions & 0 deletions packaging/hudi/src/test/scala/com.snowplowanalytics.snowplow.lakes.processing/HudiSpec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ import fs2.io.file.Path

class HudiSpec extends AbstractSparkSpec {

// TODO: After Hudi 1.0.0 is released, remove `skipAll` to re-enable these tests
override def is = skipAll ^ super.is

override def target: TestConfig.Target = TestConfig.Hudi

/** Reads the table back into memory, so we can make assertions on the app's output */
Expand Down
4 changes: 3 additions & 1 deletion project/Dependencies.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ object Dependencies {
// Spark
val delta = "3.2.0"
val hudi = "0.15.0"
val hudiAws = "1.0.0-beta2"
val iceberg = "1.5.2"
val hadoop = "3.4.0"
val gcsConnector = "hadoop3-2.2.17"
Expand Down Expand Up @@ -88,14 +89,15 @@ object Dependencies {
// spark and hadoop
val delta = "io.delta" %% "delta-spark" % V.delta
val hudi = "org.apache.hudi" %% s"hudi-spark${V.Spark.forHudiMinor}-bundle" % V.hudi
val hudiAws = "org.apache.hudi" % "hudi-aws" % V.hudi
val iceberg = "org.apache.iceberg" %% s"iceberg-spark-runtime-${V.Spark.forIcebergDeltaMinor}" % V.iceberg
val hadoopClient = "org.apache.hadoop" % "hadoop-client-runtime" % V.hadoop
val hadoopAzure = "org.apache.hadoop" % "hadoop-azure" % V.hadoop
val hadoopAws = "org.apache.hadoop" % "hadoop-aws" % V.hadoop
val gcsConnector = "com.google.cloud.bigdataoss" % "gcs-connector" % V.gcsConnector
val hiveCommon = "org.apache.hive" % "hive-common" % V.hive

val hudiAws = ("org.apache.hudi" % "hudi-aws" % V.hudiAws).excludeAll(ExclusionRule(organization = "org.apache.hudi"))

// java
val slf4j = "org.slf4j" % "slf4j-simple" % V.slf4j
val azureIdentity = "com.azure" % "azure-identity" % V.azureSdk
Expand Down
Loading