Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-451] Update Sedona Scala examples for multiples Sedona release and check against different Spark versions #1153

Merged
merged 5 commits into from
Dec 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions .github/workflows/example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,22 @@ jobs:
build:

runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
include:
- spark: 3.4.2
spark-compat: '3.4'
sedona: 1.5.0
- spark: 3.3.4
spark-compat: '3.0'
sedona: 1.5.0
# - spark: 3.5.0
# spark-compat: '3.4'
# sedona: 1.5.0

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: actions/setup-java@v1
with:
java-version: '8'
Expand All @@ -42,7 +55,9 @@ jobs:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2
- run: (cd examples/spark-rdd-colocation-mining;sbt clean assembly;java -jar target/scala-2.12/*.jar)
- run: (cd examples/spark-sql;sbt clean assembly;java -jar target/scala-2.12/*.jar)
- run: (cd examples/spark-viz;sbt clean assembly;java -jar target/scala-2.12/*.jar)
- env:
SPARK_VERSION: ${{ matrix.spark }}
SPARK_COMPAT_VERSION: ${{ matrix.spark-compat }}
SEDONA_VERSION: ${{ matrix.sedona }}
run: (cd examples/spark-sql;mvn clean install -Dspark.version=${SPARK_VERSION} -Dspark.compat.version=${SPARK_COMPAT_VERSION} -Dsedona.version=${SEDONA_VERSION};java -jar target/sedona-spark-example-1.0.0.jar)
- run: (cd examples/flink-sql;mvn clean install;java -jar target/sedona-flink-example-1.0.0.jar)
12 changes: 6 additions & 6 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@
# Sedona Template Project

This repository contains template projects for RDD, SQL and Viz. The template projects have been configured properly.
This repository contains template projects for Sedona on Spark and Flink. The template projects have been configured properly.

Note that, although the template projects are written in Scala, the same APIs can be used in Java as well.

## Folder structure
The folder structure of this repository is as follows.

* rdd-colocation-mining: a scala template shows how to use Sedona RDD API in Spatial Data Mining
* sql: a scala template shows how to use Sedona DataFrame and SQL API
* viz: a scala template shows how to use Sedona Viz RDD and SQL API
* spark-sql: a Scala template shows how to use Sedona RDD, DataFrame and SQL API
* flink-sql: a Java template show how to use Sedona SQL via Flink Table APIs


## Compile and package

### Prerequisites
Please make sure you have the following software installed on your local machine:

* For Scala: Scala 2.12, SBT
* For Scala: Scala 2.12
* For Java: JDK 1.8, Apache Maven 3

### Compile

Run a terminal command `sbt assembly` within the folder of each template
Run a terminal command `mvn clean package` within the folder of each template


### Submit your fat jar to Spark
Expand Down
3 changes: 0 additions & 3 deletions examples/spark-rdd-colocation-mining/.gitignore

This file was deleted.

62 changes: 0 additions & 62 deletions examples/spark-rdd-colocation-mining/README.md

This file was deleted.

86 changes: 0 additions & 86 deletions examples/spark-rdd-colocation-mining/build.sbt

This file was deleted.

Binary file not shown.
2 changes: 0 additions & 2 deletions examples/spark-rdd-colocation-mining/project/.gitignore

This file was deleted.

This file was deleted.

1 change: 0 additions & 1 deletion examples/spark-rdd-colocation-mining/project/plugins.sbt

This file was deleted.

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
3 changes: 1 addition & 2 deletions examples/spark-sql/.gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
/target/
/.idea/
/spark-warehouse/
/target/
86 changes: 0 additions & 86 deletions examples/spark-sql/build.sbt

This file was deleted.

Binary file added examples/spark-sql/colocationMap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading