Skip to content

Commit

Permalink
[CELEBORN-1801] Remove out-of-dated flink 1.14 and 1.15
Browse files Browse the repository at this point in the history
  • Loading branch information
codenohup committed Dec 24, 2024
1 parent 4b60dae commit 299a43b
Show file tree
Hide file tree
Showing 57 changed files with 25 additions and 5,798 deletions.
4 changes: 0 additions & 4 deletions .github/workflows/deps.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,6 @@ jobs:
- 'spark-3.3'
- 'spark-3.4'
- 'spark-3.5'
- 'flink-1.14'
- 'flink-1.15'
- 'flink-1.16'
- 'flink-1.17'
- 'flink-1.18'
Expand Down Expand Up @@ -82,8 +80,6 @@ jobs:
- 'spark-3.3'
- 'spark-3.4'
- 'spark-3.5'
- 'flink-1.14'
- 'flink-1.15'
- 'flink-1.16'
- 'flink-1.17'
- 'flink-1.18'
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/license.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,6 @@ jobs:
cache: 'maven'
check-latest: false
- run: |
build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.14
build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.15
build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.16
build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.17
build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.18
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/maven.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,6 @@ jobs:
- 8
- 11
flink:
- '1.14'
- '1.15'
- '1.16'
- '1.17'
- '1.18'
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/sbt.yml
Original file line number Diff line number Diff line change
Expand Up @@ -205,8 +205,6 @@ jobs:
- 8
- 11
flink:
- '1.14'
- '1.15'
- '1.16'
- '1.17'
- '1.18'
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/style.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,6 @@ jobs:
cache: 'maven'
check-latest: false
- run: |
build/mvn spotless:check -Pgoogle-mirror,flink-1.14
build/mvn spotless:check -Pgoogle-mirror,flink-1.15
build/mvn spotless:check -Pgoogle-mirror,flink-1.16
build/mvn spotless:check -Pgoogle-mirror,flink-1.17
build/mvn spotless:check -Pgoogle-mirror,flink-1.18
Expand Down
10 changes: 4 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,12 @@ Celeborn Worker's slot count is decided by `total usable disk size / average shu
Celeborn worker's slot count decreases when a partition is allocated and increments when a partition is freed.

## Build
1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4/3.5, Flink 1.14/1.15/1.16/1.17/1.18/1.19/1.20 and Hadoop MapReduce 2/3.
1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4/3.5, Flink 1.16/1.17/1.18/1.19/1.20 and Hadoop MapReduce 2/3.
2. Celeborn tested under Scala 2.11/2.12/2.13 and Java 8/11/17 environment.

Build Celeborn via `make-distribution.sh`:
```shell
./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pflink-1.14/-Pflink-1.15/-Pflink-1.16/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pmr
./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pflink-1.16/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pmr
```

Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
Expand All @@ -62,8 +62,6 @@ Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
| Spark 3.3 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Spark 3.4 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Spark 3.5 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Flink 1.14 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.15 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.16 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.17 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.18 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
Expand Down Expand Up @@ -114,7 +112,7 @@ Celeborn server is compatible with all clients inside various engines.
However, Celeborn clients must be consistent with the version of the specified engine.
For example, if you are running Spark 2.4, you must compile Celeborn client with -Pspark-2.4;
if you are running Spark 3.2, you must compile Celeborn client with -Pspark-3.2;
if you are running flink 1.14, you must compile Celeborn client with -Pflink-1.14.
if you are running flink 1.16, you must compile Celeborn client with -Pflink-1.16.

## Usage
Celeborn cluster composes of Master and Worker nodes, the Master supports both single and HA mode(Raft-based) deployments.
Expand Down Expand Up @@ -318,7 +316,7 @@ spark.executor.userClassPathFirst false
Copy `$CELEBORN_HOME/flink/*.jar` to `$FLINK_HOME/lib/`.

#### Flink Configuration
Celeborn supports two Flink integration strategies: remote shuffle service (since Flink 1.14) and [hybrid shuffle](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/batch/batch_shuffle/#hybrid-shuffle) (since Flink 1.20).
Celeborn supports two Flink integration strategies: remote shuffle service (since Flink 1.16) and [hybrid shuffle](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/batch/batch_shuffle/#hybrid-shuffle) (since Flink 1.20).

To use Celeborn, you can choose one of them and add the following Flink configurations.

Expand Down
4 changes: 0 additions & 4 deletions build/make-distribution.sh
Original file line number Diff line number Diff line change
Expand Up @@ -324,8 +324,6 @@ if [ "$SBT_ENABLED" == "true" ]; then
sbt_build_client -Pspark-2.4
sbt_build_client -Pspark-3.4
sbt_build_client -Pspark-3.5
sbt_build_client -Pflink-1.14
sbt_build_client -Pflink-1.15
sbt_build_client -Pflink-1.16
sbt_build_client -Pflink-1.17
sbt_build_client -Pflink-1.18
Expand Down Expand Up @@ -359,8 +357,6 @@ else
build_spark_client -Pspark-2.4
build_spark_client -Pspark-3.4
build_spark_client -Pspark-3.5
build_flink_client -Pflink-1.14
build_flink_client -Pflink-1.15
build_flink_client -Pflink-1.16
build_flink_client -Pflink-1.17
build_flink_client -Pflink-1.18
Expand Down
6 changes: 0 additions & 6 deletions build/release/release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,6 @@ upload_nexus_staging() {
echo "Deploying celeborn-client-spark-3-shaded_2.13"
${PROJECT_DIR}/build/sbt -Pspark-3.4 ++2.13 "clean;celeborn-client-spark-3-shaded/publishSigned"

echo "Deploying celeborn-client-flink-1.14-shaded_2.12"
${PROJECT_DIR}/build/sbt -Pflink-1.14 "clean;celeborn-client-flink-1_14-shaded/publishSigned"

echo "Deploying celeborn-client-flink-1.15-shaded_2.12"
${PROJECT_DIR}/build/sbt -Pflink-1.15 "clean;celeborn-client-flink-1_15-shaded/publishSigned"

echo "Deploying celeborn-client-flink-1.16-shaded_2.12"
${PROJECT_DIR}/build/sbt -Pflink-1.16 "clean;celeborn-client-flink-1_16-shaded/publishSigned"

Expand Down
137 changes: 0 additions & 137 deletions client-flink/flink-1.14-shaded/pom.xml

This file was deleted.

Loading

0 comments on commit 299a43b

Please sign in to comment.