From 4ec8e0129e18d09e4437032054fdc85075211796 Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Thu, 14 Mar 2024 13:26:09 -0700 Subject: [PATCH 1/2] doc: Update README.md with shuffle configs --- README.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 572a9d240..3f46d4556 100644 --- a/README.md +++ b/README.md @@ -113,4 +113,25 @@ INFO src/lib.rs: Comet native library initialized +- CometScan parquet [a#14] Batched: true, DataFilters: [isnotnull(a#14), (a#14 > 5)], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test], PartitionFilters: [], PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema: struct -``` \ No newline at end of file +``` + +### Enable Comet shuffle + +Comet shuffle feature is disabled by default. To enable it, please add related configs: + +``` +--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager +--conf spark.comet.exec.shuffle.enabled=true +``` + +Above configs enable Comet native shuffle which only supports hash partiting and single partition. +Comet native shuffle doesn't support complext types yet. + +To enable columnar shuffle which supports all partitioning and basic complex types, one more config is required: +``` +--conf spark.comet.columnar.shuffle.enabled=true +``` + + + + From 3b75e9cc6beb14282f3a1dd6f82ee4eeb1dc9c60 Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Thu, 14 Mar 2024 13:28:01 -0700 Subject: [PATCH 2/2] Update README.md --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 3f46d4556..3b903b14c 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,3 @@ To enable columnar shuffle which supports all partitioning and basic complex typ ``` --conf spark.comet.columnar.shuffle.enabled=true ``` - - - -