Skip to content

Commit

Permalink
doc: Update README.md with shuffle configs (#208)
Browse files Browse the repository at this point in the history
* doc: Update README.md with shuffle configs

* Update README.md
  • Loading branch information
viirya authored Mar 14, 2024
1 parent 969f683 commit 2245c86
Showing 1 changed file with 18 additions and 1 deletion.
19 changes: 18 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,4 +113,21 @@ INFO src/lib.rs: Comet native library initialized
+- CometScan parquet [a#14] Batched: true, DataFilters: [isnotnull(a#14), (a#14 > 5)],
Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test], PartitionFilters: [],
PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema: struct<a:int>
```
```

### Enable Comet shuffle

Comet shuffle feature is disabled by default. To enable it, please add related configs:

```
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
--conf spark.comet.exec.shuffle.enabled=true
```

Above configs enable Comet native shuffle which only supports hash partiting and single partition.
Comet native shuffle doesn't support complext types yet.

To enable columnar shuffle which supports all partitioning and basic complex types, one more config is required:
```
--conf spark.comet.columnar.shuffle.enabled=true
```

0 comments on commit 2245c86

Please sign in to comment.