Skip to content

how to run spark crail locally

Animesh Trivedi edited this page Aug 30, 2017 · 1 revision

Step 1: Get crail up and running on netty

Step 2: Setting up crail into Hadoop environment

Step 3: Running Spark with crail in the local mode

Step 4: Shuffling on crail

./bin/spark-submit -v --num-executors 1 --executor-cores 2 --executor-memory 1G --driver-memory 1G --master local --class com.ibm.crail.spark.tools.ParquetGenerator parquet-generator-1.0.jar -c tpcds -o crail://localhost:9060/tpcds
Clone this wiki locally