From c4099ee8fe54e76a53b36eedb65348d508bc748e Mon Sep 17 00:00:00 2001 From: Jacek Laskowski Date: Tue, 6 May 2014 23:58:15 +0200 Subject: [PATCH 1/3] sbt assembly and environment variables --- docs/index.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/index.md b/docs/index.md index 2daa208b3b903..08db9b902c154 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,7 +15,7 @@ Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). All you n # Building -Spark uses [Simple Build Tool](http://www.scala-sbt.org), which is bundled with it. To compile the code, go into the top-level Spark directory and run +Spark uses [sbt](http://www.scala-sbt.org), which is bundled with it. To compile the code, go into the top-level Spark directory and run sbt/sbt assembly @@ -58,14 +58,23 @@ Hadoop, you must build Spark against the same version that your cluster uses. By default, Spark links to Hadoop 1.0.4. You can change this by setting the `SPARK_HADOOP_VERSION` variable when compiling: - SPARK_HADOOP_VERSION=2.2.0 sbt/sbt assembly + SPARK_HADOOP_VERSION=2.4.0 sbt/sbt assembly In addition, if you wish to run Spark on [YARN](running-on-yarn.html), set `SPARK_YARN` to `true`: - SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly + SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true sbt/sbt assembly -Note that on Windows, you need to set the environment variables on separate lines, e.g., `set SPARK_HADOOP_VERSION=1.2.1`. +You may also want to set `SPARK_HIVE` to `true` to build Spark Hive module. + + SPARK_HIVE=true sbt/sbt assembly + +Mix the environment variables - `SPARK_HADOOP_VERSION`, `SPARK_YARN`, and `SPARK_HIVE` to match +your needs. + +Note that on Windows, you need to set the environment variables on separate lines, e.g. + + set SPARK_HADOOP_VERSION=2.4.0 # Where to Go from Here From 4aa7a447d70ad9257b2ac279f83984c61b28b646 Mon Sep 17 00:00:00 2001 From: Jacek Laskowski Date: Wed, 7 May 2014 00:01:18 +0200 Subject: [PATCH 2/3] A (missing) dash --- docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 08db9b902c154..7afc5a0876b6a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -69,7 +69,7 @@ You may also want to set `SPARK_HIVE` to `true` to build Spark Hive module. SPARK_HIVE=true sbt/sbt assembly -Mix the environment variables - `SPARK_HADOOP_VERSION`, `SPARK_YARN`, and `SPARK_HIVE` to match +Mix the environment variables - `SPARK_HADOOP_VERSION`, `SPARK_YARN`, and `SPARK_HIVE` - to match your needs. Note that on Windows, you need to set the environment variables on separate lines, e.g. From a137e6b0562693a8c59757d235937b3778b12a91 Mon Sep 17 00:00:00 2001 From: Jacek Laskowski Date: Wed, 7 May 2014 00:02:15 +0200 Subject: [PATCH 3/3] Expand on needs --- docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 7afc5a0876b6a..5ea53b86aed76 100644 --- a/docs/index.md +++ b/docs/index.md @@ -70,7 +70,7 @@ You may also want to set `SPARK_HIVE` to `true` to build Spark Hive module. SPARK_HIVE=true sbt/sbt assembly Mix the environment variables - `SPARK_HADOOP_VERSION`, `SPARK_YARN`, and `SPARK_HIVE` - to match -your needs. +your (assembly) needs. Note that on Windows, you need to set the environment variables on separate lines, e.g.