Skip to content

Commit

Permalink
SPARK-1121: Include avro for yarn-alpha builds
Browse files Browse the repository at this point in the history 
This lets us explicitly include Avro based on a profile for 0.23.X
builds. It makes me sad how convoluted it is to express this logic
in Maven. @tgraves and @sryza curious if this works for you.

I'm also considering just reverting to how it was before. The only
real problem was that Spark advertised a dependency on Avro
even though it only really depends transitively on Avro through
other deps.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #49 from pwendell/avro-build-fix and squashes the following commits:

8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
  • Loading branch information
@pwendell
pwendell committed Mar 2, 2014
1 parent fd31adb commit c3f5e07
Show file tree
Hide file tree
Showing 18 changed files with 234 additions and 12 deletions.
14 changes: 14 additions & 0 deletions bagel/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project Bagel</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project Core</name>
<url>http://spark.apache.org/</url>

<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<profiles>
<profile>
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
Expand Down
4 changes: 0 additions & 4 deletions docs/building-with-maven.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,3 @@ The maven build includes support for building a Debian package containing the as
$ mvn -Pdeb -DskipTests clean package

The debian package can then be found under assembly/target. We added the short commit hash to the file name so that we can distinguish individual packages built for SNAPSHOT versions.

## A note about Hadoop version 0.23.x

For building spark with hadoop 0.23.x and also yarn, you will have to manually add a dependency on avro (org.apache.avro, avro, 1.7.4).
14 changes: 14 additions & 0 deletions examples/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project Examples</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<repositories>
<repository>
<id>apache-repo</id>
Expand Down
14 changes: 14 additions & 0 deletions external/flume/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project External Flume</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions external/kafka/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project External Kafka</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions external/mqtt/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project External MQTT</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<repositories>
<repository>
<id>mqtt-repo</id>
Expand Down
14 changes: 14 additions & 0 deletions external/twitter/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project External Twitter</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions external/zeromq/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project External ZeroMQ</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions graphx/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project GraphX</name>
<url>http://spark-project.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions mllib/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project ML Library</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
15 changes: 15 additions & 0 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,21 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.7.4</version>
<exclusions>
<exclusion>
<groupId>org.jboss.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId>
Expand Down
14 changes: 14 additions & 0 deletions repl/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project REPL</name>
<url>http://spark.apache.org/</url>

<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<profiles>
<profile>
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<properties>
<deb.install.path>/usr/share/spark</deb.install.path>
<deb.user>root</deb.user>
Expand Down
14 changes: 14 additions & 0 deletions streaming/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@
<name>Spark Project Streaming</name>
<url>http://spark.apache.org/</url>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<repositories>
<repository>
<id>apache-repo</id>
Expand Down
14 changes: 14 additions & 0 deletions tools/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@
<name>Spark Project Tools</name>
<url>http://spark.apache.org/</url>

<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<profiles>
<profile>
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
14 changes: 14 additions & 0 deletions yarn/alpha/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,20 @@
<relativePath>../pom.xml</relativePath>
</parent>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn-alpha_2.10</artifactId>
<packaging>jar</packaging>
Expand Down
17 changes: 9 additions & 8 deletions yarn/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,6 @@
<artifactId>hadoop-client</artifactId>
<version>${yarn.version}</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-ipc</artifactId>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId>
Expand All @@ -78,6 +70,15 @@
<modules>
<module>alpha</module>
</modules>

<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>

<profile>
Expand Down
14 changes: 14 additions & 0 deletions yarn/stable/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,20 @@
<relativePath>../pom.xml</relativePath>
</parent>

<profiles>
<profile>
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
a Hadoop 0.23.X issue -->
<id>yarn-alpha</id>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>

<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_2.10</artifactId>
<packaging>jar</packaging>
Expand Down

0 comments on commit c3f5e07

Please sign in to comment.