From 4fe863c3ad9e4f08075264419c886e4cab15ce5d Mon Sep 17 00:00:00 2001 From: Weiqing Yang Date: Thu, 18 May 2017 13:18:34 -0700 Subject: [PATCH] Preparing SHC release v1.1.1-1.6 --- README.md | 16 ++++++++-------- core/pom.xml | 4 ++-- examples/pom.xml | 4 ++-- pom.xml | 2 +- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 6490d49b..ec8fcfd3 100644 --- a/README.md +++ b/README.md @@ -42,15 +42,15 @@ Run indiviudal test mvn -DwildcardSuites=org.apache.spark.sql.DefaultSourceSuite test Run SHC examples - ./bin/spark-submit --verbose --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-cluster --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --files /usr/hdp/current/hbase-client/conf/hbase-site.xml shc-examples-1.0.1-1.6-s_2.10-SNAPSHOT.jar + ./bin/spark-submit --verbose --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-cluster --packages com.hortonworks:shc-core:1.1.1-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --files /usr/hdp/current/hbase-client/conf/hbase-site.xml shc-examples-1.1.1-1.6-s_2.10-SNAPSHOT.jar The following illustrates how to run your application in real hbase cluster. You need to provide the hbase-site.xml. It may subject to change based on your specific cluster configuration. - ./bin/spark-submit --class your.application.class --master yarn-client --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 1 --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar + ./bin/spark-submit --class your.application.class --master yarn-client --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 1 --packages com.hortonworks:shc-core:1.1.1-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar **Note**: Running Spark applications with this connector, HBase jars of version 1.1.2 will be pulled by default. If Phoenix is enabled on HBase cluster, you need to use "--jars" to pass "phoenix-server.jar". For example: - ./bin/spark-submit --class your.application.class --master yarn-client --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 1 --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --jars /usr/hdp/current/phoenix-client/phoenix-server.jar --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar + ./bin/spark-submit --class your.application.class --master yarn-client --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 1 --packages com.hortonworks:shc-core:1.1.1-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --jars /usr/hdp/current/phoenix-client/phoenix-server.jar --files /etc/hbase/conf/hbase-site.xml /To/your/application/jar ## Application Usage The following illustrates the basic procedure on how to use the connector. For more details and advanced use case, such as Avro and composite key support, please refer to the examples in the repository. @@ -118,15 +118,15 @@ Given a data frame with specified schema, above will create an HBase table with ## Configuring Spark-package Users can use the Spark-on-HBase connector as a standard Spark package. To include the package in your Spark application use: -_**Note**: com.hortonworks:shc:1.0.0-1.6-s_2.10 has not been uploaded to [spark-packages.org](https://spark-packages.org/package/hortonworks-spark/shc), but will be there soon._ +_**Note**: com.hortonworks:shc-core:1.1.1-1.6-s_2.10 has not been uploaded to [spark-packages.org](https://spark-packages.org/package/hortonworks-spark/shc), but will be there soon._ spark-shell, pyspark, or spark-submit - $SPARK_HOME/bin/spark-shell --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 + $SPARK_HOME/bin/spark-shell --packages com.hortonworks:shc-core:1.1.1-1.6-s_2.10 Users can include the package as the dependency in your SBT file as well. The format is the spark-package-name:version - spDependencies += “com.hortonworks/shc:1.0.0-1.6-s_2.10” + spDependencies += “com.hortonworks/shc-core:1.1.1-1.6-s_2.10” ## Running in secure cluster @@ -140,9 +140,9 @@ Suppose hrt_qa is a headless account, user can use following command for kinit: kinit -k -t /tmp/hrt_qa.headless.keytab hrt_qa - /usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-client --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /To/your/application/jar + /usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-client --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc-core:1.1.1-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /To/your/application/jar - /usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-cluster --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /To/your/application/jar + /usr/hdp/current/spark-client/bin/spark-submit --class your.application.class --master yarn-cluster --files /etc/hbase/conf/hbase-site.xml --packages com.hortonworks:shc-core:1.1.1-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/ --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /To/your/application/jar If the solution above does not work and you encounter errors like : diff --git a/core/pom.xml b/core/pom.xml index 71b214f2..b0f03585 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -24,12 +24,12 @@ com.hortonworks shc - 1.1.1-1.6-s_2.10-SNAPSHOT + 1.1.1-1.6-s_2.10 ../pom.xml shc-core - 1.1.1-1.6-s_2.10-SNAPSHOT + 1.1.1-1.6-s_2.10 jar HBase Spark Connector Project Core diff --git a/examples/pom.xml b/examples/pom.xml index 66f68e5c..9f0ca184 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -24,12 +24,12 @@ com.hortonworks shc - 1.1.1-1.6-s_2.10-SNAPSHOT + 1.1.1-1.6-s_2.10 ../pom.xml shc-examples - 1.1.1-1.6-s_2.10-SNAPSHOT + 1.1.1-1.6-s_2.10 jar HBase Spark Connector Project Examples diff --git a/pom.xml b/pom.xml index 9bb4914d..868eec46 100644 --- a/pom.xml +++ b/pom.xml @@ -28,7 +28,7 @@ com.hortonworks shc - 1.1.1-1.6-s_2.10-SNAPSHOT + 1.1.1-1.6-s_2.10 pom HBase Spark Connector Project Parent POM https://github.com/hortonworks-spark/shc