[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL #1885

sarutak · 2014-08-11T06:13:49Z

No description provided.

…kSQL

SparkQA · 2014-08-11T06:19:44Z

QA tests have started for PR 1885. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18301/consoleFull

tianyi · 2014-08-11T06:34:30Z

i think the building detail has already added in https://github.com/apache/spark/blob/master/docs/sql-programming-guide.md

sarutak · 2014-08-11T07:05:58Z

Hi @tianyi .
I know that document but the document is outdated. We can not use -Phive option is no longer to use ThriftServer. In master branch, that moved into a hive-thriftserver project and we should use -Phive-thriftserver option.

In addition, the document is for programmer. People who build spark is not always programmer, but often system administrator so how to build is described in README as well as the description for using YARN.

SparkQA · 2014-08-11T07:10:25Z

QA results for PR 1885:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18301/consoleFull

tianyi · 2014-08-11T07:20:56Z

@sarutak the lastest sql-programming-guide.md had already included "-Phive-thriftserver" option

sarutak · 2014-08-11T07:32:46Z

Oh, master is updated.
But, as I mentioned, It's not friendly for builders.

liancheng · 2014-08-11T15:25:03Z

@sarutak Just checked the SQL programming guide in master, as @tianyi said, Thrift server and CLI related contents were both added. But we didn't mention sbt -Phive-thriftserver assembly is needed to enable Spark SQL CLI. Please go ahead if you'd like to complete that part, thanks! And what do you mean when you say "not friendly for builders"?

sarutak · 2014-08-11T15:32:21Z

@liancheng Thanks for your reply!
"Not friendly" means, the description about the place "-Phive-thrift-server is needed" is not good.
Currently, the description is mentioned in "Programmer's guide" but a person who build spark is not always a programmer. Sometime, a person who build may a system administrator.
I don't think, NON-programmer read Programmer's guide before main README.

Actually, how to build with YARN is mentioned in main README.

liancheng · 2014-08-11T15:54:06Z

Ah OK, fair enough. I agree with you.

liancheng · 2014-08-11T16:13:05Z

Then I guess we should add -Phive and -Phive-thriftserver related instructions to both our main README file and building-with-maven.md.

…SparkSQL to building-with-maven.md

sarutak · 2014-08-11T16:56:34Z

Ah, you're right.
Now I've modified building-with-maven.md. Thanks!

SparkQA · 2014-08-11T16:59:41Z

QA tests have started for PR 1885. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18311/consoleFull

liancheng · 2014-08-11T17:33:08Z

README.md

@@ -115,6 +115,15 @@ If your project is built with Maven, add this to your POM file's `<dependencies>
    </dependency>


+## A Note About HiveServer and CLI for SparkSQL


It should be "Hive Thrift server", which is compatible with HiveServer2, rather than HiveServer. And usually we use "Spark SQL" instead of "SparkSQL".

In sql-programming-guide.md, it's called "Thrift JDBC server".
Should I use this notation?
The latest PR uses "Thrift JDBC server".

liancheng · 2014-08-11T17:38:36Z

Since I'm not a native tongue either, we'd better wait for some native speaker to have a look. Otherwise LGTM, thanks!

SparkQA · 2014-08-11T17:50:27Z

QA results for PR 1885:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18311/consoleFull

SparkQA · 2014-08-11T18:09:41Z

QA tests have started for PR 1885. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18314/consoleFull

SparkQA · 2014-08-11T19:02:42Z

QA results for PR 1885:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18314/consoleFull

marmbrus · 2014-08-12T03:32:59Z

Thanks for adding this! One suggestion: instead of just adding documentation about the thrift server we should probably make these general sections about Spark SQL's Hive compatibility and cover both -Phive and -Phive-thrift-server. It doesn't look to me like that is covered in these documents, correct?

sarutak · 2014-08-12T03:51:58Z

Thanks @marmbrus !
The main issue I mention in this ticket is how to build to use CLI / Thrift JDBC server is not written on the proper place.

As you said, exactly documents illustrating Spark SQL's Hive compatibility is not present.
We should improve Spark SQL's document step by step.

First of all, we should add how to build to existing documents which illustrate about building for every user to use CLI / Thrift JDBC server. If they don't know how to build to use CLI / Thrift JDBC server, they cannot use those...

Next step, we should build Spark SQL's comprehensive document not only programmer's guide.

marmbrus · 2014-08-13T21:43:28Z

Thanks, merged to master and 1.1

…erver and CLI for SparkSQL Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #1885 from sarutak/SPARK-2963 and squashes the following commits: ed53329 [Kousuke Saruta] Modified description and notaton of proper noun 07c59fc [Kousuke Saruta] Added a description about how to build to use HiveServer and CLI for SparkSQL to building-with-maven.md 6e6645a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2963 c88fa93 [Kousuke Saruta] Added a description about building to use HiveServer and CLI for SparkSQL (cherry picked from commit 869f06c) Signed-off-by: Michael Armbrust <michael@databricks.com>

…g CLI and Thrift JDBC server is absent in proper document - The most important things I mentioned in #1885 is as follows. * People who build Spark is not always programmer. * If a person who build Spark is not a programmer, he/she won't read programmer's guide before building. So, how to build for using CLI and JDBC server is not only in programmer's guide. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #2080 from sarutak/SPARK-2963 and squashes the following commits: ee07c76 [Kousuke Saruta] Modified regression of the description about building for using Thrift JDBC server and CLI ed53329 [Kousuke Saruta] Modified description and notaton of proper noun 07c59fc [Kousuke Saruta] Added a description about how to build to use HiveServer and CLI for SparkSQL to building-with-maven.md 6e6645a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2963 c88fa93 [Kousuke Saruta] Added a description about building to use HiveServer and CLI for SparkSQL

…erver and CLI for SparkSQL Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes apache#1885 from sarutak/SPARK-2963 and squashes the following commits: ed53329 [Kousuke Saruta] Modified description and notaton of proper noun 07c59fc [Kousuke Saruta] Added a description about how to build to use HiveServer and CLI for SparkSQL to building-with-maven.md 6e6645a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2963 c88fa93 [Kousuke Saruta] Added a description about building to use HiveServer and CLI for SparkSQL

…g CLI and Thrift JDBC server is absent in proper document - The most important things I mentioned in apache#1885 is as follows. * People who build Spark is not always programmer. * If a person who build Spark is not a programmer, he/she won't read programmer's guide before building. So, how to build for using CLI and JDBC server is not only in programmer's guide. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes apache#2080 from sarutak/SPARK-2963 and squashes the following commits: ee07c76 [Kousuke Saruta] Modified regression of the description about building for using Thrift JDBC server and CLI ed53329 [Kousuke Saruta] Modified description and notaton of proper noun 07c59fc [Kousuke Saruta] Added a description about how to build to use HiveServer and CLI for SparkSQL to building-with-maven.md 6e6645a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2963 c88fa93 [Kousuke Saruta] Added a description about building to use HiveServer and CLI for SparkSQL

Added a description about building to use HiveServer and CLI for Spar…

c88fa93

…kSQL

sarutak changed the title ~~[SPARK-2963]~~ [SPARK-2963] There no documentation about building to use HiveServer and CLI for SparkSQL Aug 11, 2014

sarutak changed the title ~~[SPARK-2963] There no documentation about building to use HiveServer and CLI for SparkSQL~~ [SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL Aug 11, 2014

sarutak added 2 commits August 12, 2014 01:46

Merge branch 'master' of git://git.apache.org/spark into SPARK-2963

6e6645a

Added a description about how to build to use HiveServer and CLI for …

07c59fc

…SparkSQL to building-with-maven.md

liancheng reviewed Aug 11, 2014
View reviewed changes

Modified description and notaton of proper noun

ed53329

asfgit closed this in 869f06c Aug 13, 2014

sarutak mentioned this pull request Aug 21, 2014

[SPARK-2963] REGRESSION - The description about how to build for using CLI and Thrift JDBC server is absent in proper document - #2080

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL #1885

[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL #1885

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

tianyi commented Aug 11, 2014

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

tianyi commented Aug 11, 2014

sarutak commented Aug 11, 2014

liancheng commented Aug 11, 2014

sarutak commented Aug 11, 2014

liancheng commented Aug 11, 2014

liancheng commented Aug 11, 2014

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

liancheng Aug 11, 2014

sarutak Aug 11, 2014

liancheng commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

marmbrus commented Aug 12, 2014

sarutak commented Aug 12, 2014

marmbrus commented Aug 13, 2014

		@@ -115,6 +115,15 @@ If your project is built with Maven, add this to your POM file's `<dependencies>
		</dependency>


		## A Note About HiveServer and CLI for SparkSQL

[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL #1885

[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL #1885

Conversation

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

tianyi commented Aug 11, 2014

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

tianyi commented Aug 11, 2014

sarutak commented Aug 11, 2014

liancheng commented Aug 11, 2014

sarutak commented Aug 11, 2014

liancheng commented Aug 11, 2014

liancheng commented Aug 11, 2014

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

liancheng Aug 11, 2014

Choose a reason for hiding this comment

sarutak Aug 11, 2014

Choose a reason for hiding this comment

liancheng commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

marmbrus commented Aug 12, 2014

sarutak commented Aug 12, 2014

marmbrus commented Aug 13, 2014