Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 #30478

Closed
wants to merge 3 commits into from
Closed

[SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 #30478

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
17deafb
Update hive-service-rpc to 3.1.2
wangyum Nov 24, 2020
98e80e8
Fix Unrecognized GetInfoType value: CLI_ODBC_KEYWORDS
wangyum Nov 24, 2020
43d90ca
Update manifest
wangyum Nov 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-2.7-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ hive-jdbc/2.3.7//hive-jdbc-2.3.7.jar
hive-llap-common/2.3.7//hive-llap-common-2.3.7.jar
hive-metastore/2.3.7//hive-metastore-2.3.7.jar
hive-serde/2.3.7//hive-serde-2.3.7.jar
hive-service-rpc/2.3.7//hive-service-rpc-2.3.7.jar
hive-service-rpc/3.1.2//hive-service-rpc-3.1.2.jar
hive-shims-0.23/2.3.7//hive-shims-0.23-2.3.7.jar
hive-shims-common/2.3.7//hive-shims-common-2.3.7.jar
hive-shims-scheduler/2.3.7//hive-shims-scheduler-2.3.7.jar
Expand Down
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3.2-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ hive-jdbc/2.3.7//hive-jdbc-2.3.7.jar
hive-llap-common/2.3.7//hive-llap-common-2.3.7.jar
hive-metastore/2.3.7//hive-metastore-2.3.7.jar
hive-serde/2.3.7//hive-serde-2.3.7.jar
hive-service-rpc/2.3.7//hive-service-rpc-2.3.7.jar
hive-service-rpc/3.1.2//hive-service-rpc-3.1.2.jar
hive-shims-0.23/2.3.7//hive-shims-0.23-2.3.7.jar
hive-shims-common/2.3.7//hive-shims-common-2.3.7.jar
hive-shims-scheduler/2.3.7//hive-shims-scheduler-2.3.7.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2088,7 +2088,7 @@
<dependency>
<groupId>${hive.group}</groupId>
<artifactId>hive-service-rpc</artifactId>
<version>${hive.version}</version>
<version>3.1.2</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we have a JIRA to recover this when we upgrade Hive version to 3.1.2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wangyum . Did you create a JIRA? I cannot find it yet. If you created, please link to SPARK-33525 .

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. it's https://issues.apache.org/jira/browse/SPARK-33550

<exclusions>
<exclusion>
<groupId>*</groupId>
Expand Down
10 changes: 10 additions & 0 deletions sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
import org.apache.hive.service.cli.operation.Operation;
import org.apache.hive.service.cli.session.HiveSession;
import org.apache.hive.service.cli.session.SessionManager;
import org.apache.hive.service.rpc.thrift.TOperationHandle;
import org.apache.hive.service.rpc.thrift.TProtocolVersion;
import org.apache.hive.service.server.HiveServer2;
import org.slf4j.Logger;
Expand Down Expand Up @@ -567,6 +568,15 @@ public void renewDelegationToken(SessionHandle sessionHandle, HiveAuthFactory au
LOG.info(sessionHandle + ": renewDelegationToken()");
}

@Override
public String getQueryId(TOperationHandle opHandle) throws HiveSQLException {
Operation operation = sessionManager.getOperationManager().getOperation(
new OperationHandle(opHandle));
final String queryId = operation.getParentSession().getHiveConf().getVar(ConfVars.HIVEQUERYID);
LOG.debug(opHandle + ": getQueryId() " + queryId);
return queryId;
}

public SessionManager getSessionManager() {
return sessionManager;
}
Expand Down
3 changes: 2 additions & 1 deletion sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/GetInfoType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@ public enum GetInfoType {
CLI_DESCRIBE_PARAMETER(TGetInfoType.CLI_DESCRIBE_PARAMETER),
CLI_CATALOG_NAME(TGetInfoType.CLI_CATALOG_NAME),
CLI_COLLATION_SEQ(TGetInfoType.CLI_COLLATION_SEQ),
CLI_MAX_IDENTIFIER_LEN(TGetInfoType.CLI_MAX_IDENTIFIER_LEN);
CLI_MAX_IDENTIFIER_LEN(TGetInfoType.CLI_MAX_IDENTIFIER_LEN),
CLI_ODBC_KEYWORDS(TGetInfoType.CLI_ODBC_KEYWORDS);

private final TGetInfoType tInfoType;

Expand Down
3 changes: 3 additions & 0 deletions sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/ICLIService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@


import org.apache.hive.service.auth.HiveAuthFactory;
import org.apache.hive.service.rpc.thrift.TOperationHandle;

public interface ICLIService {

Expand Down Expand Up @@ -98,6 +99,8 @@ RowSet fetchResults(OperationHandle opHandle, FetchOrientation orientation,
String getDelegationToken(SessionHandle sessionHandle, HiveAuthFactory authFactory,
String owner, String renewer) throws HiveSQLException;

String getQueryId(TOperationHandle operationHandle) throws HiveSQLException;

void cancelDelegationToken(SessionHandle sessionHandle, HiveAuthFactory authFactory,
String tokenStr) throws HiveSQLException;

Expand Down
13 changes: 13 additions & 0 deletions ...thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,11 @@
import org.apache.hive.service.ServiceException;
import org.apache.hive.service.auth.HiveAuthFactory;
import org.apache.hive.service.cli.CLIService;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.hive.service.rpc.thrift.TGetQueryIdReq;
import org.apache.hive.service.rpc.thrift.TGetQueryIdResp;
import org.apache.hive.service.server.ThreadFactoryWithGarbageCleanup;
import org.apache.thrift.TException;
import org.apache.thrift.TProcessorFactory;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.server.TThreadPoolServer;
Expand Down Expand Up @@ -107,6 +111,15 @@ protected void initializeServer() {
}
}

@Override
public TGetQueryIdResp GetQueryId(TGetQueryIdReq req) throws TException {
try {
return new TGetQueryIdResp(cliService.getQueryId(req.getOperationHandle()));
} catch (HiveSQLException e) {
throw new TException(e);
}
}

@Override
public void run() {
try {
Expand Down
31 changes: 31 additions & 0 deletions sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,28 @@ public TOpenSessionResp OpenSession(TOpenSessionReq req) throws TException {
return resp;
}

@Override
public TSetClientInfoResp SetClientInfo(TSetClientInfoReq req) throws TException {
// TODO: We don't do anything for now, just log this for debugging.
dongjoon-hyun marked this conversation as resolved.
Show resolved Hide resolved
// We may be able to make use of this later, e.g. for workload management.
if (req.isSetConfiguration()) {
StringBuilder sb = null;
for (Map.Entry<String, String> e : req.getConfiguration().entrySet()) {
if (sb == null) {
SessionHandle sh = new SessionHandle(req.getSessionHandle());
sb = new StringBuilder("Client information for ").append(sh).append(": ");
} else {
sb.append(", ");
}
sb.append(e.getKey()).append(" = ").append(e.getValue());
}
if (sb != null) {
LOG.info("{}", sb);
}
}
return new TSetClientInfoResp(OK_STATUS);
}

private String getIpAddress() {
String clientIpAddress;
// Http transport mode.
Expand Down Expand Up @@ -674,6 +696,15 @@ public TGetCrossReferenceResp GetCrossReference(TGetCrossReferenceReq req)

protected abstract void initializeServer();

@Override
public TGetQueryIdResp GetQueryId(TGetQueryIdReq req) throws TException {
try {
return new TGetQueryIdResp(cliService.getQueryId(req.getOperationHandle()));
} catch (HiveSQLException e) {
throw new TException(e);
}
}

@Override
public abstract void run();

Expand Down
9 changes: 9 additions & 0 deletions ...thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,4 +490,13 @@ public OperationHandle getCrossReference(SessionHandle sessionHandle,
throw new HiveSQLException(e);
}
}

@Override
public String getQueryId(TOperationHandle operationHandle) throws HiveSQLException {
try {
return cliService.GetQueryId(new TGetQueryIdReq(operationHandle)).getQueryId();
} catch (TException e) {
throw new HiveSQLException(e);
}
}
}
1 change: 1 addition & 0 deletions ...riftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIService.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ private[hive] class SparkSQLCLIService(hiveServer: HiveServer2, sqlContext: SQLC
case GetInfoType.CLI_SERVER_NAME => new GetInfoValue("Spark SQL")
case GetInfoType.CLI_DBMS_NAME => new GetInfoValue("Spark SQL")
case GetInfoType.CLI_DBMS_VER => new GetInfoValue(sqlContext.sparkContext.version)
case GetInfoType.CLI_ODBC_KEYWORDS => new GetInfoValue("Unimplemented")
dongjoon-hyun marked this conversation as resolved.
Show resolved Hide resolved
case _ => super.getInfo(sessionHandle, getInfoType)
}
}
Expand Down