[KSHC] Support Kerberized HMS in cluster mode w/o keytab

[pan3793]

Why are the changes needed?

This PR aims to make Kyuubi Spark Hive Connector(KSHC) support kerberized HMS in cluster mode w/o keytab(which is the typical use case in Kyuubi) by implementing a HadoopDelegationTokenProvider.

To enable access to an kerberized HMS using KSHC, the minimal configurations are

spark.sql.catalog.warm=org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
spark.sql.catalog.warm.hive.metastore.uris=<thrift-uris>

then it's able to run federation query across metastores

SELECT * FROM spark_catalog.db1.tbl1 JOIN warm.db2.tbl2 ON ...

In addition, it allows disabling token renewal for each catalog explicitly

spark.sql.catalog.warm.delegation.token.renewal.enabled=false

The current implementation has some limitations:

the catalog configuration must be present on the Spark application bootstrap, which means the catalog configurations should be set in spark-defaults.conf or append as --conf like:

spark-[shell|submit] \
  --conf spark.sql.catalog.xxx=org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
  --conf spark.sql.catalog.xxx.hive.abc=xyz

note: spark-sql may not work properly, because of some tricky code inside Spark ...

but does not work for dynamic registering through SET statement, e.g. SET spark.sql.catalog.xxx=

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

> (select count(*) from hive_2.mammut.test_7) union ( select count(*) from spark_catalog.test.test01 limit 1);
+-----------+
| count(1)  |
+-----------+
| 4         |
| 1         |
+-----------+
2 rows selected (8.378 seconds)

• Run test locally before make a pull request

[codecov-commenter]

Codecov Report

Merging #4560 (8511595) into master (acdfa6c) will decrease coverage by 0.03%.
The diff coverage is n/a.

❗ Current head 8511595 differs from pull request most recent head fe8cd0c. Consider uploading reports for the commit fe8cd0c to get more accurate results

@@             Coverage Diff              @@
##             master    #4560      +/-   ##
============================================
- Coverage     53.34%   53.31%   -0.03%     
  Complexity       13       13              
============================================
  Files           577      577              
  Lines         31546    31557      +11     
  Branches       4244     4244              
============================================
- Hits          16827    16825       -2     
- Misses        13135    13148      +13     
  Partials       1584     1584              

see 11 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[pan3793]

cc @zhouyifan279

[yaooqinn]

The Hadoop configuration here seems to be shared by all hive clients, which means you share the same security key to open all the doors of different HMS-s?

[pan3793]

The catalog hive2 could overwrite some dedicated Hadoop configurations by

spark.sql.catalog.hive2.<hadoop-conf-key>=<value>

What do you mean "security key"?

[pan3793]

@yaooqinn main changes after your last review, please take a look again

• switch from RetryingMetaStoreClient to HiveMetaStoreClient
• one token request failure does not block the subsequent token obtaining
• allow to disable token renewal for each catalog explicitly

[zhouyifan279]

Most user may not be familiar with hive.metastore.token.signature's usage.
Can we generate a hive.metastore.token.signature for each catalog by default?

[pan3793]

@zhouyifan279 good suggestion, I updated code to make it fallback to hive.metastore.uris when hive.metastore.token.signature is absent.

The change passes local tests w/ hive.metastore.token.signature and w/o hive.metastore.token.signature, PR description is updated as well.

[pan3793]

Thanks all, merged to master