Skip to content

Commit

Permalink
[CARMEL-6012] Enhance rewrite table location to absolute location bas…
Browse files Browse the repository at this point in the history
…ed on database location (#970)
  • Loading branch information
wangyum authored and GitHub Enterprise committed Jun 17, 2022
1 parent 8b1c5a3 commit 2a16fb3
Showing 1 changed file with 8 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -540,9 +540,14 @@ private[hive] class HiveClientImpl(
createTime = h.getTTable.getCreateTime.toLong * 1000,
lastAccessTime = h.getLastAccessTime.toLong * 1000,
storage = CatalogStorageFormat(
locationUri = shim.get().getDataLocation(h).map { location =>
val dbUri = Option(h.getDbName).map(db => new URI(client.getDatabase(db).getLocationUri))
CatalogUtils.correctURIWithHost(new URI(location), dbUri)
locationUri = shim.get().getDataLocation(h).map { loc =>
val tableUri = CatalogUtils.stringToURI(loc)
// Before SPARK-19257, created data source table does not use absolute uri.
// This makes Spark can't read these tables across HDFS clusters.
// Rewrite table location to absolute uri based on database uri to fix this issue.
val absoluteUri = Option(tableUri).filterNot(_.isAbsolute)
.map(_ => CatalogUtils.stringToURI(client.getDatabase(h.getDbName).getLocationUri))
CatalogUtils.correctURIWithHost(tableUri, absoluteUri)
},
// To avoid ClassNotFound exception, we try our best to not get the format class, but get
// the class name directly. However, for non-native tables, there is no interface to get
Expand Down

0 comments on commit 2a16fb3

Please sign in to comment.