Skip to content

Commit

Permalink
[SPARK-38652][K8S] uploadFileUri should preserve file scheme
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR replaces `new Path(fileUri.getPath)` with `new Path(fileUri)`.
By using `Path` class constructor with URI parameter, we can preserve file scheme.

### Why are the changes needed?

If we use, `Path` class constructor with `String` parameter, it loses file scheme information.
Although the original code works so far, it fails at Apache Hadoop 3.3.2 and breaks dependency upload feature which is covered by K8s Minikube integration tests.

```scala
test("uploadFileUri") {
   val fileUri = org.apache.spark.util.Utils.resolveURI("/tmp/1.txt")
   assert(new Path(fileUri).toString == "file:/private/tmp/1.txt")
   assert(new Path(fileUri.getPath).toString == "/private/tmp/1.txt")
}
```

### Does this PR introduce _any_ user-facing change?

No, this will prevent a regression at Apache Spark 3.3.0 instead.

### How was this patch tested?

Pass the CIs.

In addition, this PR and #36009 will recover K8s IT `DepsTestsSuite`.

Closes #36010 from dongjoon-hyun/SPARK-38652.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit cab8aa1)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Mar 30, 2022
1 parent d81ad57 commit 01600ae
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ object KubernetesUtils extends Logging {
fs.mkdirs(new Path(s"${uploadPath}/${randomDirName}"))
val targetUri = s"${uploadPath}/${randomDirName}/${fileUri.getPath.split("/").last}"
log.info(s"Uploading file: ${fileUri.getPath} to dest: $targetUri...")
uploadFileToHadoopCompatibleFS(new Path(fileUri.getPath), new Path(targetUri), fs)
uploadFileToHadoopCompatibleFS(new Path(fileUri), new Path(targetUri), fs)
targetUri
} catch {
case e: Exception =>
Expand Down

0 comments on commit 01600ae

Please sign in to comment.