Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not set scheme if unset #667

Merged
merged 4 commits into from
Mar 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions changelog/@unreleased/pr-667.v2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type: fix
fix:
description: StandaloneEncryptedFileSystem returns to its old behavior, of only
overriding the scheme when the uri uses a scheme
links:
- https://github.com/palantir/hadoop-crypto/pull/667
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

package com.palantir.crypto2.hadoop;

import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.net.URI;
import java.util.EnumSet;
Expand Down Expand Up @@ -146,7 +147,8 @@ public FileChecksum getFileChecksum(Path path) throws IOException {
return fs.getFileChecksum(to(path));
}

private Path to(Path path) {
@VisibleForTesting
Path to(Path path) {
return toFunc.apply(path);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,9 @@ private static Function<Path, Path> setSchemeFunc(final String scheme) {

private static Function<URI, URI> setUriSchemeFunc(final String scheme) {
return uri -> {
if (uri.getScheme() == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two quick things:

  • Do you have an example where things have broken when this scheme is set?
  • Can we write a test that illustrates this behavior in action?

return uri;
}
try {
return new URI(
scheme,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,25 @@ public void testCopyFromLocalFile() throws IOException {
assertThat(readBytes).containsExactly(DATA_BYTES);
}

/**
* Preserving paths with no scheme present is helpful to bypass validation in
* {@link org.apache.hadoop.fs.s3native.S3xLoginHelper#checkPath} when using S3A.
*/
@Test
public void testNoScheme() {
// Convert the file path, because that normally happens by the time FileSystem calls checkPath internally
StandaloneEncryptedFileSystem standaloneEncryptedFileSystem = (StandaloneEncryptedFileSystem) efs;
EncryptedFileSystem encryptedFileSystem =
(EncryptedFileSystem) standaloneEncryptedFileSystem.getRawFileSystem();
PathConvertingFileSystem pathConvertingFileSystem =
(PathConvertingFileSystem) encryptedFileSystem.getRawFileSystem();
Path convertedPath = pathConvertingFileSystem.to(path);

// Just like the original path, the converted path should not have a scheme
assertThat(path.toUri().getScheme()).isNull();
assertThat(convertedPath.toUri().getScheme()).isNull();
}

private static Configuration getBaseConf() {
Configuration conf = new Configuration();
conf.set("fs.efile.impl", StandaloneEncryptedFileSystem.class.getCanonicalName());
Expand Down