Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove binary dependency on org.apache.hadoop.fs.s3.S3FileSystem #94

Closed
wants to merge 1 commit into from
Closed

Remove binary dependency on org.apache.hadoop.fs.s3.S3FileSystem #94

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions src/main/scala/com/databricks/spark/redshift/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ import com.amazonaws.services.s3.{AmazonS3URI, AmazonS3Client}
import com.amazonaws.services.s3.model.BucketLifecycleConfiguration
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.s3.S3FileSystem
import org.slf4j.LoggerFactory

/**
Expand Down Expand Up @@ -108,8 +107,10 @@ private[redshift] object Utils {
def assertThatFileSystemIsNotS3BlockFileSystem(uri: URI, hadoopConfig: Configuration): Unit = {
val fs = FileSystem.get(uri, hadoopConfig)
// Note that we do not want to use isInstanceOf here, since we're only interested in detecting
// exact matches
if (fs.getClass == classOf[S3FileSystem]) {
// exact matches. We compare the class names as strings in order to avoid introducing a binary
// dependency on classes which belong to the `hadoop-aws` JAR, as that artifact is not present
// in some environments (such as EMR). See #92 for details.
if (fs.getClass.getCanonicalName == "org.apache.hadoop.fs.s3.S3FileSystem") {
throw new IllegalArgumentException(
"spark-redshift does not support the S3 Block FileSystem. Please reconfigure `tempdir` to" +
"use a s3n:// or s3a:// scheme.")
Expand Down