Move Avro sampler to FileSystems API #140
Conversation
| extends Sampler[GenericRecord] { | ||
|
|
||
| private val logger: Logger = LoggerFactory.getLogger(classOf[AvroSampler]) | ||
|
|
||
| private def getFileContext: FileContext = FileContext.getFileContext(GcsConfiguration.get()) | ||
| // private def getFileContext: FileContext = FileContext.getFileContext(GcsConfiguration.get()) |
| val fs = FileSystem.get(path.toUri, GcsConfiguration.get()) | ||
| if (fs.isFile(path)) { | ||
| new AvroFileSampler(getFileContext, path, seed).sample(n, head) | ||
| // val fs = FileSystem.get(path.toUri, GcsConfiguration.get()) |
| @@ -92,51 +98,44 @@ class AvroSampler(path: Path, protected val seed: Option[Long] = None) | |||
|
|
|||
| } | |||
|
|
|||
| private class AvroFileSampler(fc: FileContext, path: Path, protected val seed: Option[Long] = None) | |||
| private class AvroFileSampler(r: ResourceId, protected val seed: Option[Long] = None) | |||
There was a problem hiding this comment.
wouldn't it be more generic to expect the file path instead of the resource id?
There was a problem hiding this comment.
I guess it doesn't matter that much since the class is private
| @@ -18,7 +18,7 @@ | |||
| package com.spotify.ratatool.samplers | |||
|
|
|||
| import java.io.File | |||
| import java.nio.file.Files | |||
| import java.nio.file.{Files, Paths} | |||
| @@ -82,15 +82,15 @@ public static Configuration get() { | |||
|
|
|||
| conf.setIfUnset( | |||
| HadoopCredentialConfiguration.BASE_KEY_PREFIX + | |||
| HadoopCredentialConfiguration.ENABLE_SERVICE_ACCOUNTS_SUFFIX, | |||
| HadoopCredentialConfiguration.ENABLE_SERVICE_ACCOUNTS_SUFFIX, | |||
There was a problem hiding this comment.
wondering if we can get rid of this class.
Seems like it is only being used in ParquetIO (besides AvroSampler and an example)
There was a problem hiding this comment.
Yea, Parquet cleanup is not a priority for me at the moment but I think we can get rid of it once that's done.
Codecov Report
@@ Coverage Diff @@
## master #140 +/- ##
=========================================
Coverage ? 67.58%
=========================================
Files ? 36
Lines ? 1447
Branches ? 169
=========================================
Hits ? 978
Misses ? 469
Partials ? 0
Continue to review full report at Codecov.
|
|
Cool, will merge this and solve parquet separately after |
#139 for Avro only