#222 Add file based source and metastore table format

[yruslan]

• Added a new source RawFileSource that allows loading files to the metastore without looking inside them. Just file copy.
• Added a new format raw for metastore tables that allows keeping files without understanding which format they contain.
  • If you query such a table using metastore.getTable(), you will get a DataFrame of the list of files in that partition. Ranged queries are supported as well.
• Added a new built-in transformer ConversionTransformer that allows transforming a raw table into parquet or delta metastore table
• Added a new method to the MetastoreReader trait (Scala only, for now):

  /**
    * Gets definition of a metastore table. Please, use with caution and do not write to the underlying path
    * from transformers.
    *
    * @param tableName The name of the table to query.
    * @return The table definition.
    */
  def getTableDef(tableName: String): MetaTableDef

  • It is used to properly handle tables that have raw format, in transformers and sinks.
  • But can also be used for logging table metadata (description, format etc) from custom transformers.
  • Should not be used to get direct access to table internals and never for writing to them.
• Bumped up the minor version to 1.5.0-SNAPSHOT due to the interface change.

[github-actions]

Unit Test Coverage

File	Coverage [89.54%]	🍏
ConversionTransformer.scala	100%	🍏
HiveConfig.scala	100%	🍏
TransformationJob.scala	100%	🍏
MetaTable.scala	98.39%	🍏
MetastorePersistenceRaw.scala	98.13%	🍏
TransferTable.scala	97.77%	🍏
PythonTransformationJob.scala	96.01%	🍏
RawFileSource.scala	92.98%	🍏
SparkUtils.scala	92.15%	🍏
MetastorePersistenceParquet.scala	90.89%	🍏
MetastoreImpl.scala	90.89%	🍏
FsUtils.scala	84.29%	🍏
DataFormatParser.scala	82.31%	🍏
MetastorePersistence.scala	76.15%	🍏
JobBase.scala	75.89%	🍏
AppRunner.scala	73.7%	❌

Total Project Coverage	79.56%	🍏

[jirifilip]

This is very cool functionality! We used to do this stuff in Airflow and the DAG code got really large and unwieldy because of it. It's pretty cool that you were able to generalize this functionality (landing -> raw -> transformation) in a nice way.