#85 Implement passing of format-specific options to pramen-py MetaSto…

[jirifilip]

Implemented passing of format-specific options from Pramen (HOCON config) to Pramen-Py (YAML config).

HOCON config example:

pramen.metastore {
  tables = [
    {
        name = "table"
        format = "parquet"
        path = "/some/path"
        read.option {
          compression = "gzip"
          mergeSchema = "true"
        }
        write.option {
          compression = "gzip"
        }
    },
  ]
}

Generated YAML which is then passed to Pramen-Py:

metastore_tables:
- name: table
  format: parquet
  path: /some/path
  records_per_partition: 500000
  info_date_settings:
    column: pramen_info_date
    format: yyyy-MM-dd
    start: 2020-01-01
  reader_options:
    compression: "gzip"
  writer_options:
    compression: gzip

Somes ideas for discussion

• on Scala side, I modified MetastoreSpy to allow specifying readOptions and writeOptions but those options are then applied to every mocked metastore table. I did this for simplicity reasons but is that ok/clean?
• on Python side, I kind of wrote "dumb" tests 😃 (one function per test, not any parametrization). I was toying with @pytest.mark.parametrize but the variation of my test cases was too big and I thought the intention of what is tested was lost. But happy to revise and rework it to a different form or to use the parametrization since I understand the tests maybe don't exactly match the repo's style.

[yruslan]

Looks great! Just one tiny point to consider