We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
If you define a table in metastore with table option instead of path option
table
path
{ name = some_table format = "delta" table = "some_db.some_table" # <---- pramen-py cannot read this }
Pramen-py attemps to read the table with "path" option (and path is empty) and throws the following error
23/03/13 08:08:24 INFO Pramen-Py(err): pyspark.sql.utils.IllegalArgumentException: Path must be absolute: some_db.some_table
def test_metastore_reader_with_delta_table_in_catalog(spark, get_data_stub): metastore_table = MetastoreTable( name="test_table", format=TableFormat.delta, table="test_table", info_date_settings=InfoDateSettings(column="info_date"), ) writer = MetastoreWriter( spark=spark, tables=[metastore_table], info_date=d(2022, 3, 23), ) writer.write("test_table", get_data_stub) metastore_reader = MetastoreReader(spark=spark, tables=[metastore_table]) dataframe = metastore_reader.get_table("test_table") assert set(dataframe.columns) == {"A", "B", "C", "D", "info_date"}
Pramen-py reads the table using table option.
The text was updated successfully, but these errors were encountered:
#165 - Create initial draft of reading delta tables directly from cat…
59b919c
…alog
367473e
b948d90
Successfully merging a pull request may close this issue.
Describe the bug
If you define a table in metastore with
table
option instead ofpath
optionPramen-py attemps to read the table with "path" option (and path is empty) and throws the following error
Failing test case
Expected behavior
Pramen-py reads the table using
table
option.The text was updated successfully, but these errors were encountered: