Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#475 Add Delta Lake location or managed table as a bookkeeping storage option #478

Merged
merged 7 commits into from
Aug 28, 2024

Conversation

yruslan
Copy link
Collaborator

@yruslan yruslan commented Aug 28, 2024

Closes #475

This requires Delta Lake format support from the cluster you are running pipelines at.

You can use wither a path:

pramen.bookkeeping {
  enabled = "true"
  hadoop.format = "delta"
  location = "s3://path"
}

or a set of managed tables:

pramen.bookkeeping {
  enabled = "true"
  hadoop.format = "delta"
  delta.database = "my_db"  # Optional. 'default' will be used if not speified
  delta.table.prefix = "bk_"
}

@yruslan yruslan changed the title #475 Add Delta Lake location or managed table as a bookkeeping storage #475 Add Delta Lake location or managed table as a bookkeeping storage option Aug 28, 2024
Copy link

github-actions bot commented Aug 28, 2024

Unit Test Coverage

File Coverage [53.44%]
JournalHadoopCsv.scala 100% 🍏
BookkeeperConfig.scala 87.74% 🍏
JournalHadoopDeltaTable.scala 87.35% 🍏
TokenLockHadoopPath.scala 83.85% 🍏
HadoopFormat.scala 72.55%
Bookkeeper.scala 69.23%
BookkeeperDeltaPath.scala 50.2%
TokenLockFactoryHadoopPath.scala 45%
BookkeeperText.scala 36.58%
JournalHadoopDeltaPath.scala 11.85%
BookkeeperDeltaBase.scala 2.6%
BookkeeperDeltaTable.scala 0%
Total Project Coverage 81.04% 🍏

@yruslan
Copy link
Collaborator Author

yruslan commented Aug 28, 2024

Do not mind low coverage. These files are test covered with integration tests that Jacoco does not run.

@yruslan yruslan merged commit 8d327ff into main Aug 28, 2024
8 checks passed
@yruslan yruslan deleted the feature/475-add-delta-bookkeeperstorage branch August 28, 2024 08:07
@yruslan yruslan mentioned this pull request Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support of delta table for bookkeeper
1 participant