Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to ignore late data for specified transformations #129

Closed
yruslan opened this issue Jan 12, 2023 · 0 comments
Closed

Allow to ignore late data for specified transformations #129

yruslan opened this issue Jan 12, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request Pramen-Scala

Comments

@yruslan
Copy link
Collaborator

yruslan commented Jan 12, 2023

Background

Sometime a table can be late as of BAU. And Pramen can pick it up and run the corresponding transformation as soon as the table loads. But before that Pramen currently considers it a failure.

Feature

Allow specifying for a transformation that it can be late (e.g. dependency checks fail), skip it, but not trigger the pipeline failure.

Example

For a transformation dependency

dependencies = [
  {
    tables = [ dependent_table ]
    date.from = "@infoDate"

    passive = true
   }
]

Update. Found a good name for a dependency that is not optional, but does not raise errors if the job is skipped because of it - passive.

@yruslan yruslan added the enhancement New feature or request label Jan 12, 2023
@yruslan yruslan self-assigned this Jan 12, 2023
@yruslan yruslan changed the title Allow to ignore late data for specified tables or transformations Allow to ignore late data for specified transformations Jan 12, 2023
yruslan added a commit that referenced this issue Jan 16, 2023
yruslan added a commit that referenced this issue Jan 17, 2023
@yruslan yruslan closed this as completed Jan 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Pramen-Scala
Projects
None yet
Development

No branches or pull requests

1 participant