Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a version of Enceladus sink that uses Enceladus library instead of service #162

Closed
yruslan opened this issue Mar 2, 2023 · 0 comments
Assignees
Labels
DE enhancement New feature or request Pramen-Scala

Comments

@yruslan
Copy link
Collaborator

yruslan commented Mar 2, 2023

Background

Enceladus has 2 modes of operation. The first integrates with Menas service, its datasets and conformance rules. The other is the library that does the standardization. The former mode of operation is supported by Pramen. The latter is not.

Feature

Create the Standardization sink that uses the Enceladus standardization library:
https://github.com/AbsaOSS/spark-data-standardization

Proposed Solution

  • Use a different sink name and class
  • The writing to the raw folder could be shared across Enceladus and Standardization sinks, the common code can be put to EnceladusUtils
@yruslan yruslan added the enhancement New feature or request label Mar 2, 2023
@yruslan yruslan self-assigned this Mar 2, 2023
yruslan added a commit that referenced this issue Mar 3, 2023
yruslan added a commit that referenced this issue Mar 8, 2023
@yruslan yruslan closed this as completed Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DE enhancement New feature or request Pramen-Scala
Projects
None yet
Development

No branches or pull requests

1 participant