Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Take into account string maximum length when comparing schemas #468

Closed
yruslan opened this issue Aug 19, 2024 · 0 comments · Fixed by #471
Closed

Take into account string maximum length when comparing schemas #468

yruslan opened this issue Aug 19, 2024 · 0 comments · Fixed by #471
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented Aug 19, 2024 •
edited
Loading

Background

Currently, Pramen tracks changes to schema over time. Data types, however, do not take into account string metadata (maxLength).

It is okay for processing data in Spark since these metadata keys do not affect Spark processing. However, for relational database consumers these changes could be breaking.

Feature

Take into account string and array metadata when comparing schemas.

Example [Optional]

--

Proposed Solution [Optional]

--
@yruslan yruslan added the enhancement New feature or request label Aug 19, 2024
yruslan added a commit that referenced this issue Aug 21, 2024
@yruslan
#468 Take into account maximum string length when comparing schemas.
f71aef2
@yruslan yruslan mentioned this issue Aug 21, 2024
Merged
@yruslan yruslan changed the title Take into account string and array metadata when comparing schemas Take into account string maximum length when comparing schemas Aug 21, 2024
yruslan added a commit that referenced this issue Aug 21, 2024
@yruslan
#468 Take into account maximum string length when comparing schemas.
69825e7
yruslan added a commit that referenced this issue Aug 21, 2024
@yruslan
#468 Take into account maximum string length when comparing schemas.
31b0b94
@yruslan yruslan mentioned this issue Aug 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#468 Take into account maximum string length when comparing schemas. AbsaOSS/pramen
1 participant
@yruslan