Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDBC reader - get column descriptions by getting 'REMARKS' metadata field from a connection #362

Closed
yruslan opened this issue Feb 20, 2024 · 0 comments · Fixed by #364
Closed

JDBC reader - get column descriptions by getting 'REMARKS' metadata field from a connection #362

yruslan opened this issue Feb 20, 2024 · 0 comments · Fixed by #364
Labels
DE enhancement New feature or request Pramen-Scala

Comments

@yruslan
Copy link
Collaborator

yruslan commented Feb 20, 2024

Background

Some RDMS provide column descriptions via REMARKS column metadata field.

It can be used to populate COMMENT metadata field of Spark columns propagating it to Paqruet, Delta, and Catalogs.

Feature

JDBC reader - get column descriptions by getting 'REMARKS' metadata field from a connection.

Example

TBA

Proposed Solution

TBA
@yruslan yruslan added enhancement New feature or request Pramen-Scala DE labels Feb 20, 2024
yruslan added a commit that referenced this issue Feb 22, 2024
@yruslan
#362 Implement fetching of column descriptions from JDBC databases.
875efda
yruslan added a commit that referenced this issue Feb 22, 2024
@yruslan
#362 Add 'comment' metadata when using JDBC source.
f9a9b97
yruslan added a commit that referenced this issue Feb 23, 2024
@yruslan
#362 Add unit tests for the new feature.
4a708c1
@yruslan yruslan mentioned this issue Feb 23, 2024
Merged
yruslan added a commit that referenced this issue Feb 23, 2024
@yruslan
#362 Improve fetching descriptions for databases that keep table cata…
199206d 
…log always in lowercase.
yruslan added a commit that referenced this issue Feb 27, 2024
@yruslan
#362 Implement fetching of column descriptions from JDBC databases.
23c1125
yruslan added a commit that referenced this issue Feb 27, 2024
@yruslan
#362 Add 'comment' metadata when using JDBC source.
0e70c98
yruslan added a commit that referenced this issue Feb 27, 2024
@yruslan
#362 Add unit tests for the new feature.
3ff801e
yruslan added a commit that referenced this issue Feb 27, 2024
@yruslan
#362 Improve fetching descriptions for databases that keep table cata…
248b41c 
…log always in lowercase.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
DE enhancement New feature or request Pramen-Scala
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#362 Get column descriptions from JDBC connection when possible AbsaOSS/pramen
1 participant
@yruslan