-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add programmatic descriptions parser for [AtlasProxy] #152
feat: Add programmatic descriptions parser for [AtlasProxy] #152
Conversation
cc @verdan |
Codecov Report
@@ Coverage Diff @@
## master #152 +/- ##
==========================================
+ Coverage 72.91% 73.25% +0.34%
==========================================
Files 26 26
Lines 1233 1249 +16
Branches 128 132 +4
==========================================
+ Hits 899 915 +16
Misses 307 307
Partials 27 27
Continue to review full report at Codecov.
|
cc @verdan |
4143cea
to
40cafc9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a few comments
2c226cc
to
723841a
Compare
039e887
to
73c81c9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 LGTM
* commit '369685cc715e95af82dfa4dc14d0c58af8bb1ac9': chore: replace references to Lyft -> Amundsen (amundsen-io#174) feat: Data Owner Implementation of Atlas Proxy (amundsen-io#156) chore: fix docker push action (amundsen-io#172) chore: add docker publish action and remove travis (amundsen-io#171) chore: add pypi publish action (amundsen-io#170) fix: removing OidcConfig file and making statsd configurable through envrionment variable (amundsen-io#157) ci: add dependabot config (amundsen-io#169) Update repo name in travis file (amundsen-io#163) feat: Populate is_view property in AtlasProxy (amundsen-io#155) fix: Overlapping table name issue in Readers [AtlasProxy] feat: Add resource_reports field in Table API ( Atlas proxy) (amundsen-io#149) chore: apply license headers to all the source files (amundsen-io#153) feat: Add programmatic descriptions parser for [AtlasProxy] (amundsen-io#152) feat: Add Frequent Users feature in [AtlasProxy] (amundsen-io#147) feat: Implement configurable minimum number of readers for popular tables (amundsen-io#146) chore: update the email for the project (amundsen-io#148) # Conflicts: # README.md # docs/configurations.md # docs/structure.md # metadata_service/config.py # metadata_service/oidc_config.py # metadata_service/proxy/neo4j_proxy.py # requirements.txt # setup.py
@mgorsk1 Source:
My previous comment on test case: |
Summary of Changes
This MR introduces parsing of programmatic descriptions using parameters field of Table entity in Atlas Proxy. The parameters field is a map of String -> Any and seems like a perfect fit for this usecase.
Moreover, this property can be set for example on hive_table entity programatically with spark sql:
ALTER [TABLE|VIEW] table_name SET TBLPROPERTIES (key1=val1, key2=val2, ...)
- Atlas hive-hook will propagate such action in Hive Metastore to Atlas metadataAll of those key1, key2 etc then become programmatic description entries.
The idea is also to have a filter to remove unwanted properties (like spark technical ones that appear in parameters after creating table with spark). This could also be of use for other proxies.
Tests
Tested for presence and filtering of programmatic desriptions.
Documentation
What documentation did you add or modify and why? Add any relevant links then remove this line
CheckList
Make sure you have checked all steps below to ensure a timely review.
make test