Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Atlas databuilder compatibility #1225

Merged
merged 7 commits into from
Aug 3, 2021

Conversation

mgorsk1
Copy link
Contributor

@mgorsk1 mgorsk1 commented Jun 12, 2021

Summary of Changes

Introduces support between Atlas and majority of databuilder models.

The change was done with minimal amount of changes in the proxy itself, it's backwards compatible and unlocks both push and pull mode for ingesting data to Atlas:

  • push using Hive Hook and Spark Atlas Connector to collect hive table & lineage metadata
  • pull using Amundsen Databuilder to leverage available extractors

New compatible models

  • Application
  • Badge
  • Custer
  • Column
  • Column Usage
  • Dashboard Usage
  • Dashboard Owner
  • Database
  • Schema
  • Source
  • Table
  • Table Lineage
  • Table Owner
  • Table Usage
  • Tag
  • User
  • Watermarks

Already supported models

  • Dashboard Chart
  • Dashboard Execution
  • Dashboard Last Modified
  • Dashboard Metadata
  • Dashboard Query
  • Dashboard Table

Two models were temporarily excluded from this effort as they might require redesign in metadata AtlasProxy

  • Column Statistics
  • Programmatic Descriptions

Corresponding PR for introducing new Atlas Types

dwarszawski/amundsen-atlas-types#35

Tests

tbd

Documentation

  • updated

CheckList

Make sure you have checked all steps below to ensure a timely review.

  • PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • PR includes a summary of changes.
  • PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

@boring-cyborg boring-cyborg bot added area:common From common folder area:databuilder From databuilder folder area:metadata From the metadata folder category:models area:all Related to all the project area:frontend From the Frontend folder labels Jun 12, 2021
@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch from 832064b to ffb5f21 Compare June 12, 2021 16:49
@mgorsk1 mgorsk1 removed area:frontend From the Frontend folder category:ui labels Jun 12, 2021
@mgorsk1 mgorsk1 changed the title WIP: feat: Atlas databuilder compatibility feat: Atlas databuilder compatibility Jun 13, 2021
@mgorsk1 mgorsk1 marked this pull request as ready for review June 13, 2021 08:51
@mgorsk1 mgorsk1 requested a review from a team as a code owner June 13, 2021 08:51
@mgorsk1
Copy link
Contributor Author

mgorsk1 commented Jun 13, 2021

cc @dechoma

@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch 3 times, most recently from e0e4719 to f8dea7a Compare June 23, 2021 05:55
@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch from 69ae8d3 to 3f645b7 Compare June 30, 2021 06:11
@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch 2 times, most recently from b578852 to b471a60 Compare July 9, 2021 18:26
@stale
Copy link

stale bot commented Jul 23, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale label Jul 23, 2021
@mgorsk1
Copy link
Contributor Author

mgorsk1 commented Jul 23, 2021

#keepalive

@stale stale bot removed the stale label Jul 23, 2021
@mgorsk1 mgorsk1 added the keep fresh Disables stalebot from closing an issue label Jul 23, 2021
@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch 2 times, most recently from 5e72f8a to 227bac5 Compare July 27, 2021 07:57
@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch from 0c314fe to 227bac5 Compare July 27, 2021 08:00
mgorsk1 added 4 commits August 3, 2021 11:36
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
… changes.

Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
@mgorsk1 mgorsk1 force-pushed the feat/atlas_table_metadata_databuilder branch from 5f445b7 to 21f40cf Compare August 3, 2021 09:36
mgorsk1 added 3 commits August 3, 2021 11:41
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
@mgorsk1 mgorsk1 merged commit e0c6bbb into main Aug 3, 2021
@mgorsk1 mgorsk1 deleted the feat/atlas_table_metadata_databuilder branch October 15, 2021 14:03
zacr pushed a commit to SaltIO/amundsen that referenced this pull request May 13, 2022
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
hansadriaans pushed a commit to DataChefHQ/amundsen that referenced this pull request Jun 30, 2022
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:all Related to all the project area:common From common folder area:databuilder From databuilder folder area:metadata From the metadata folder keep fresh Disables stalebot from closing an issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants