Lineage Stage 0 #25

allisonsuarez · 2021-02-23T18:40:36Z

Closed #24 in favor of this one

This is an RFC describing the first iteration of Lineage related work.

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

…et number now Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

allisonsuarez · 2021-02-23T18:43:08Z

@danwom use this one instead

feng-tao · 2021-02-23T21:17:26Z

rfcs/025-lineage-stage-0.md

+## Summary
+
+
+Currently Amundsen doesn't have a way of surfacing lineage information for tables and columns. The idea for this first iteration is to have a way to show upstream and downstream tables and columns to users through the Table Details page so they can explore the current resource's lineage as well as navigate to related resources in Amundsen.


FYI, actually we could use programmatic description to surface the lineage . I think what it is lacking, it is a graph UI to surface the lineage intuitively.

rfcs/025-lineage-stage-0.md

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

- Added photos into a nested assets/ folder. Signed-off-by: Daniel Won <dwon@lyft.com>

feng-tao · 2021-03-02T22:24:14Z

the png in https://github.com/amundsen-io/rfcs/blob/59bd5b98c15e823fcf1757fcd5ff97790e215dae/rfcs/025-lineage-stage-0.md doesn't seem to work

feng-tao · 2021-03-02T23:45:40Z

I will take a look tonight

verdan · 2021-03-03T10:12:50Z

rfcs/025-lineage-stage-0.md

+
+![Lineage Stage 0 Architecture](assets/025/lineage-arch.png)
+
+Implementing this feature will require defining a Lienage API on the metadata service for Tables and Columns. When the API is called it will make calls to neo4j and whatever the source of lineage data is. An interface needs to be created to interact with an implementer's lineage service in a generic way. The data from the calls to these services will be put together to form the lineage response as defined below.


Would this generic API call be made to any of the metadata proxies? or is this simply a configurable function which users will be able to extend themselves?
Suppose a user has a proxy set to Atlas. For Stage 0, will this endpoint call a function inside the atlas_proxy?

Yeah on the Base Proxy class I added a get_lineage method that can be implemented and will be called for any proxy when the endpoint is hit https://github.com/amundsen-io/amundsenmetadatalibrary/blob/master/metadata_service/proxy/base_proxy.py#L160

So the idea is that with this method any lineage provider (like Atlas, Marquez, spline or open lineage) can be supported, right? Any proxy would support any lineage provider really.

@mgorsk1 I think this is just step 0 which we should bring the backend model (or I assume we will bring) with ingestion in step 1/2. The reason on why we are doing this is because Lyft still uses a 3rd party vendor lineage service . It is easier for the implementation in these kinds of steps by focusing on FE first with backend proxy; then build a backend model/ ingestion with push mechanism in a subsequent step.

Yeah we will hopefully add a backend model for lineage in the future so that we can query the db directly for lineage rather than making ad hoc calls to a provider from metadata. In that case we would have to add support for lineage extraction on databuilder.

Clear, for me support for external service is more than fine, just wanted to make sure I got it right.

verdan

This looks good to me for Stage 0.

feng-tao · 2021-03-03T17:18:27Z

@danwom @allisonsuarez here is the preview (https://github.com/amundsen-io/rfcs/blob/59bd5b98c15e823fcf1757fcd5ff97790e215dae/rfcs/025-lineage-stage-0.md) . Could you update and fix the broken image links? thanks

feng-tao · 2021-03-03T19:34:24Z

rfcs/025-lineage-stage-0.md


 We will add two additional tabs to the `Table Details` page, `Upstream` and `Downstream`. Each tab will contain a list of tables from which data is inherited or consumed. This allows users view a table's lineage in a very simple manner.

-![Column Lineage Preview](assets/025/column-lineage-preview.png)
+![Column Lineage Preview](../assets/025/column-lineage-preview.png)


any reason we don't go with lineage tab with column and dashboard, then have upstream/downstream subtab underneath?

I think there's a preference to avoid nested tabs when possible. At some point if we have too many tabs then we can reconsider nesting or other options.

For me it feels weird to break it into two tabs. lineage is, after all, a big picture of data flow - so would make sense to have it in one place. It'd make it easier to analyze it for end user.

yeah, +1 on @mgorsk1 , I felt the same to have single lineage tab with nested tab for upstream/downstream (not sure how complexity it changes for FE implementation). @danwom @allisonsuarez could you share more about the details of the preferences?

+1 for not using the nested tabs where possible.

For Stage 0, it will be a list of items for each direction, which IMHO is okay to be placed in different tabs. But yes, for the next milestone, where we'll have a graph/chart of the complete lineage (and not the list of tables), that must be placed in one view under 1 tab to have a clear view of the complete lineage.

I am not really advocating for nested tabs as this sounds more like implementation preference rather than UX question and im fine avoiding it if we don't like nesting.

But is nesting the only way this could be implemented in single tab ? I would even see something like single list (sorted by upstream and then downstream) with table names and icon/abbreviation if the table is upstream or downstream.

Signed-off-by: Daniel Won <dwon@lyft.com>

feng-tao

lgtm for stage 0

* started writing rfc and added dir Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com> * made small change on README and added more to RFC will create PR to get number now Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com> * had to redo this Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com> * reverted weird changes Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com> * changed naming again Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com> * Added details to rfc 025 lineage. - Added photos into a nested assets/ folder. Signed-off-by: Daniel Won <dwon@lyft.com> * Fixed relative image links in rfc 025 Signed-off-by: Daniel Won <dwon@lyft.com> Co-authored-by: Daniel Won <dwon@lyft.com> Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

allisonsuarez added 5 commits August 24, 2020 15:42

started writing rfc and added dir

56e25c1

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

made small change on README and added more to RFC will create PR to g…

6b0de98

…et number now Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

Merge branch 'master' of https://github.com/amundsen-io/rfcs

d40b7d2

Merge branch 'master' of https://github.com/amundsen-io/rfcs

86dff5c

had to redo this

b18673c

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

allisonsuarez requested a review from a team as a code owner February 23, 2021 18:40

reverted weird changes

03689e9

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

allisonsuarez mentioned this pull request Feb 23, 2021

Lineage Stage 0 #24

Closed

allisonsuarez added the Status: Draft label Feb 23, 2021

feng-tao reviewed Feb 23, 2021

View reviewed changes

rfcs/025-lineage-stage-0.md Show resolved Hide resolved

feng-tao reviewed Feb 23, 2021

View reviewed changes

rfcs/025-lineage-stage-0.md Outdated Show resolved Hide resolved

allisonsuarez and others added 2 commits February 23, 2021 14:41

changed naming again

d43148f

Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>

Added details to rfc 025 lineage.

59bd5b9

- Added photos into a nested assets/ folder. Signed-off-by: Daniel Won <dwon@lyft.com>

danwom force-pushed the asm-lineage-0 branch from b54a314 to 59bd5b9 Compare March 2, 2021 21:48

allisonsuarez requested review from feng-tao, dorianj and markgrover March 2, 2021 22:46

allisonsuarez added Status: Active and removed Status: Draft labels Mar 2, 2021

allisonsuarez requested a review from Golodhros March 2, 2021 22:47

verdan reviewed Mar 3, 2021

View reviewed changes

verdan approved these changes Mar 3, 2021

View reviewed changes

feng-tao reviewed Mar 3, 2021

View reviewed changes

Fixed relative image links in rfc 025

cbb4171

Signed-off-by: Daniel Won <dwon@lyft.com>

danwom force-pushed the asm-lineage-0 branch from 8ae4018 to cbb4171 Compare March 3, 2021 19:43

feng-tao approved these changes Mar 4, 2021

View reviewed changes

allisonsuarez added Status: Final Comment Period (FCP) On final comment period (seven days) and removed Status: Active labels Mar 4, 2021

feng-tao added Status: Landed The proposed changes are shipped in an actual release and removed Status: Final Comment Period (FCP) On final comment period (seven days) labels Mar 9, 2021

feng-tao merged commit e52fb0d into master Mar 9, 2021

feng-tao deleted the asm-lineage-0 branch March 9, 2021 05:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lineage Stage 0 #25

Lineage Stage 0 #25

allisonsuarez commented Feb 23, 2021

allisonsuarez commented Feb 23, 2021

feng-tao Feb 23, 2021

feng-tao commented Mar 2, 2021

feng-tao commented Mar 2, 2021

verdan Mar 3, 2021

allisonsuarez Mar 3, 2021

mgorsk1 Mar 3, 2021 •

edited

Loading

feng-tao Mar 3, 2021 •

edited

Loading

allisonsuarez Mar 3, 2021

mgorsk1 Mar 4, 2021

verdan left a comment

feng-tao commented Mar 3, 2021

feng-tao Mar 3, 2021

danwom Mar 3, 2021

mgorsk1 Mar 3, 2021

feng-tao Mar 3, 2021

verdan Mar 3, 2021

mgorsk1 Mar 4, 2021 •

edited

Loading

feng-tao left a comment

		## Summary


		Currently Amundsen doesn't have a way of surfacing lineage information for tables and columns. The idea for this first iteration is to have a way to show upstream and downstream tables and columns to users through the Table Details page so they can explore the current resource's lineage as well as navigate to related resources in Amundsen.


		![Lineage Stage 0 Architecture](assets/025/lineage-arch.png)

		Implementing this feature will require defining a Lienage API on the metadata service for Tables and Columns. When the API is called it will make calls to neo4j and whatever the source of lineage data is. An interface needs to be created to interact with an implementer's lineage service in a generic way. The data from the calls to these services will be put together to form the lineage response as defined below.

Lineage Stage 0 #25

Lineage Stage 0 #25

Conversation

allisonsuarez commented Feb 23, 2021

allisonsuarez commented Feb 23, 2021

Choose a reason for hiding this comment

feng-tao commented Mar 2, 2021

feng-tao commented Mar 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgorsk1 Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

feng-tao Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

verdan left a comment

Choose a reason for hiding this comment

feng-tao commented Mar 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgorsk1 Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

feng-tao left a comment

Choose a reason for hiding this comment

mgorsk1 Mar 3, 2021 •

edited

Loading

feng-tao Mar 3, 2021 •

edited

Loading

mgorsk1 Mar 4, 2021 •

edited

Loading