WIP: add internal lineage RFC to open source #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: feng-tao <fengtao04@gmail.com>
|
this RFC is not ready for review |
Golodhros
reviewed
Oct 23, 2020
| @@ -0,0 +1,133 @@ | |||
| - Feature Name: (fill me in with a unique ident, my_awesome_feature) | |||
| - Start Date: (fill me in with today's date, YYYY-MM-DD) | |||
| - RFC PR: [amundsen-io/rfcs#0000](https://github.com/amundsen-io/rfcs/pull/0000) (after opening the RFC PR, update this with a link to it and update the file name) | |||
There was a problem hiding this comment.
Suggested change
| - RFC PR: [amundsen-io/rfcs#0000](https://github.com/amundsen-io/rfcs/pull/0000) (after opening the RFC PR, update this with a link to it and update the file name) | |
| - RFC PR: [amundsen-io/rfcs#0009](https://github.com/amundsen-io/rfcs/pull/9) |
Golodhros
reviewed
Oct 23, 2020
| ## Support Lineage In Amundsen | ||
|
|
||
|
|
||
| This RFC drafts how lineage should be supported in Amundsen and why the feature is important for the trust in data initiative. |
There was a problem hiding this comment.
Suggested change
| This RFC drafts how lineage should be supported in Amundsen and why the feature is important for the trust in data initiative. | |
| This RFC drafts how lineage should be supported in Amundsen and why the feature is important. |
Golodhros
reviewed
Oct 23, 2020
| * Column lineage: It tracks information in fine detail, at the level of particular columns rather than entire tables. It shows how data of a given column is generated based on upstream columns, and how it affects which downstream columns. | ||
| * _Service and event lineage_: which service produces and generates the message for a given event table. | ||
|
|
||
| Currently Amundsen doesn’t support showing Table level lineage in the UI but instead redirect users to use , which is a 3rd party tool at Lyft, to refer to the lineage information. |
There was a problem hiding this comment.
to use X ,
Something missing here, right?
active-rfcs/0003-lineage.md
Outdated
|
|
||
|
|
||
|
|
||
| * The link to in-house lineage tool doesn’t always work. If the table/dataset is not indexed in in-house lineage tool, the user will get a 404 (e.g [user report](https://lyft.slack.com/archives/CBHCUQB2Q/p1584997367078000)). |
There was a problem hiding this comment.
We'll probably want to remove that report from our internal Slack
active-rfcs/0003-lineage.md
Outdated
|
|
||
|
|
||
| * The link to in-house lineage tool doesn’t always work. If the table/dataset is not indexed in in-house lineage tool, the user will get a 404 (e.g [user report](https://lyft.slack.com/archives/CBHCUQB2Q/p1584997367078000)). | ||
| * The UI in in-house lineage tool is hard to browse the actual lineage as it displays the full lineage instead one level at a time (E.g [graph](https://in-house lineage tool.lyft.net/lineage-report/relation/warehouse.core.fact_rides.html?cwRelationContent=graph&cwRelationGraph=dataflow) ) |
| * Other popular OSS projects (e.g Apache Atlas) provide the UI support on lineage which is a feature gap for Amundsen adoption in the community. | ||
|
|
||
|
|
||
| ### Why is lineage important for trust in data? |
There was a problem hiding this comment.
Suggested change
| ### Why is lineage important for trust in data? | |
| ### Why is lineage important? |
|
close this pr to favor of #24 which is a more updated version. |
Golodhros
added
Status: Rejected
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Move the internal google doc RFC to open source, I will refine it or others will help to refine it once we have the UX mock.
Signed-off-by: feng-tao fengtao04@gmail.com