Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Builder Record Abstraction #5

Conversation

AndrewCiambrone
Copy link
Contributor

No description provided.

@AndrewCiambrone AndrewCiambrone force-pushed the ajc-data-builder-record-abstraction-rfc branch from 612ef6c to b5e1c5c Compare October 6, 2020 16:39
@feng-tao
Copy link
Member

feng-tao commented Oct 8, 2020

will take a look, thanks for the RFC

Co-authored-by: Marcos Iglesias <190833+Golodhros@users.noreply.github.com>
@AndrewCiambrone AndrewCiambrone requested a review from a team as a code owner October 8, 2020 18:43
@AndrewCiambrone AndrewCiambrone force-pushed the ajc-data-builder-record-abstraction-rfc branch 2 times, most recently from 8e76c0c to 30461fe Compare October 8, 2020 18:47
Co-authored-by: Marcos Iglesias <190833+Golodhros@users.noreply.github.com>
Signed-off-by: Andrew <andrjc4@vt.edu>
@AndrewCiambrone AndrewCiambrone force-pushed the ajc-data-builder-record-abstraction-rfc branch from 30461fe to 61eab0c Compare October 8, 2020 18:58
Signed-off-by: Andrew <andrjc4@vt.edu>
Copy link
Member

@feng-tao feng-tao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you see any backward compatibility issue?

@feng-tao
Copy link
Member

also the DCO fails as well


## Reference-level Explanation (aka Technical Details)

Each of the models in the databuilder are built off of the `GraphSerializable` class. The `GraphSerializable` class is an abstract class that is responsible for validating and returning the nodes and relationships of its subclasses. The interface on the `GraphSerializable` is `next_node` with a return type of `Union[GraphNode, None]` and `next_relation` with a return type of `Union[GraphRelationship, None]`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sense


## Drawbacks

This adds a level of complexity because it adds an additional layer to the databuilder. Instead of having the neo4j serialization built into the models it becomes a separate process. This also would add a small performance degradation because the models are producing named tuples and then they are being convered into another format.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you elaborate a bit or maybe you could create the pr and point me to the place of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah what I was trying to get at here is that there is a slight performance degradation because you are now converting from a named tuple to a dictionary.

https://github.com/amundsen-io/amundsendatabuilder/pull/380/files#diff-1e0a50145a1e195178c2554706e02567df347f84dd1953e2b9308065de371bf7R21

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see, it should be fine.

@feng-tao
Copy link
Member

also could you create the pr with your branch? thanks a lot

@AndrewCiambrone
Copy link
Contributor Author

@AndrewCiambrone
Copy link
Contributor Author

do you see any backward compatibility issue?

I am not sure if its a bug or a typo but some of the timestamps were not unquoted. I currently have it so it is unquoted let me know if that is unwanted.

@feng-tao
Copy link
Member

feng-tao commented Nov 5, 2020

@AndrewCiambrone I just merged the change. Could you update this rfc and move it under rfcs folder and name it 002-xxx.md? thanks.

Signed-off-by: Andrew <andrjc4@vt.edu>
Signed-off-by: Andrew <andrjc4@vt.edu>
@feng-tao feng-tao merged commit 193f897 into amundsen-io:master Nov 5, 2020
@feng-tao feng-tao added Status: Landed The proposed changes are shipped in an actual release and removed Status: Final Comment Period (FCP) On final comment period (seven days) labels Nov 5, 2020
allisonsuarez pushed a commit that referenced this pull request Feb 23, 2021
* RFC for introducing an abstraction layer to the databuilder records

Signed-off-by: Andrew <andrjc4@vt.edu>

* Update templates/000-databuilder-record-abstraction.md

Co-authored-by: Marcos Iglesias <190833+Golodhros@users.noreply.github.com>

* RFC for introducing an abstraction layer to the databuilder records

Signed-off-by: Andrew <andrjc4@vt.edu>

* remove merge tags

Signed-off-by: Andrew <andrjc4@vt.edu>

* move the rfc to the rfc folder

Signed-off-by: Andrew <andrjc4@vt.edu>

* delete old rfc in template folder

Signed-off-by: Andrew <andrjc4@vt.edu>

Co-authored-by: Marcos Iglesias <190833+Golodhros@users.noreply.github.com>
Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>
allisonsuarez pushed a commit that referenced this pull request May 5, 2021
* RFC for introducing an abstraction layer to the databuilder records

Signed-off-by: Andrew <andrjc4@vt.edu>

* Update templates/000-databuilder-record-abstraction.md

Co-authored-by: Marcos Iglesias <190833+Golodhros@users.noreply.github.com>

* RFC for introducing an abstraction layer to the databuilder records

Signed-off-by: Andrew <andrjc4@vt.edu>

* remove merge tags

Signed-off-by: Andrew <andrjc4@vt.edu>

* move the rfc to the rfc folder

Signed-off-by: Andrew <andrjc4@vt.edu>

* delete old rfc in template folder

Signed-off-by: Andrew <andrjc4@vt.edu>

Co-authored-by: Marcos Iglesias <190833+Golodhros@users.noreply.github.com>
Signed-off-by: Allison Suarez Miranda <asuarezmiranda@lyft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Landed The proposed changes are shipped in an actual release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants