-
Notifications
You must be signed in to change notification settings - Fork 318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use io.openlineage.server.*
pkg and add class Metadata
#2853
Conversation
Signed-off-by: wslulciuc <willy@datakin.com>
✅ Deploy Preview for peppy-sprite-186812 canceled.
|
@Builder | ||
@ToString | ||
@EqualsAndHashCode | ||
public static final class Dataset { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's worth including column lineage as well as the integration type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given this PR has been out for a while, and the Metadata
class isn't yet used within the codebase, I'll add column lineage facet extraction as follow up.
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
…`, and more Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2853 +/- ##
============================================
- Coverage 83.08% 82.15% -0.93%
- Complexity 1500 1504 +4
============================================
Files 265 268 +3
Lines 6888 7258 +370
Branches 320 325 +5
============================================
+ Hits 5723 5963 +240
- Misses 1007 1134 +127
- Partials 158 161 +3 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
…sOnly()` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! This is going to improve the write path and improve interacting with and enhancing the core events with updates to the OL Spec.
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com>
This PR removes the OpenLineage models that were first introduced in the PoC in favor of
io.openlineage.server.*
defined specifically for consumers. This PR also introduces classesMetadata
andVersionId
. Below, we outlined more specifically on their importance and usage.class
Metadata
Wrapper class for parsing:
OpenLineage.RunEvent
OpenLineage.JobEvent
OpenLineage.DatasetEvent
Methods:
Metadata.Run.forEvent(OpenLineage.RunEvent):
Metadata.Run
Metadata.Job.forEvent(OpenLineage.JobEvent):
Metadata.Job
Metadata.Dataset.forEvent(OpenLineage.DatasetEvent):
Metadata.Dataset
class
VersionId
Factory class for
VersionId
s.Methods:
VersionId.forJob():
VersionId
VersionId.forDataset():
VersionId
Closes: #1650