Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental transformations and sinks #520

Open
yruslan opened this issue Nov 18, 2024 · 0 comments · May be fixed by #526
Open

Incremental transformations and sinks #520

yruslan opened this issue Nov 18, 2024 · 0 comments · May be fixed by #526
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented Nov 18, 2024 •
edited
Loading

Background

Incremental ingestion is already supported. Now we need to be able to do processing and send data to sinks incrementally as well.

Feature

Incremental transformations and sinks.

Example [Optional]

To be added...

Proposed Solution [Optional]

To be added...
@yruslan yruslan added the enhancement New feature or request label Nov 18, 2024
yruslan added a commit that referenced this issue Nov 20, 2024
@yruslan
#520 Write batchid information for raw tables in the metastore.
df02404
yruslan added a commit that referenced this issue Nov 20, 2024
@yruslan
#520 Write batchid information for raw tables in the metastore.
a753387
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Fix handling of incremental ingestion for 'raw' format of metast…
07d1cc8 
…ore tables.
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Fix handling of incremental ingestion for 'raw' format of metast…
fe94be8 
…ore tables. (#521)
yruslan added a commit that referenced this issue Nov 21, 2024
@yruslan
#520 Prepare interfaces for incremental transformer processing.
e2034c1
yruslan added a commit that referenced this issue Nov 22, 2024
@yruslan
#520 Prepare interfaces for incremental transformer processing.
6f944bf
yruslan added a commit that referenced this issue Nov 22, 2024
@yruslan
#520 Make the default batchid field for tables having 'raw' format to…
d90bd7d 
… match raw file ingestion.
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Implement offset management for incremental processing.
e5755e7
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Allow gracefully adding 'pramen_batchid' field to metastore tabl…
b8b85b1 
…es consistently with the old behavior.
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Add unit tests for the incremental processing.
273e615
yruslan added a commit that referenced this issue Nov 25, 2024
@yruslan
#520 Add unit tests for the incremental processing.
53acd43
yruslan added a commit that referenced this issue Nov 27, 2024
@yruslan
#520 Add chain-commit for incremental jobs with transient job depende…
aeac5df 
…ncies.
yruslan added a commit that referenced this issue Nov 28, 2024
@yruslan
#520 Add chain-commit transaction for incremental jobs with transient…
254dafa 
… job dependencies.
yruslan added a commit that referenced this issue Nov 28, 2024
@yruslan
#520 Move some common offset management code to OffsetManagerUtils.
c381cf5
yruslan added a commit that referenced this issue Nov 29, 2024
@yruslan
#520 Remove redundant flag from the metastore reader.
c9924f4
yruslan added a commit that referenced this issue Nov 29, 2024
@yruslan
#520 Rename the commit flag to make it easier to understand.
b700025
yruslan added a commit that referenced this issue Nov 29, 2024
@yruslan
#520 Add the cache layer to the offset manager calls that incur bigge…
ffa27ae 
…st load on the database.
yruslan added a commit that referenced this issue Nov 29, 2024
@yruslan
#520 Simplify the way metastore reader is created for various purposes.
ec92fc8
yruslan added a commit that referenced this issue Nov 29, 2024
@yruslan
#520 Implement incremental transformations job tracking logic.
2636b3d
yruslan added a commit that referenced this issue Dec 2, 2024
@yruslan
#520 Improve performance of committing of transformers and sinks outp…
b9d4aba 
…ut tables.
yruslan added a commit that referenced this issue Dec 2, 2024
@yruslan
#520 Improve performance of committing of transformers and sinks outp…
fe130d3 
…ut tables.
yruslan added a commit that referenced this issue Dec 10, 2024
@yruslan
#520 Move OffsetCommitRequest to the model package.
7ff08a4
yruslan added a commit that referenced this issue Dec 10, 2024
@yruslan
#520 Increase the size of offset table names, and various fixups.
5bb749b 
The table name in 'offsets' table should be bigger than in bookkeeping because there are virtual tables that take other table names as part of it.
yruslan added a commit that referenced this issue Dec 11, 2024
@yruslan
#520 Make 'isRaw' an interface function instead of relying on 'isInst…
24fc1de 
…anceOf[]'.
yruslan added a commit that referenced this issue Dec 12, 2024
@yruslan
#520 Add unit test suite for the new incremental methods of the raw f…
bc5019e 
…iles source.
yruslan added a commit that referenced this issue Dec 12, 2024
@yruslan
#520 Add unit test suite for the offset manager utils.
365461d
@yruslan yruslan linked a pull request Dec 12, 2024 that will close this issue
Open
yruslan added a commit that referenced this issue Dec 12, 2024
@yruslan
#520 Update README of the new functionality.
7157901
yruslan added a commit that referenced this issue Dec 13, 2024
@yruslan
#520 Add documentation for the identity transformer.
f65794b
@yruslan yruslan mentioned this issue Dec 19, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#520 Add ability to run incremental transformers and sinks AbsaOSS/pramen
1 participant
@yruslan