[vdk-core] Introduce post-ingest-sequence env var #682
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
doks5
force-pushed
the
person/andonova/ingestion-design
branch
from
b754466 to
9030974
Compare
ivakoleva
reviewed
Jan 27, 2022
ivakoleva
requested changes
Jan 28, 2022
doks5
force-pushed
the
person/andonova/ingestion-design
branch
from
9030974 to
88dbcf8
Compare
Conversations resolved. Ready for review |
ivakoleva
requested changes
Jan 28, 2022
doks5
force-pushed
the
person/andonova/ingestion-design
branch
from
88dbcf8 to
2cc360b
Compare
ivakoleva
requested changes
Jan 28, 2022
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_base.py
Outdated
Show resolved
Hide resolved
doks5
force-pushed
the
person/andonova/ingestion-design
branch
3 times, most recently
from
9d7bbdc to
0175ce5
Compare
ivakoleva
approved these changes
Jan 31, 2022
doks5
force-pushed
the
person/andonova/ingestion-design
branch
from
0175ce5 to
6747c80
Compare
ivakoleva
reviewed
Jan 31, 2022
doks5
force-pushed
the
person/andonova/ingestion-design
branch
2 times, most recently
from
dd931e4 to
3a53ab5
Compare
ivakoleva
approved these changes
Jan 31, 2022
doks5
force-pushed
the
person/andonova/ingestion-design
branch
2 times, most recently
from
df340a1 to
1f696fc
Compare
antoniivanov
approved these changes
Jan 31, 2022
doks5
force-pushed
the
person/andonova/ingestion-design
branch
3 times, most recently
from
c656c16 to
e803cc7
Compare
As part of the effort to extend the Ingestion API to support plugin chaining, we need to improve the interfaces that are to be used by data engineers and plugin developers. This change adds a new environment variable, `INGEST_PAYLOAD_POSTPROCESS_SEQUENCE` that data engineers can use to specify a plugin/s that would deal with post-ingestion processing tasks, such as sending telemetry, etc. Additionally, the interfaces that plugin developers use in ingestion pre-processing and post-processing tasks, are extended to pass payload, destination_table, target and collection_id as read-only parameters to allow for better workflows. Testing Done: Unit tests pass, and ran a local test ingestion data job as described in https://github.com/vmware/versatile-data-kit/wiki/Ingesting-data-from-DB-into-Database to verify that the existing ingestion workflows are not affected by the interface changes. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
force-pushed
the
person/andonova/ingestion-design
branch
2 times, most recently
from
6f12670 to
9b19e7f
Compare
Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
force-pushed
the
person/andonova/ingestion-design
branch
from
9b19e7f to
ad2fb22
Compare
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682, the signature of the ingest_payload() method was updated to accept a metadata object as part of the ingestion flow. Once the interface change is adopted in vdk-core, this would cause data jobs to fail, due to missing keyword argument "metadata". This change updates the signatures of the ingest_payload() methods, implemented in the vdk-plugins that provide ingestion capabilities. This is done in order to avoid the abovementioned error. Testing Done: Not required, as no plugins currently process the metadata. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682, the signature of the ingest_payload() method was updated to accept a metadata object as part of the ingestion flow. Once the interface change is adopted in vdk-core, this would cause data jobs to fail, due to missing keyword argument "metadata". This change updates the signatures of the ingest_payload() methods, implemented in the vdk-plugins that provide ingestion capabilities. This is done in order to avoid the abovementioned error. Testing Done: Not required, as no plugins currently process the metadata. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682, the signature of the ingest_payload() method was updated to accept a metadata object as part of the ingestion flow. Once the interface change is adopted in vdk-core, this would cause data jobs to fail, due to missing keyword argument "metadata". This change updates the signatures of the ingest_payload() methods, implemented in the vdk-plugins that provide ingestion capabilities. This is done in order to avoid the abovementioned error. Testing Done: Not required, as no plugins currently process the metadata. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682, the signature of the ingest_payload() method was updated to accept a metadata object as part of the ingestion flow. Once the interface change is adopted in vdk-core, this would cause data jobs to fail, due to missing keyword argument "metadata". This change updates the signatures of the ingest_payload() methods, implemented in the vdk-plugins that provide ingestion capabilities. This is done in order to avoid the abovementioned error. Testing Done: Not required, as no plugins currently process the metadata. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682, the signature of the ingest_payload() method was updated to accept a metadata object as part of the ingestion flow. Once the interface change is adopted in vdk-core, this would cause data jobs to fail, due to missing keyword argument "metadata". This change updates the signatures of the ingest_payload() methods, implemented in the vdk-plugins that provide ingestion capabilities. This is done in order to avoid the abovementioned error. Testing Done: Not required, as no plugins currently process the metadata. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682, the signature of the ingest_payload() method was updated to accept a metadata object as part of the ingestion flow. Once the interface change is adopted in vdk-core, this would cause data jobs to fail, due to missing keyword argument "metadata". This change updates the signatures of the ingest_payload() methods, implemented in the vdk-plugins that provide ingestion capabilities. This is done in order to avoid the abovementioned error. Testing Done: Not required, as no plugins currently process the metadata. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 1, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 2, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 2, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 3, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 3, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 3, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 3, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 3, 2022
* [vdk-core] Implementation of new ingestion interfaces As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com> Co-authored-by: ivakoleva <ikoleva@vmware.com>
doks5
added a commit
that referenced
this pull request
Feb 3, 2022
As part of #682 and #672 the ingestion interfaces were extended to support ingestion plugin chaining for pre- and post- processing operations. This change introduces the implementation of the extended ingestion functionality. The IngesterRouter and IngesterBase classes are extended to support the use of pre- and post- ingestion processing plugins. Testing Done: unit tests; functional tests to follow in a separate PR. Signed-off-by: Andon Andonov <andonova@vmware.com>
This was referenced Aug 26, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of the effort to extend the Ingestion API to support plugin
chaining, we need to improve the interfaces that are to be used by
data engineers and plugin developers.
This change adds a new environment variable,
INGEST_PAYLOAD_POSTPROCESS_SEQUENCEthat data engineers can use to specify a plugin/s that would deal with post-ingestion
processing tasks, such as sending telemetry, etc.
Additionally, the interfaces that plugin developers use in ingestion pre-processing
and post-processing tasks, are extended to pass payload, destination_table, target and
collection_id as read-only parameters to allow for better workflows.
Testing Done: Unit tests pass, and ran a local test ingestion data job as described in
https://github.com/vmware/versatile-data-kit/wiki/Ingesting-data-from-DB-into-Database
to verify that the existing ingestion workflows are not affected by the interface changes.
Signed-off-by: Andon Andonov andonova@vmware.com