Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[vdk-core] Implementation of new ingestion interfaces #690

Merged
merged 3 commits into from
Feb 3, 2022
Merged

[vdk-core] Implementation of new ingestion interfaces #690

merged 3 commits into from
Feb 3, 2022

Conversation

doks5
Copy link
Contributor

@doks5 doks5 commented Feb 1, 2022

As part of #682
and #672 the ingestion
interfaces were extended to support ingestion plugin chaining for pre- and
post- processing operations.

This change introduces the implementation of the extended ingestion functionality.
The IngesterRouter and IngesterBase classes are extended to support the use of pre-
and post- ingestion processing plugins.

Testing Done: unit tests; functional tests to follow in a separate PR.

Signed-off-by: Andon Andonov andonova@vmware.com

@doks5 doks5 linked an issue Feb 1, 2022 that may be closed by this pull request
vdk-core: ingestion payload decoration capability #584
Closed
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch 3 times, most recently from b849dd9 to e13b2b8 Compare February 1, 2022 21:10
doks5 added a commit that referenced this pull request Feb 1, 2022
@doks5
[vdk-core] Add ingestion functional tests
c3dae15 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5 doks5 mentioned this pull request Feb 1, 2022
Merged
@doks5 doks5 self-assigned this Feb 1, 2022
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch from e13b2b8 to 88a9b3b Compare February 2, 2022 07:35
doks5 added a commit that referenced this pull request Feb 2, 2022
@doks5
[vdk-core] Add ingestion functional tests
d391d0c 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
ivakoleva
ivakoleva requested changes Feb 2, 2022
View reviewed changes
Copy link
Contributor

@ivakoleva ivakoleva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing test_ignester_base.py scenarios about:

  1. pre ingestion
  2. payload modification propagation from pre-processing to ingestion
  3. metadata propagation verification when chaining (from pre, to ingest, to post -processing)
  4. multiple pre-ingest/post-ingest triggered in a particular sequence

projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_base.py Show resolved Hide resolved
projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_base.py Outdated Show resolved Hide resolved
projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_base.py Outdated Show resolved Hide resolved
projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_router.py Outdated Show resolved Hide resolved
projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_router.py Outdated Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_router.py Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_router.py Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_router.py Outdated Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_base.py Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_base.py Outdated Show resolved Hide resolved
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch from 88a9b3b to 84366b9 Compare February 2, 2022 22:05
@doks5 doks5 requested a review from ivakoleva February 2, 2022 22:16
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
f160519 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
6afb039 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
ivakoleva
ivakoleva requested changes Feb 3, 2022
View reviewed changes
Copy link
Contributor

@ivakoleva ivakoleva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add router tests to verify processors initialisation, then merge?

projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_router.py Outdated Show resolved Hide resolved
projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_router.py Outdated Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_base.py Outdated Show resolved Hide resolved
projects/vdk-core/tests/vdk/internal/builtin_plugins/ingestion/test_ingester_router.py Show resolved Hide resolved
@doks5
[vdk-core] Implementation of new ingestion interfaces
d8dc457 
As part of #682
and #672 the ingestion
interfaces were extended to support ingestion plugin chaining for pre- and
post- processing operations.

This change introduces the implementation of the extended ingestion functionality.
The IngesterRouter and IngesterBase classes are extended to support the use of pre-
and post- ingestion processing plugins.

Testing Done: unit tests; functional tests to follow in a separate PR.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch from 84366b9 to d8dc457 Compare February 3, 2022 09:48
@doks5
Copy link
Contributor Author

doks5 commented Feb 3, 2022

maybe add router tests to verify processors initialisation, then merge?

@ivakoleva
Test added. Thanks for the review

doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
f23c158 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
588b888 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5 doks5 requested a review from ivakoleva February 3, 2022 10:16
@ivakoleva
Copy link
Contributor

maybe add router tests to verify processors initialisation, then merge?

@ivakoleva Test added. Thanks for the review

couldn't find pre-processors tests, sorry if I missed them by any chance,
what I found on processors coverage,
test_ingester_router.py covers:

  • test_router_chained_ingest_plugins <- pre&post processors count initialised corresponds to configured count

test_ingester_base.py covers:

  • test_ingest_payload_and_post_ingestion_operation <- post processor evaluation with parameters expected, in case no exception
  • test_post_ingestion_operation_with_exceptions <- post processor evaluation with parameters expected, in case exception

doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
b60f546 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5
Copy link
Contributor Author

doks5 commented Feb 3, 2022

maybe add router tests to verify processors initialisation, then merge?

@ivakoleva Test added. Thanks for the review

couldn't find pre-processors tests, sorry if I missed them by any chance, what I found on processors coverage, test_ingester_router.py covers:

* test_router_chained_ingest_plugins <- pre&post processors count initialised corresponds to configured count

test_ingester_base.py covers:

* test_ingest_payload_and_post_ingestion_operation <- post processor evaluation with parameters expected, in case no exception

* test_post_ingestion_operation_with_exceptions <- post processor evaluation with parameters expected, in case exception

@ivakoleva
Added tests in test_ingester_base to verify the operations of pre-processors. Thanks

doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
a436b35 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
8a3b3d1 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch from b3085a2 to 528f54b Compare February 3, 2022 13:38
@doks5 doks5 requested a review from ivakoleva February 3, 2022 13:39
@ivakoleva ivakoleva closed this Feb 3, 2022
@ivakoleva ivakoleva deleted the person/andonova/ingestion-chaining branch February 3, 2022 13:45
@doks5 doks5 restored the person/andonova/ingestion-chaining branch February 3, 2022 13:52
@ivakoleva ivakoleva reopened this Feb 3, 2022
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch from 528f54b to 8e80470 Compare February 3, 2022 14:07
@doks5
[vdk-core]: Added tests
664a072 
Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5 doks5 force-pushed the person/andonova/ingestion-chaining branch from 8e80470 to 664a072 Compare February 3, 2022 14:12
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
913982f 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
9c76195 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
1fa00c8 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@doks5 doks5 merged commit f168487 into main Feb 3, 2022
@doks5 doks5 deleted the person/andonova/ingestion-chaining branch February 3, 2022 15:09
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
04e44be 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
0fc5e51 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests
d32c66f 
As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
doks5 added a commit that referenced this pull request Feb 3, 2022
@doks5
[vdk-core] Add ingestion functional tests (#691)
c5cd250 
* [vdk-core] Implementation of new ingestion interfaces

As part of #690, the
IngesterRouter and IngesterBase classes were extended to support ingestion
plugin chaining.

Although unit tests were added as part of the aforementioned PR, no functional
tests were implemented.

This change adds functional tests, that verify the new end-to-end ingestion flow
with pre- and post- processors.

Testing Done: functional tests introduced.

Signed-off-by: Andon Andonov <andonova@vmware.com>
@exalate-issue-sync exalate-issue-sync bot mentioned this pull request Aug 30, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ivakoleva ivakoleva ivakoleva approved these changes

Assignees

@doks5 doks5

Labels
cla-not-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

vdk-core: ingestion payload decoration capability
3 participants
@doks5 @ivakoleva @vmwclabot