feat(stream_processor): initial version for processing product changes #76

heitorlessa · 2023-09-30T19:18:55Z

closes #75

Summary

Please provide a summary of what's being changed

Initial skeleton to create stream processor to handle product updates coming from CRUD.

Changes

Tasks for a separate PR due to size

Create integ tests (it'll need careful thought in infra)
Create E2E tests (will incorporate if PR isn't big)
Docs/examples for suboptimal and optimal
Implement Powertools
- Batch
- Create model for Stream Input
We're including dev requirements in the final build (needs issue, pants2 should make all of this simpler)

unrelated but noticeable changes

Support for cdk watch to speed prototyping
Add Pytest config to ignore tests in cdk.out
Allow git local branches named with _
Use getpass.getuser() over outdated os.getlogin() to take into account env vars (mine was root, now it's lessa)
Enable Pydantic Plugin for Mypy

User experience

If applicable, please share what the user experience looks like before and after this change

Checklist

If your change doesn't seem to apply, please leave them unchecked.

I have performed a self-review of this change
Changes have been tested
Changes are documented
PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

Migration process documented
Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

…on already Signed-off-by: heitorlessa <lessa@amazon.co.uk>

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

…typed Signed-off-by: heitorlessa <lessa@amazon.co.uk>

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa · 2023-10-04T12:36:09Z

make pr passing locally and should be in CI. I've enabled Pydantic plugin for Mypy to fix other false positive errors.

Waiting for CI before I rename dal to integrations to match the deck and make it easier to spot its intent.

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa · 2023-10-04T12:43:18Z

I believe the deployment credentials error is because it's coming from my fork, and not as a branch from the repo. Everything else passed.

Gonna add docstrings now but implementation wise is ready for review.

ran-isenberg · 2023-10-04T13:13:59Z

@heitorlessa I'll need to port lots of code to the template , yikes. but thx for fixing lots of side issues

ran-isenberg · 2023-10-04T13:24:39Z

@heitorlessa I forked your branch and it passes all tests . coverage is down 2% though
you can run make coverage-tests and look at the coverage.xml file.
we can sort it later

heitorlessa · 2023-10-04T13:25:25Z

@heitorlessa I forked your branch and it passes all tests . coverage is down 2% though you can run make coverage-tests and look at the coverage.xml file. we can sort it later

sure thing. I'm working on docstrings now.

Check how cool this looks ;)

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

ran-isenberg · 2023-10-04T14:30:54Z

when i run make docs i a 404 page, when i click on homepage link to the left, it works
@heitorlessa

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

ran-isenberg · 2023-10-04T14:42:53Z

pyproject.toml

@@ -55,6 +55,12 @@ types-requests = "*"
 toml = "*"


+[tool.poetry.group.dev.dependencies]


we already have the group:
[tool.poetry.dev-dependencies]
so which is the best dev deps name?

ran-isenberg · 2023-10-04T14:43:20Z

tests/integration/conftest.py

@@ -18,7 +18,7 @@ def init():
    os.environ[POWER_TOOLS_LOG_LEVEL] = 'DEBUG'
    os.environ['REST_API'] = 'https://www.ranthebuilder.cloud/api'
    os.environ['ROLE_ARN'] = 'arn:partition:service:region:account-id:resource-type:resource-id'
-    os.environ['AWS_DEFAULT_REGION'] = 'us-east-1'  # used for appconfig mocked boto calls
+    os.environ['AWS_DEFAULT_REGION'] = os.environ.get('AWS_DEFAULT_REGION', 'us-east-1')  # used for appconfig mocked boto calls


remove the appconfig comment, it's my bad :)

ran-isenberg · 2023-10-04T14:55:07Z

product/stream_processor/integrations/events/functions.py

+_pascal_to_snake_pattern = re.compile(rf'({_exclude_underscores}{_pascal_case}{_or}{_followed_by_lower_case_or_digit}')
+
+
+def convert_model_to_event_name(model_name: str) -> str:


i know that pydantic has support for automatic serializations thagt tranfser from whatever format to another
see this example:

def to_camel_lower_case(snake_str: str) -> str:
components = snake_str.split('_')
return components[0] + ''.join(x.capitalize() for x in components[1:])

then you add to the pydantic class this config. it means that when you do .model_dump it will run the to_camel_lower_case function automatically
class Config:
alias_generator = to_camel_lower_case
allow_population_by_field_name = False

ran-isenberg · 2023-10-04T14:55:48Z

product/stream_processor/integrations/events/functions.py

+
+T = TypeVar('T')
+
+_exclude_underscores = r'(?!^)(?<!_)'


functions is a bit of generic name ? maybe utils? or move the schema related ones to the schema folder?

I chose functions as a more narrow version of utils. Similar to tests/.../data_builders.py instead of utils.py. That way I know where utility functions are, instead of bin packing into utils.py.

Makes sense?

ran-isenberg · 2023-10-04T15:20:48Z

tests/unit/stream_processor/data_builder.py

+from product.models.products.product import ProductChangeNotification
+
+
+def generate_dynamodb_stream_events(product_id: str = '8c18c85a-0f10-4b73-b54a-07ab0d381018',) -> dict[str, Any]:


you move the generate_product_id from crud_utils.py and you use it instead of a default id

ran-isenberg · 2023-10-04T16:19:26Z

product/stream_processor/integrations/events/models/input.py

+class EventMetadata(BaseModel):
+    event_name: str
+    event_source: str
+    event_version: str


let's add a default value here?

ran-isenberg · 2023-10-04T16:20:07Z

product/stream_processor/integrations/events/models/input.py

+
+
+class Event(BaseModel, Generic[AnyModel]):
+    data: AnyModel


shouldnt the timestamp and version reside here? they are like a hint for parsing everything else

ran-isenberg · 2023-10-04T16:21:41Z

product/stream_processor/integrations/events/models/output.py

+
+
+class EventReceiptFail(BaseModel):
+    receipt_id: str


same here for field min length, so they are not empty

product/stream_processor/integrations/events/providers/eventbridge.py

ran-isenberg · 2023-10-04T16:23:57Z

product/stream_processor/integrations/events/providers/eventbridge.py

+                error_message = exc.response['Error']['Message']
+
+                receipt = EventReceiptFail(receipt_id='', error='error_message', details=exc.response['ResponseMetadata'])
+                raise ProductNotificationDeliveryError(f'Failed to deliver all events: {error_message}', receipts=[receipt]) from exc


about the error handling here, if we raise the exception, we stop going over the batch right? shouldnt we do failed.extend(batch somehow) and do continue in the loop?

ran-isenberg · 2023-10-04T16:46:34Z

@heitorlessa top class code, really.
I've left many questions, comments like : very nice, and small suggestions.
lets merge and fix that in next PR.
I'll start working on my event handler.

but damn, this one was hard, lots of files and required concentration.

* main: fix: fix broken main, add automatic github oages publish (#80) feat(stream_processor): initial version for processing product changes (#76) Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa added 16 commits September 29, 2023 10:12

chore: support for cdk watch to speedup dev

ba6c401

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

fix: ensure pytest only searches tests folder

01bb800

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: cdk watch for infra too

a452f86

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: use getpass over outdated getlogin

57fc38e

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: allow branches with _

880834e

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: standardize stream resource names

6d5fa6d

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add hello world stream proc lambda

409b58e

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add watch target in makefile

afbac00

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: make product models independent

8d7dc71

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: remove stream specific schema for now

1821ef9

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore(domain): create skeleton to notify updates

956a2e1

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add placeholder code for handler

657bdb8

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add placeholder unit test

a622cf2

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: rename stream processor to align w/ crud

5cb8b49

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add format-fix, fix yapf errors

68788d0

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

fix(tests): stack not found when running integ

ba92b7f

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa marked this pull request as draft September 30, 2023 19:19

chore: align handler and fn handler name w/ crud

e7b59b7

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa force-pushed the stream_processor branch from 792b6a6 to e7b59b7 Compare September 30, 2023 19:25

heitorlessa had a problem deploying to dev October 1, 2023 05:20 — with GitHub Actions Failure

heitorlessa added 2 commits October 1, 2023 18:02

chore: add mypy boto3 events dev dep

11ab329

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

feat: add initial DAL protocol and eventbridge

6b85eb7

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa had a problem deploying to dev October 1, 2023 16:15 — with GitHub Actions Failure

heitorlessa added 7 commits October 1, 2023 20:28

refactor: use status field over change_status since it's a notificati…

6c746bf

…on already Signed-off-by: heitorlessa <lessa@amazon.co.uk>

refactor: move test doubles and fixtures to conftest

689e09e

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: test product_notifications

013b608

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add EventReceipt output model

c340352

chore: ignore .idea dir

1dd1fb0

chore: create and use Event model

c38a33c

chore: use generic container for emit

d2bd0b7

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa added 3 commits October 4, 2023 14:28

chore: explicit type as mypy can't infer

c6e8c48

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: actually apply pydantic plugin; skip pytest_socket missing py.…

4a0aaef

…typed Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: make pr fixes

cd157e9

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa had a problem deploying to dev October 4, 2023 12:38 — with GitHub Actions Failure

refactor: rename dal to integrations

fd003d7

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa marked this pull request as ready for review October 4, 2023 12:43

heitorlessa had a problem deploying to dev October 4, 2023 12:54 — with GitHub Actions Error

heitorlessa added 7 commits October 4, 2023 15:46

docs(domain): add initial docstrings

1841f95

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

chore: add mkdocs and mkdocstrings for documentation

9fcf867

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

docs(handlers): add docstring for process_stream

9c17f07

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

docs(domain): use markdown to create anchor

e1f6476

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

docs: increase indentation to improve nav

be733bc

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

docs(handler): add integrations section

6cc898c

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

docs(domain): add integration section

bddc7b4

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

heitorlessa added 2 commits October 4, 2023 16:44

docs(models): add Pydantic models

0d5728b

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

docs(stream_processor): handlers first

5ce39a3

Signed-off-by: heitorlessa <lessa@amazon.co.uk>

ran-isenberg approved these changes Oct 4, 2023

View reviewed changes

ran-isenberg merged commit 576c669 into ran-isenberg:main Oct 4, 2023
3 of 4 checks passed

This was referenced Oct 4, 2023

docs(stream_processor): complete docstrings #81

Merged

refactor(events): rename ABC to Base*, and keep a single EventHandler #83

Merged

refactor: inject event handler in domain logic rather than init #85

Merged

heitorlessa had a problem deploying to dev November 3, 2023 14:54 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(stream_processor): initial version for processing product changes #76

feat(stream_processor): initial version for processing product changes #76

heitorlessa commented Sep 30, 2023 •

edited

Loading

heitorlessa commented Oct 4, 2023 •

edited

Loading

heitorlessa commented Oct 4, 2023

ran-isenberg commented Oct 4, 2023

ran-isenberg commented Oct 4, 2023

heitorlessa commented Oct 4, 2023 •

edited

Loading

ran-isenberg commented Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

heitorlessa Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg Oct 4, 2023

ran-isenberg commented Oct 4, 2023

		@@ -55,6 +55,12 @@ types-requests = "*"
		toml = "*"


		[tool.poetry.group.dev.dependencies]

		_pascal_to_snake_pattern = re.compile(rf'({_exclude_underscores}{_pascal_case}{_or}{_followed_by_lower_case_or_digit}')


		def convert_model_to_event_name(model_name: str) -> str:

		from product.models.products.product import ProductChangeNotification


		def generate_dynamodb_stream_events(product_id: str = '8c18c85a-0f10-4b73-b54a-07ab0d381018',) -> dict[str, Any]:

feat(stream_processor): initial version for processing product changes #76

feat(stream_processor): initial version for processing product changes #76

Conversation

heitorlessa commented Sep 30, 2023 • edited Loading

Summary

User experience

Checklist

Acknowledgment

heitorlessa commented Oct 4, 2023 • edited Loading

heitorlessa commented Oct 4, 2023

ran-isenberg commented Oct 4, 2023

ran-isenberg commented Oct 4, 2023

heitorlessa commented Oct 4, 2023 • edited Loading

ran-isenberg commented Oct 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ran-isenberg commented Oct 4, 2023

heitorlessa commented Sep 30, 2023 •

edited

Loading

heitorlessa commented Oct 4, 2023 •

edited

Loading

heitorlessa commented Oct 4, 2023 •

edited

Loading