Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(batch): Batch processing wrapper function #1605

Merged
merged 4 commits into from
Jul 13, 2023
Merged

feat(batch): Batch processing wrapper function #1605

merged 4 commits into from
Jul 13, 2023

Conversation

erikayao93
Copy link
Contributor

Description of your changes

  • Renamed itemIdentifier and batchResponse types to match Python naming
  • Renamed testing factories to be more accurate to their function
  • Added processPartialResponse() function that acts as a function wrapper for batch processing
  • Added unit tests for processPartialResponse() function
  • Added some TODO comments for future async/sync function work

Related issues, RFCs

Issue number: #1602

Checklist

  • My changes meet the tenets criteria
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my change is effective and works
  • The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

  • I have documented the migration process
  • I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@erikayao93 erikayao93 requested a review from a team as a code owner July 12, 2023 22:18
@boring-cyborg boring-cyborg bot added batch This item relates to the Batch Processing Utility tests PRs that add or change tests labels Jul 12, 2023
@pull-request-size pull-request-size bot added the size/L PRs between 100-499 LOC label Jul 12, 2023
@github-actions github-actions bot added the feature PRs that introduce new features or minor changes label Jul 13, 2023
@am29d am29d requested review from am29d and removed request for a team July 13, 2023 11:35
am29d
am29d reviewed Jul 13, 2023
View reviewed changes
Copy link
Contributor

@am29d am29d left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @erikayao93 , looks great! I left just one comment, otherwise we can merge it.

packages/batch/src/processPartialResponse.ts Outdated Show resolved Hide resolved
@erikayao93
Copy link
Contributor Author

Hey @am29d, thanks for the review on this!

Just a question, I have a /* eslint-disable @typescript-eslint/no-explicit-any */ line at the top of my test file, is there any alternative to doing this? I did try // eslint-disable-next-line @typescript-eslint/no-explicit-any right before it, which wasn't working.

If it's okay to leave it is as well, I'm fine with that, I just didn't see that used for a whole file in anything else in the repo so I wanted to double check.

dreamorosi
dreamorosi reviewed Jul 13, 2023
View reviewed changes
Copy link
Contributor

@dreamorosi dreamorosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left two suggestions, if you accept them you should be able to remove the eslint-disable from the file.

packages/batch/tests/unit/processPartialResponse.test.ts Outdated Show resolved Hide resolved
packages/batch/tests/unit/processPartialResponse.test.ts Outdated Show resolved Hide resolved
@dreamorosi dreamorosi linked an issue Jul 13, 2023 that may be closed by this pull request
Feature request: implement processPartialResponse wrapper function #1602
Closed
@dreamorosi dreamorosi self-requested a review July 13, 2023 19:26
dreamorosi
dreamorosi approved these changes Jul 13, 2023
View reviewed changes
Copy link
Contributor

@dreamorosi dreamorosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and for the quick turnaround!

@dreamorosi dreamorosi merged commit cf499e1 into aws-powertools:feat/batch Jul 13, 2023
dreamorosi pushed a commit that referenced this pull request Jul 15, 2023
@erikayao93 @dreamorosi
feat(batch): Batch processing wrapper function (#1605)
a68db2c 
* Refactored some types, added function wrapper and base test

* Added record check and tests, renamed factories

* Refactored type check logic in function

* Refactor test to remove error ignore
dreamorosi pushed a commit that referenced this pull request Jul 15, 2023
@erikayao93 @dreamorosi
feat(batch): Batch processing wrapper function (#1605)
7b9bdee 
* Refactored some types, added function wrapper and base test

* Added record check and tests, renamed factories

* Refactored type check logic in function

* Refactor test to remove error ignore
dreamorosi added a commit that referenced this pull request Jul 25, 2023
@dreamorosi @erikayao93
@dependabot @github-actions
feat(batch): add batch processing utility (#1625)
c4e6b19 
* chore: init workspace

* chore: init workspace

* feat(batch): Implementation of base batch processing classes (#1588)

* chore: init workspace

* chore: init workspace

* Initial base class implementation

* Added BatchProcessor implementation, attempted fix for async

* Added unit tests

* Refactoring unit tests

* Lint fix, updated docstrings

* Added response and identifier typings

* test(idempotency): improve integration tests for utility (#1591)

* docs: new name

* chore: rename e2e files

* tests(idempotency): expand integration tests

* chore(idempotency): remove unreachable code

* Removed unnecessary type casting

* Moved exports for handlers and factories

* Updated imports, refactored randomization in factories

* Refactored EventType to be const instead of enum

* Refactored and added documentation for errors

* Removed debugging line

* chore(ci): add canary to layer deployment (#1593)

* docs(idempotency): write utility docs (#1592)

* docs: base docs

* wip

* chore: added paths to snippets tsconfig

* chore: added page to docs menu

* docs(idempotency): utility docs

* highlights

* chore: remove CDK mention

* build(internal): bump semver from 5.7.1 to 5.7.2 (#1594)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(idempotency): mark the utility ready public beta (#1595)

* chore(idempotency): mark utility as public beta

* chore: manually increment version in commons

* docs(internal): update AWS SDK links to new docs (#1597)

* chore(maintenance): remove parameters utility from layer bundling and layers e2e tests (#1599)

* remove parameter from e2e tests

* remove parameters from canary stack as well

* chore(release): v1.11.1 [skip ci]

* fix canary deploy in ci with correct workspace name (#1601)

* chore: update layer ARN on documentation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Andrea Amorosi <dreamorosi@gmail.com>
Co-authored-by: Alexander Schueren <amelnyk@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com>

* feat(batch): Batch processing wrapper function (#1605)

* Refactored some types, added function wrapper and base test

* Added record check and tests, renamed factories

* Refactored type check logic in function

* Refactor test to remove error ignore

* feat(batch): Implement SQS FIFO processor class (#1606)

* Added SQS FIFO processor and unit tests

* Added docstring for pbatch processing function

* feat(batch): Support for Lambda context access in batch processing (#1609)

* Added types and parameter for lambda context, added unit tests

* Refactor parameter checking

* Added test for malformed context handling

* docs: created utility docs

* docs: fixed white spaces

* feat(batch): add async processor (#1616)

* feat(batch): add async processor

* tests: improved unit tests

* chore: removed docstring + edited test handler

* chore: fix typos

* docs: added README

* chore: added package to beta release

* chore: marked package as public

* chore: added new batch page to docs

* chore: added utility to lerna workspace

* chore: added utility to main readme

* chore: added utility to CI

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Erika Yao <71943596+erikayao93@users.noreply.github.com>
Co-authored-by: Alexander Schueren <amelnyk@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@am29d am29d am29d left review comments

@dreamorosi dreamorosi dreamorosi approved these changes

Assignees

@erikayao93 erikayao93

Labels
batch This item relates to the Batch Processing Utility feature PRs that introduce new features or minor changes size/L PRs between 100-499 LOC tests PRs that add or change tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature request: implement processPartialResponse wrapper function
3 participants
@erikayao93 @am29d @dreamorosi