Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use put_records for kinesis due to rate exceeded #92

Merged
merged 1 commit into from
Sep 24, 2024

Conversation

RanbirAulakh
Copy link
Collaborator

@RanbirAulakh RanbirAulakh commented Sep 18, 2024

Issue #, if available: n/a

Notes

This pull request enhances the KinesisSink class by updating the _flush_stream method to improve the handling of records sent to the Kinesis stream. The changes ensure compliance with Kinesis limits, specifically allowing a maximum of 500 records per request, while the total request size does not exceed 5 MB.

Ran the load test for an hour:

Metric Before PR This PR
Total Images Sent 112 115
Total Images In-Progress 14 2
Total Images Processed 98 113
Total Images Succeeded 91 113
Total Images Failed 7 0
Total GB Processed 18.63 30.55
Total Pixels Processed 6,667,674,894 10,933,475,922

The above load test uses a variety of rareplane images of different sizes (ranging from small to large) and runs for one hour, sending an image request every 30 seconds.

Checklist

Before you submit a pull request, please make sure you have the following:

  • Code changes are compact and well-structured to facilitate easy review
  • Changes are documented in the README.md and other relevant documentation pages
  • PR title and description accurately reflect the changes and are detailed enough for historical tracking
  • PR contains tests that cover all new code and the code has been manual tested
  • All new dependencies are declared (if any), and no unnecessary libraries are added
  • Performance impacts (if any) of the changes are evaluated and documented
  • Security implications of the changes (if any) are reviewed and addressed
  • I have read the Contributing Guidelines and agree to follow the Code of Conduct

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Collaborator

@drduhe drduhe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@RanbirAulakh RanbirAulakh merged commit 01fd37a into main Sep 24, 2024
2 checks passed
@RanbirAulakh RanbirAulakh deleted the bug-kinesis-rates branch September 24, 2024 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants