Release/0.7.0 #80
Merged
Release/0.7.0 #80
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
common-streams has a feature in which a timed window of events is allowed to start processing before the previous window has finalized. This is a great feature for making full use of available cpu. It means are are always working the cpu hard, even if some slow I/O is required to finalize the window. Until now, the eagerness only stretched to consecutive windows. E.g. If window 1 is still finalizing then it is allowed for window 2 to start processing; but it is not allowed for window 3 to start processing. For Lake Loader I found it is better to let the eagerness stretch further. E.g. window 3 is allowed to start processing even if windows 1 and 2 are both still finalizing. I also found it is better to allow consecutive windows to be finalizing at the same time. E.g. window 2 can start its finalization even if window 1 is still finishing its finalization. This PR makes configurable how many windows may start eagerly ahead of a finalzing window.
common-streams has a feature where the first window is randomly adjusted to a different size. This helps in a deployment where a large number of pods might all start at the same time, but we pods to have mutually staggered windows, e.g. to avoid write conflicts in the lake loader. Unfortunately I broke the feature in #64 so this fixes it again. I also add a test so it won't get broken again.
The PubSub Source parameter `parallelPullCount` was used to set the parallelism of the underlying Subscriber. With a higher pull count, the Subscriber can supply events more quickly to the downstream of the application, but there is more overhead. For typical Snowplow apps, a pull count of 1 is sufficient on small instances. But when there is more cpu availalbe, the downstream app processes events more quickly, and therefore we need a higher pull count to provide the events. This PR makes it so pull count is picked dynamically based on available cpu. Snowplow apps on bigger instances will automatically get the benefit of this change, without requiring the user to explicitly set the pull count.
The KCL is sadly not very good at crashing and exiting. If the underlying Kinesis client has errors (e.g. permissions errors) then KCL tends to stay alive and not propagate the exceptions to our application code. We want the app to crash under these circumstances because that triggers an alert. common-streams already has a health check feature, in which a health probe becomes unhealthy if a single event gets stuck without making progress. This PR leans on the existing health check feature, so it also becomes unhealthy if the Kinesis client is not regularly receiving healthy responses. I configured KCL to invoke our record processor every time it polls for records, even if the batch is empty. This means the health check still works even if there are no events in the stream.
pondzix
approved these changes
May 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.