Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow InitialPositionInStreamExtended to be specified in properties file #804

Merged
merged 3 commits into from
Jun 1, 2021

Conversation

kevioke
Copy link
Contributor

@kevioke kevioke commented Apr 5, 2021

*Issue #, if available: #803

*Description of changes: This change allows a user of the multilang daemon to specify an initial timestamp in which the daemon will process records. E.g. if a user specifies initialPositionInStreamExtended = 1617305352 in the properties file AND the stream does not currently have checkpoints present, the stream will start processing at Thursday, April 1, 2021 7:29:12 PM GMT

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

E.g. initialPositionInStreamExtended = 1617305352
Copy link

@isurues isurues left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified that "InitialPositionInStream.AT_TIMESTAMP" is not currently supported in KCL 2.0 multilang. However, this is supported in KCL 1.x. Thanks for your contribution. Left couple of comments in the PR.

@@ -93,6 +94,16 @@ public void testWithLongVariables() {
assertEquals(config.getShardSyncIntervalMillis(), 500);
}

@Test
public void testWithInitialPositionInStreamExtended() {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like similar tests are in KCL 1.0, but not in 2.0. In addition to this one, can you please add the following test case (testKCLConfigurationWithInvalidInitialPositionInStream) here? You will have to make "config.getInitialPositionInStreamExtended()" public. But that's fine.

https://github.com/awslabs/amazon-kinesis-client/blob/v1.x/src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfigurationTest.java#L319

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! Definitely good to test the exception cases as well.

I've updated the tests. I tried to copy the essence of the unit tests you mentioned over, but didn't exactly copy everything word for word since the 1.0 tests seem to be testing the KinesisClientLibConfiguration instead of the MultiLangDaemonConfiguration.

So to recap, I believe the unit tests in this test file should now cover these scenarios:

  1. When InitialPositionInStreamExtended is configured with a timestamp, it sets the AT_TIMESTAMP as well as time in the config.getInitialPositionInStreamExtended(). config.getInitialPositionInStreamExtended() seems to already be public or at least package private for MultiLangDaemonConfiguration. I don't explicitly see the declaration but I'm assuming it's generated via lomboc .
  2. Exceptions should be thrown when InitialPositionInStreamExtended is not a long value
  3. Exceptions should be thrown when InitialPositionInStream is set to AT_TIMESTAMP
  4. TRIM_HORIZON is a valid value for initialPositionInStream
  5. LATEST is a valid value for initialPositionInStream

@kevioke kevioke requested a review from isurues May 10, 2021 21:10
@AvinashChowdary AvinashChowdary merged commit 4e9e86e into awslabs:master Jun 1, 2021
@AvinashChowdary AvinashChowdary added this to the v2.3.5 milestone Jun 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants