Skip to content
This repository has been archived by the owner on Mar 22, 2024. It is now read-only.

Switch AWS Kinesis example to new Beam API #37

Merged
merged 2 commits into from
Apr 17, 2023

Conversation

psolomin
Copy link
Contributor

@psolomin psolomin commented Apr 15, 2023

Old Kinesis connector is deprecated by apache/beam#22093

How this was tested

Run with real Kinesis:

$ cd data-ingestion

$ mvn clean package

$ mvn exec:java -Dexec.mainClass=org.apache.beam.samples.data.ingestion.DataToKinesis -Pdirect-runner -Dexec.args="--runner=DirectRunner --stream=stream-01" 

Old Kinesis connector is deprecated by apache/beam#22093
@psolomin
Copy link
Contributor Author

psolomin commented Apr 17, 2023

@aromanenko-dev it looks like the issue is caused by kinesis client lib:

            $ mvn clean dependency:tree -pl data-ingestion
            ...
            [INFO] +- org.apache.beam:beam-sdks-java-io-amazon-web-services2:jar:2.47.0-SNAPSHOT:compile
            ...
            [INFO] |  +- software.amazon.kinesis:amazon-kinesis-client:jar:2.3.4:compile
            [INFO] |  |  +- software.amazon.glue:schema-registry-serde:jar:1.0.0:compile
            [INFO] |  |  |  +- org.projectlombok:lombok:jar:1.18.2:compile
            [INFO] |  |  |  \- org.projectlombok:lombok-utils:jar:1.18.12:compile
            ...

I've pushed a dependency overwrite in this PR and created issue for Beam to bump amazon-kinesis-client version:

apache/beam#26295

Copy link
Contributor

@aromanenko-dev aromanenko-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks a lot @psolomin for this change! Tbh, I completely forgot about that.
LGTM

@aromanenko-dev aromanenko-dev merged commit aa3a520 into Talend:master Apr 17, 2023
@psolomin psolomin deleted the switch-aws-examples-to-new-api branch April 17, 2023 15:14
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants