Fix kafka benches to actually bench shotover #1372
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this PR our shotover benches were actually benching directly with kafka completely bypassing shotover!
The KafkaSinkSingle transform is designed to run only on the same machine as the kafka broker.
However the bench currently sets up kafka and shotover on different instances.
One would expect this to result in an error somewhere and the bench to fail.
But what actually happens is that shotover fails to rewrite the metadata query containing the kafka nodes in the cluster and as a result once the bencher receives the metadata query back from shotover it proceeds to send all the benchmark queries directly to kafka.
In more detail:
Shotover will overwrite the port of any brokers returned by the metadata requests.
When shotover and kafka are on different instances they both have ports of 9092.
So we are just overwriting 9092 with 9092, an unintentional noop, and so the client just connects directly to kafka without any error.
This silent failure is really bad and I've raised the following to prevent similar issues from occurring again:
This PR fixes the benchmark by moving shotover and kafka onto the same machine for the kafka benchmarks.
kafka windsock results with this PR: