Benchmarks: create new topics for producers tests #917
Conversation
There was a problem hiding this comment.
Not sure how that would affect the benchmarks, but let's see what happens. I think it's a good practice to always use new topics anyway, because topic deletions are asynchronous. Unless we block and poll until a topic no longer exists anymore, it's possible to have a race condition where we begin a test against a topic that's not deleted yet, it gets deleted, and then recreated with automatic topic creation. That would be bad because the automatically created topic would have the default settings for replicas, partitions, and topic-level config.
| runPerfTest(cmd, KafkaProducerFixtures.initializedProducer(cmd), KafkaProducerBenchmarks.plainFlow) | ||
| } | ||
|
|
||
| it should "bench with normal messages written to 8 partitions" in { | ||
| val cmd = | ||
| RunTestCommand(prefix + "-normal-msg-8-partitions", bootstrapServers, topic_2000_5000_8) | ||
| RunTestCommand(prefix + "-normal-msg-8-partitions", bootstrapServers, topic_2000_5000_8.freshTopic) |
There was a problem hiding this comment.
I don't see what changed here between #902 and here. If it was reusing the same topic, it was doing that before as well. The new benchmark for 500b messages would have had its own random topic id.
|
I agree this is weird. It didn't have the effect I hoped for, but suddenly the Plain Consumer hopped back to where it was earlier?! https://jenkins.akka.io:8498/job/alpakka-kafka-perf-master/plot/#1446559666.csv |
Purpose
Try if topic name reuse causes the benchmarks to show the unexpected drop in Raw Kafka performance since build 550.
References
This drop must be connected to #902
Changes
Use a fresh topic name for producer benchmarks so they do not interfere with consumer benchmarks.