Use testcontainers for multi-broker integration tests and benchmarks

[seglo]

Purpose

The sbt-docker-compose plugin is abandoned and difficult to manage. The only reason it was used was to support multi-broker Kafka clusters for integration testing. Now that we can setup equivalent clusters using testcontainers there's no need to include this plugin.

References

• Kafka cluster example testcontainers/testcontainers-java#1984
• [WIP] Testkit: start configurable Testcontainer via mixin #936

Changes

• Integration Tests: Set minimum in sync replicas for topics to 2 so produced messages get retried when a broker is unexpectedly shutdown
• Integration Tests: Use takeWhile to signal consumer stream shutdowns.
• Uses testcontainers-java to spin up Kafka clusters. This required changes to KafkaContainer. I created an upstream PR to add this support to testcontainer-java, but I've copied the implementations into our internal testkit DSL in the meantime.
• Add TestcontainersKafkaPerClassLike to set testcontainer lifetime to test class
• Remove sbt-docker-compose and sbt-build-info SBT plugins
• Testcontainers settings KafkaTestkitTestcontainersSettings

TODO

• Enable logs from running containers?
• Compare benchmark test results before and after this change (after merge) No strange difference visible.

[ennru]

This looks really powerful and much more structured than the former solution with lots of moving parts.

With this competent Testcontainers support, we should create a separate documentation page for it with a proper separation of Scala and Java.

An overview table of the possible testing variants seems necessary now.

[ennru]

Great work.
LGTM.
We can follow up with a PR to use Testcontainers for more of our tests so that they run on Scala 2.13 and move some long-running tests to the IntegrationTest scope.