rd_kafka_topic_info_new_with_rack insufficient buffer allocation

[wolfchimneyrock]

Description

When running unit tests for a consumer integration framework where we create a mock producer with some messages and then subscribing to topics and consuming the messages, we run into an assertion thrown by rdkafka_buf.h that a rd_tmpabuf does not have enough space allocated.

My suspicion is that the new KIP-881 code in 2.2.0 is triggering this new method

rd_kafka_topic_info_new_with_rack()

which allocates a buffer respecting 8 byte padding for everything except the initial struct added first - but rdkafka_buf keeps track of its own internal offset rounding up to 8 byte padding even for the first added entry.

setting the client.id="" doesn't trigger the new code, and all of our test cases pass then.

I can get the tests to run by patching rdkafka_topic.c - rd_kafka_topic_info_new_with_rack() with

rd_tmpabuf_new(&tbuf, RD_ROUNDUP(sizeof(*ti), 8) + tlen + total_rack_size, 1);

instead of

rd_tmpabuf_new(&tbuf, sizeof(*ti) + tlen + total_rack_size, 1);

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 2.2.0
• Apache Kafka version: N/A (unit test)
• librdkafka client configuration:
  auto.offset.reset=beginning
  bootstrap.servers=rd_kafka_mock_cluster_bootstraps()
  group.id=<Name of GTEST test suite + test>
  client.id=<Name of GTEST test suite + test>
• Operating system: RHEL 7.9
• Provide logs (with debug=.. as necessary) from librdkafka

6: %5|1692291244.204|CONFWARN|mock#producer-1| [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster
6: rd_tmpabuf_alloc0: rd_kafka_topic_info_new_with_rack:1860: requested size 16 + 61 > 76
6: xxxx_unit_tests: rdkafka_buf.h:102: rd_tmpabuf_alloc0: Assertion `!*"rd_tmpabuf_alloc: not enough size in buffer"' failed.

• Provide broker log excerpts
  N/A (unit test)
• Critical issue

[emasab]

@wolfchimneyrock Thanks for the contribution! We'll review your PR. Did the fault happen on a 32bit architecture? It seems given aligned struct size is 12 (rounded to 16).

[wolfchimneyrock]

this assert failed when running test on our integration library that builds for both 32 bit and 64 bit for intel, solaris, and aix. I can confirm that the assert failure occurs during x86 32-bit unit tests, and succeeds in all other cases.

[wolfchimneyrock]

I can also confirm that even if I build 32bit librdkafka 2.2.0 with -malign-double GCC flag, the assertion still fails.