[Bug]: Zookeeper can fail to bind port 3888 and 2888 because it cannot resolve its own full hostname.

[avb-systematic]

Bug Description

If the pod only has a short hostname (i.e. no domain), hostname -f will resolve according to the DNS search domains.
If these search domains have not been set yet (as these are created/updated) by Istio, the domain name will be set to /<unresolved>

Then the log will fill with errors like

2024-12-16T03:44:38.969474713+01:00 stdout F {"exception":{"stacktrace":"java.net.SocketException: Unresolved address
at java.base/java.net.ServerSocket.bind(ServerSocket.java:380)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:342)
at org.apache.zookeeper.server.quorum.Leader.createServerSocket(Leader.java:326)
at org.apache.zookeeper.server.quorum.Leader.lambda$new$0(Leader.java:301)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.apache.zookeeper.server.quorum.Leader.<init>(Leader.java:304)
at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1340)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1551)",
"exception_class":"java.net.SocketException",
"exception_message":"Unresolved address"},
"source_host":"project-zookeeper-0",
"method":"createServerSocket",
"level":"ERROR",
"message":"Couldn't bind to project-zookeeper-0./<unresolved>:2888",
"mdc":{"myid":"1"},
"@timestamp":"2024-12-16T02:44:38.969Z",
"file":"Leader.java",
"line_number":"329",
"thread_name":"QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)",
"@version":1,
"logger_name":"org.apache.zookeeper.server.quorum.Leader",
"class":"org.apache.zookeeper.server.quorum.Leader"}

Steps to reproduce

1. Ensure that the zookeeper pod does not have search domains defined
2. Ensure that the zookeeper pod has an Unqualified hostname
3. Start the pod

Expected behavior

Zookeeper should fail to startup, or wait for the DNS system to come online.

It should NOT think that my domain name is /<unresolved>

Strimzi version

0.39.0

Kubernetes version

1.29.6+rke2r1

Installation method

Helm chart

Infrastructure

No response

Configuration files and logs

No response

Additional context

We fixed the problem by adding an ENTRYPOINT to the strimzi-zookeeper image calling waitForDNS.sh

waitForDNS.sh

#!/usr/bin/env sh
set -e

echo "Starting up $1"
# If it should run a check on DNS name resolving before starting ZooKeeper
# This is useful in environment (i.e. minikube) where DNS registration is slow
# Related to https://github.com/strimzi/strimzi-kafka-operator/issues/8675
# It is also related to https://issues.apache.org/jira/browse/ZOOKEEPER-4708

# This code is inspired by https://github.com/strimzi/strimzi-kafka-operator/pull/8702
# which fixes the case where hostname is correct, but DNS is not ready to resolve it

# This fixes the case where the FQDN hostname cannot be resolved
# (as the hostname is short and qualified from /etc/resolv.conf search domains)
# This has been reported as https://github.com/strimzi/strimzi-kafka-operator/issues/10977
while ! nslookup "$(hostname -f)"; do
  sleep 1
  echo "Trying to resolve $(hostname -f) ..."
done

exec "$1"

[avb-systematic]

We also expanded zookeeper_healthcheck.sh to actually check if the 3888 port was opened

#!/usr/bin/env bash
set -e
# Adapted from https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.41.x/docker-images/kafka-based/kafka/scripts/zookeeper_healthcheck.sh

OK=$(echo ruok | nc 127.0.0.1 12181)
if [ "$OK" != "imok" ]; then
   exit 1
fi

# Check if port 3888 is open
# We cannot use netcat as we only open on localhost when using a single zookeeper and
# only open on (hostname -f) when using multiple servers
if ! netstat -l -t | grep -q -F ":3888 " && echo true ; then
    exit 1
fi

[avb-systematic]

The start of the log is

2024-12-16T02:44:57.358948843+01:00 stderr F hostname: project-zookeeper-0: Try again
2024-12-16T02:44:57.36069918+01:00 stdout F Detected Zookeeper ID 1
2024-12-16T02:44:57.365218946+01:00 stdout F Preparing truststore
2024-12-16T02:44:57.36666561+01:00 stdout F Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
2024-12-16T02:44:57.701470128+01:00 stderr F Certificate was added to keystore
2024-12-16T02:44:58.111436229+01:00 stdout F Preparing truststore is complete
2024-12-16T02:44:58.111495982+01:00 stdout F Looking for the right CA
2024-12-16T02:44:58.121848343+01:00 stdout F Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
2024-12-16T02:44:58.12186863+01:00 stdout F Preparing keystore for client and quorum listeners
2024-12-16T02:44:58.130944062+01:00 stdout F Preparing keystore for client and quorum listeners is complete
2024-12-16T02:44:58.131086831+01:00 stdout F Starting Zookeeper with configuration:
2024-12-16T02:44:58.133463112+01:00 stdout F # The directory where the snapshot is stored.
2024-12-16T02:44:58.133743743+01:00 stdout F dataDir=/var/lib/zookeeper/data
2024-12-16T02:44:58.133747585+01:00 stdout F 
2024-12-16T02:44:58.133749752+01:00 stdout F # Other options
2024-12-16T02:44:58.133752179+01:00 stdout F 4lw.commands.whitelist=*
2024-12-16T02:44:58.133754239+01:00 stdout F standaloneEnabled=false
2024-12-16T02:44:58.133756358+01:00 stdout F reconfigEnabled=true
2024-12-16T02:44:58.133758061+01:00 stdout F 
2024-12-16T02:44:58.133760188+01:00 stdout F # TLS options
2024-12-16T02:44:58.133762757+01:00 stdout F serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
2024-12-16T02:44:58.133764624+01:00 stdout F ssl.clientAuth=need
2024-12-16T02:44:58.133766492+01:00 stdout F ssl.quorum.clientAuth=need
2024-12-16T02:44:58.133768399+01:00 stdout F secureClientPort=2181
2024-12-16T02:44:58.133770155+01:00 stdout F sslQuorum=true
2024-12-16T02:44:58.133771847+01:00 stdout F 
2024-12-16T02:44:58.133774052+01:00 stdout F ssl.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
2024-12-16T02:44:58.133775823+01:00 stdout F ssl.trustStore.password=[hidden]
2024-12-16T02:44:58.133777678+01:00 stdout F ssl.trustStore.type=PKCS12
2024-12-16T02:44:58.133779561+01:00 stdout F ssl.quorum.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
2024-12-16T02:44:58.133781677+01:00 stdout F ssl.quorum.trustStore.password=[hidden]
2024-12-16T02:44:58.133783816+01:00 stdout F ssl.quorum.trustStore.type=PKCS12
2024-12-16T02:44:58.13378546+01:00 stdout F 
2024-12-16T02:44:58.133787792+01:00 stdout F ssl.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
2024-12-16T02:44:58.133789533+01:00 stdout F ssl.keyStore.password=[hidden]
2024-12-16T02:44:58.133791279+01:00 stdout F ssl.keyStore.type=PKCS12
2024-12-16T02:44:58.133793132+01:00 stdout F ssl.quorum.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
2024-12-16T02:44:58.133794982+01:00 stdout F ssl.quorum.keyStore.password=[hidden]
2024-12-16T02:44:58.133796817+01:00 stdout F ssl.quorum.keyStore.type=PKCS12
2024-12-16T02:44:58.133798465+01:00 stdout F 
2024-12-16T02:44:58.133800338+01:00 stdout F # Provided configuration
2024-12-16T02:44:58.1338022+01:00 stdout F tickTime=2000
2024-12-16T02:44:58.133804051+01:00 stdout F initLimit=5
2024-12-16T02:44:58.133805838+01:00 stdout F syncLimit=2
2024-12-16T02:44:58.133807598+01:00 stdout F autopurge.purgeInterval=1
2024-12-16T02:44:58.133809456+01:00 stdout F admin.enableServer=false
2024-12-16T02:44:58.133811319+01:00 stdout F ssl.quorum.hostnameVerification=false
2024-12-16T02:44:58.133813046+01:00 stdout F 
2024-12-16T02:44:58.13381471+01:00 stdout F 
2024-12-16T02:44:58.13381655+01:00 stdout F # Zookeeper nodes configuration
2024-12-16T02:44:58.133818477+01:00 stdout F server.1=project-zookeeper-0.:2888:3888:participant;127.0.0.1:12181
2024-12-16T02:44:58.133856224+01:00 stdout F 
2024-12-16T02:44:58.195574188+01:00 stdout F Configuring Java heap: -Xms778504765 -Xmx778504765
2024-12-16T02:44:58.19575681+01:00 stderr F + exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/zookeeper-server-start.sh /tmp/zookeeper.properties

[scholzj]

If the pod only has a short hostname (i.e. no domain), hostname -f will resolve according to the DNS search domains.
If these search domains have not been set yet (as these are created/updated) by Istio, the domain name will be set to /

Strimzi does not support Istio. So if you use Istio, you should configure it to leave Strimzi alone an don't interfere with it.

[avb-systematic]

I found the easiest way to reproduce the error

Run the zookeeper pod as root
docker run -it -u root --rm ....strimzi-kafka-alpine:*****

remove the machine entry from /etc/hosts

Run hostname -f and see it fail

/opt/kafka # hostname -f
hostname: f71694e62fda: Host not found
/opt/kafka #

[scholzj]

You realize Strimzi has nothing to do with running in Docker? One also has to wonder what ....strimzi-kafka-alpine:***** is - we do not use Alpine Linux anywhere if that is what you do.

[avb-systematic]

Sorry, I had to hide some company names above. it is

FROM quay.io/strimzi/kafka:0.41.0-0-kafka-3.6.2 AS strimzi
FROM alpine:3.19

COPY --from=strimzi /opt/kafka /opt/kafka
RUN apk update \
    && apk add bash openjdk17-jre-headless openssl gawk tini netcat-openbsd \
    && ln -s /sbin/tini /usr/bin/tini \
    && addgroup -g 1000 -S zookeeper \
    && adduser -u 1000 -S zookeeper -G zookeeper \
    && chown -R zookeeper:zookeeper /opt/kafka \
    && chown -R zookeeper:zookeeper /tmp \
    && mkdir -p /var/lib/zookeeper \
    && chown -R zookeeper:zookeeper /var/lib/zookeeper
WORKDIR /opt/kafka

ENV KAFKA_HOME=/opt/kafka
ENV STRIMZI_VERSION=0.39.0

USER 1000

[scholzj]

Sorry. But it seems like you have problems with some sort of Strimzi fork that you use in a very specific ways. You supply your own images with your own operating system, you don't respect the Strimzi versions, you seem to run it in unsupported configurations - not sure whether it is actually Istio or Docker. This is of course fine, Strimzi is an open-source project and you can do anything with it. But you might need to solve your own problems and you should not raise them as Strimzi bugs.

[avb-systematic]

Yes, we will handle this ourselves.
The problem remains in Strimzi, however. If you cannot resolve your own hostname to an IP, zookeeper will fail to get a FQDN.
This situation might be impossible outside of our weird system, but if it arise you WILL see the errors I presented above.

We will build our own images to work around it.

Sorry about bothering you with this.

[scholzj]

I'm still not sure what exactly is the error you are referring to. The Pods are expected to have hostnames defined by the Pod name and the name of the headless service that gives them the name. This is a basic Kubernetes functionality. If you make changes to how the cluster is deployed and this does not apply anymore, you might need to make other changes to compensate for that. It is as simple as that. You cannot customize half of it and then ask the Strimzi community to fix the other half of it for you.