Let kafka-headless service resolve even before pods are ready #56

[scholzj]

No description provided.

[tombentley]

Since which version of k8s is this available?

[scholzj]

I don't know ... it works at least from Kubernetes 1.6.

[scholzj]

@tombentley It was added in Kubernetes 1.3.

[matzew]

I did use the stateful-set template, like:

oc new-project someproject

oc new-app -f https://raw.githubusercontent.com/scholzj/barnabas/1b893e8ccaa8a2bea868bc6bbf1863898f3a86a3/kafka-statefulsets/resources/openshift-template.yaml -n someproject

It does provision three nodes each (zk and kafka).

Now I did the following, locally (w/ console scripts from kafka_2.12-0.11.0.0)

➜  bin oc get services 

NAME                 CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka                172.30.210.122   <none>        9092/TCP                     20m
kafka-headless       None             <none>        9092/TCP                     20m
zookeeper            172.30.13.87     <none>        2181/TCP                     20m
zookeeper-headless   None             <none>        2181/TCP,2888/TCP,3888/TCP   20m

Using the zk and kafka IP addresses for some CLI fu, like:

➜  bin ./kafka-topics.sh --create --zookeeper 172.30.13.87:2181 --replication-factor 1 --partitions 1 --topic test

Created topic "test".
➜  bin ./kafka-console-producer.sh --broker-list 172.30.210.122:9092 --topic test
>HELLO
>[2017-10-17 13:46:56,278] ERROR Error when sending message to topic test with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 10053 ms has passed since batch creation plus linger time

[scholzj]

@matzew What is the timing for the operations? Did you wait after oc new-app for all the components to get up and running? Or did you just run it command after command. The way the open shift template works right now is that it creates all resources and is done. But inside OpenShift / Kubernetes things take more time. It starts creating the Zookeeper pods one by one. At the same time it starts creating the Kafka pods. But the Kafka pods will basically crash once or twice because the Zookeeper will be not yet ready. So the question is, did you waited long enough for all pods to be up and running? It this issue something what is fixed over time?

Another point which I'm curious about - where does the CLI where you run the kafka-topics.sh and kafka-console-producer.sh run? Do you connect from outside of OpenShift or from inside OpenShift?

[matzew]

@scholzj yeah, I've waited until all are up (three nodes ZK, and three for Kafka).

I just tried again:

➜  bin ./kafka-topics.sh --create --zookeeper 172.30.13.87:2181 --replication-factor 1 --partitions 1 --topic testy

Created topic "testy".
➜  bin ./kafka-console-producer.sh --broker-list 172.30.210.122:9092 --topic testy                                 
>HELLO
>[2017-10-17 14:11:24,973] ERROR Error when sending message to topic testy with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for testy-0: 10050 ms has passed since batch creation plus linger time

new topic / "old" problem.

Do you connect from outside of OpenShift or from inside OpenShift?

From outside, from the shell on my Fedora26 notebook

[scholzj]

@matzew Hmm, ok. I guess that is related to #50 - it now uses DNS names as advertised hostnames and you will probably not have them resolved outside of OpenShift. The error doesn't really say anything, but this is what normally prints in this situation. I have to think about the best way to deal with this.

[matzew]

Back in September, this was all working fine :)

[scholzj]

I guess we can just rollback some of the changes from #50 to get you up and running. We should get back to #50 once we have better solution for external access.

[matzew]

@scholzj I am happy to test any PR :)

[scholzj]

After rolling back #50 this is not needed anymore and can be closed.