Configure SASL_SSL using keycloak and give custom ssl certificates

[shobishani]

Hi,
I've successfully configured SASL_PLAINTEXT using an internal listener with keycloak and it works fine, But when I try to configure it using SASL_SSL with keycloak and giving my custom SSL certificates it does run but I'm unable to connect to the cluster using kafka-python
I got this error ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1125).
I'm using nginx ingress controller and I've enabled TLS passthrough
Here is my helm chart deployment in --dry-run mode.

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  labels:
    helm.sh/chart: kafka-0.1.0
    app.kubernetes.io/name: kafka
    app.kubernetes.io/instance: kafka
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  kafka:
    version: 2.5.0
    replicas: 1
    listeners:
    - name: external
      port: 9094
      type: ingress
      tls: true
      authentication:
        type: oauth
        validIssuerUri: https://keycloak.fgasp.com/auth/realms/kafka-authz
        jwksEndpointUri: https://keycloak.fgasp.com/auth/realms/kafka-authz/protocol/openid-connect/certs
        userNameClaim: preferred_username
        clientId: kafka
        clientSecret:
          secretName: kafka-client-secret
          key: client-secret
        maxSecondsWithoutReauthentication: 3600
      configuration:
        brokerCertChainAndKey:
          secretName: kafka-custom-tls
          certificate: fgasp-cert.crt
          key: fgasp-key.key
        bootstrap:
          host: kafka.fgasp.com
        brokers:
          - broker: 0
            host: broker-0.kafka.fgasp.com
    authorization:
      type: keycloak
      tokenEndpointUri: https://keycloak.fgasp.com/auth/realms/kafka-authz/protocol/openid-connect/token
      delegateToKafkaAcls: true
      clientId: kafka
      disableTlsHostnameVerification: true
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 5Gi
        deleteClaim: false
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.5"
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
  entityOperator:
    topicOperator: {}
    userOperator: {}

And here is how I generated certificates maybe these are the wrong certificates?
OpenSSL config file for SAN for kafka cluster as per docs:

#
# fgasp.com.cnf
#

[ req ]
prompt = no
distinguished_name = server_distinguished_name
req_extensions = v3_req

[ server_distinguished_name ]
commonName = *.fgasp.com
stateOrProvinceName = PUN
countryName = PK
emailAddress = shababqaisar2@gmail.com
organizationName = Fiverivers
organizationalUnitName = Development

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
DNS.0 = *.fgasp.com
DNS.1 = *.kafka-kafka-brokers
DNS.2 = *.kafka-kafka-brokers.kafka-cluster.svc
DNS.3 = kafka-kafka-bootstrap
DNS.4 = kafka-kafka-bootstrap.kafka-cluster.svc

Certificate authority:
openssl genrsa  -out CA-key.pem 2048
openssl req -new -key CA-key.pem -x509 -days 1000 -out CA-cert.pem

Self signed certificates:
openssl genrsa -out server-key.pem 2048
openssl req –new –config fgasp.com.cnf –key server-key.pem –out signingReq.csr
openssl x509 -req -days 365 -in signingReq.csr -CA CA-cert.pem -CAkey CA-key.pem -CAcreateserial -out server-cert.pem

Then I renamed the certificates to create kubernetes secret

mv server-key.pem fgasp-key.key
mv server-cert.pem fgasp-cert.crt

Then to created secret:
kubectl create secret generic kafka-custom-tls --from-file=fgasp-key.key --from-file=fgasp-cert.crt -n kafka-cluster

after creating secret I deployed kafka and here is how I'm connecting to the kafka cluster using these certs

import logging
from kafka import KafkaProducer
from kafka.oauth import AbstractTokenProvider

logging.basicConfig(level=logging.DEBUG)

host = "kafka.fgasp.com:443"

class OAuthProvider(AbstractTokenProvider):
    def token(self):
        return ...fetched access token from keycloak...

producer = KafkaProducer(
    bootstrap_servers=host,
    ssl_cafile='CA-cert.pem',
    ssl_certfile='fgasp-cert.crt',
    ssl_keyfile='fgasp-key.key',
    sasl_mechanism="OAUTHBEARER",
    security_protocol='SASL_SSL',
    sasl_oauth_token_provider=OAuthProvider()
)

Here are some logs from pod/kafka-kafka-0 and that's all I get whenever I try to connect.

2021-02-28 20:21:23,996 INFO [SocketServer brokerId=0] Failed authentication with 10.104.1.5/10.104.1.5 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SASL_SSL-4]

So I'm not sure If I'm trying to connect using incorrect certificates or perhaps I created incorrect certificates or the deployment is incorrect, I've gone through the documentation multiple times and I'm not sure what is it that I'm doing wrong.

Please tell me if something else is required to help me.
Thanks

[scholzj]

It is a long time since I used OpenSSL ... it looks roughly correct, but it is possible I missed something there. The Kafka CR looks fine to me. Java has a nice SSL debugging feature, but I'm not sure if something similar exists in Python to see what exactly the handshake issue is. Maybe doing openssl s_client might help as well to see if you get the right server certificate from the server.

[shobishani]

Thanks.
A quick question though have you used cert-manager with let's encrypt to generate certificates for strimzi deployment using ingress?

[scholzj]

I did, I have a work in progress blog post about it.

[scholzj]

Or, well, not specifically with Ingress ... but the Ingress is TLS passthrough ... so you need to anyway configure the certificates in the Kafka CR and not in Ingress. So it should not differ from using it with loadbalancers for example.

[shobishani]

Great. Please do share the blog post So I can follow up on that or is there some link I can subscribe to?

[scholzj]

There is an RSS feed: https://strimzi.io/feed.xml ... we also always tweet about the blgo posts (https://twitter.com/strimziio) and share them on Slack.