-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Using SASL with librdkafka on Windows
For a trivial zookeeper/kafka ensemble/cluster all running on machine: HOST, perform the following steps to enable SASL via SSPI.
For a Unix-based system (Debian/Ubuntu, RedHat, MacOS/OSX) please follow guide Using SASL with librdkafka.
Kerberos keytabs (file-based pre-authenticated keys) are created for each broker in the cluster. The client will use SSPI and does not need keytab files. They keytabs are distributed to the broker nodes.
Decide on the following things:
REALM - Your Kerberos realm, typically your operational domain in upper case. E.g., YOURDOMAIN.COM. HOST - Broker hostname, E.g., broker1. Quantify as necessary. Any casing works. DOMAIN - The Windows domain of the broker HOST. E.g. LONDON. Any casing works.
Create AD user for Zookeeper/Kafka (the format is AD DOMAIN\AD user name):
• DOMAIN\Zookeeper_test (say the password is zk_password)
• DOMAIN\Kafka_test (say the password is kfk_password)
Ask the AD administrator to run:
SETSPN -S zookeeper/HOST@REALM DOMAIN\Zookeeper_test
SETSPN -S zookeeper/HOST_FQDN@REALM DOMAIN\Zookeeper_test
SETSPN -S kafka/HOST@REALM DOMAIN\Kafka_test
SETSPN -S kafka/HOST_FQDN@REALM DOMAIN\Kafka_test
https://technet.microsoft.com/en-us/library/cc753771(v=ws.11).aspx
You will need the respective AD user password from step 1; the SPN from step 2; and a folder for the generated output (you can call this folder C:\keytabs).
We will create keytab data for both the simple and fully qualified DOMAIN name of the HOST.
Note that the ktpass utility might not be available on all versions of Windows:
ktpass -princ zookeeper/HOST@REALM -mapuser DOMAIN\Zookeeper_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass zk_password -out C:\keytabs\zookeeper.ktab
ktpass -princ zookeeper/HOST_FQDN@REALM -mapuser DOMAIN\Zookeeper_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass zk_password -in C:\keytabs\zookeeper.ktab -out C:\keytabs\zookeeper.ktab
ktpass -princ kafka/HOST@REALM -mapuser DOMAIN\Kafka_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass kfk_password -out C:\keytabs\kafka.ktab
ktpass -princ kafka/HOST_FQDN@REALM -mapuser DOMAIN\Kafka_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass kfk_password -in C:\keytabs\kafka.ktab -out C:\keytabs\kafka.ktab
Once the keytabs are created place them in a similar directory where Zookeeper/Kafka can access them. We will use these local files in the next sections.
1.Add the following entries (at the end of the file) to the zookeeper config file (zoo.cfg):
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
2.In the same directory where the zoo.cfg file is edit the jaas.conf file so that it has this content:
(Description of values) https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab="true"
principal="zookeeper/HOST_FQDN@REALM"
storeKey="true"
serviceName="zookeeper"
keyTab="C:/keytabs/zookeeper.ktab";
};
3.Edit the zookeeper runner batch file (zkServer.cmd) to have this:
-Djava.security.auth.login.config="%~dp0../conf/jaas.conf"
The updated command file should look like this:
call %JAVA% "-Dzookeeper.log.dir=%ZOO_LOG_DIR%" "-Dzookeeper.root.logger=%ZOO_LOG4J_PROP%" -Dsun.security.krb5.debug=true -Djava.security.auth.login.config="%~dp0../conf/jaas.conf" -cp "%CLASSPATH%" %ZOOMAIN% "%ZOOCFG%" %*
Note: the bold entry is optional and only required if you want to see detailed information from the security module.
4.When you run Zookeeper process make sure it runs as DOMAIN\Zookeeper_test.
1.Add the following entries (in the Socket Server Settings) to the kafka server config file (server.properties):
listeners=SASL_PLAINTEXT://HOST_FQDN:9093
security.inter.broker.protocol=SASL_PLAINTEXT
zookeeper.set.acl=true
sasl.mechanism=GSSAPI
2.In the same directory where server.properties file is edit the kafka_server_jaas.conf file so that it has this content:
//For connections to Kafka.
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab="true"
principal="kafka/HOST_FQDN@REALM"
storeKey="true"
serviceName="kafka"
keyTab="C:/keytabs/kafka.ktab";
};
//For connections to Zookeeper.
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab="true"
principal="kafka/HOST_FQDN@REALM"
serviceName="kafka"
keyTab="C:/keytabs/kafka.ktab";
};
3.Edit the kafka runner batch file (windows\kafka-server-start.bat) to have this:
-Djava.security.auth.login.config="%~dp0../../config/kafka_server_jaas.conf"
The updated command file should look like this:
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:"%~dp0../../config/log4j.properties" -Dkafka.logs.dir="%Dkafka_logs_dir%" -Djava.security.auth.login.config="%~dp0../../config/kafka_server_jaas.conf" -Dsun.security.krb5.debug=true
Note: the bold entry is optional and only required if you want to see detailed information from the security module.
If all works immediately then great, otherwise you to have a look at the console output with the following modifier in the jaas files (both Zookeeper and Kafka): debug="true"
4.When you run Kafka process make sure it runs as DOMAIN\Kafka_test.
Now you can try out the SASL_WIN32 kafka driver as updated by @zyzil. Please see here. For example:
rdkafka_example.exe -P -t SOME_TOPIC -b HOST_FQDN:9093 -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.service.name= kafka/HOST_FQDN@REALM -X sasl.kerberos.principal=kafka -d security,protocol,broker