-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Using SASL with librdkafka on Windows
For a trivial zookeeper/kafka ensemble/cluster all running on machine: HOST, perform the following steps to enable SASL via SSPI.
Create AD user for Zookeeper/Kafka (the format is AD domain\AD user name):
• domain\Zookeeper_test (say the password is zk_password)
• domain\Kafka_test (say the password is kfk_password)
Ask the AD administrator to run (where UPPERCASE_REALM is the Kerberos realm for the HOST - for example COMPANY.COM):
SETSPN -S zookeeper/**_HOST_**@UPPERCASE_REALM domain\Zookeeper_test SETSPN -S zookeeper/**_HOST_**.fully_qualified@UPPERCASE_REALM domain\Zookeeper_test
SETSPN -S kafka/**_HOST_**@UPPERCASE_REALM domain\Kafka_test SETSPN -S kafka/**_HOST_**.fully_qualified@UPPERCASE_REALM domain\Kafka_test
https://technet.microsoft.com/en-us/library/cc753771(v=ws.11).aspx
You will need the respective AD user password from step 1; the SPN from step 2; and a folder for the generated output (you can call this folder C:\keytabs).
We will create keytab data for both the simple and fully qualified domain name of the HOST.
Note that the ktpass utility might not be available on all versions of Windows:
ktpass -princ zookeeper/**_HOST_**@UPPERCASE_REALM -mapuser domain\Zookeeper_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass zk_password -out C:\keytabs\zookeeper.ktab
ktpass -princ zookeeper/**_HOST_**.fully_qualified@UPPERCASE_REALM -mapuser domain\Zookeeper_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass zk_password -in C:\keytabs\zookeeper.ktab -out C:\keytabs\zookeeper.ktab
ktpass -princ kafka/**_HOST_**@UPPERCASE_REALM -mapuser domain\Kafka_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass kfk_password -out C:\keytabs\kafka.ktab
ktpass -princ kafka/**_HOST_**.fully_qualified@UPPERCASE_REALM -mapuser domain\Kafka_test -crypto RC4-HMAC-NT -ptype KRB5_NT_PRINCIPAL -pass kfk_password -in C:\keytabs\kafka.ktab -out C:\keytabs\kafka.ktab
Once the keytabs are created place them in a similar directory where Zookeeper/Kafka can access them. We will use these local files in the next sections.
1.Add the following entries (at the end of the file) to the zookeeper config file (zoo.cfg):
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
2.In the same directory where the zoo.cfg file is edit the jaas.conf file so that it has this content:
(Description of values) https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab="true"
principal="zookeeper/**_HOST_**.fully_qualified@UPPERCASE_REALM"
storeKey="true"
serviceName="zookeeper"
keyTab="C:/keytabs/zookeeper.ktab";
};
3.Edit the zookeeper runner batch file (zkServer.cmd) to have this:
-Djava.security.auth.login.config="%~dp0../conf/jaas.conf"
The updated command file should look like this:
call %JAVA% "-Dzookeeper.log.dir=%ZOO_LOG_DIR%" "-Dzookeeper.root.logger=%ZOO_LOG4J_PROP%" -Dsun.security.krb5.debug=true -Djava.security.auth.login.config="%~dp0../conf/jaas.conf" -cp "%CLASSPATH%" %ZOOMAIN% "%ZOOCFG%" %*
Note: the bold entry is optional and only required if you want to see detailed information from the security module.
4.When you run Zookeeper process make sure it runs as domain\Zookeeper_test.
1.Add the following entries (in the Socket Server Settings) to the kafka server config file (server.properties):
listeners=SASL_PLAINTEXT://**_HOST_**.fully_qualified:9093
security.inter.broker.protocol=SASL_PLAINTEXT
zookeeper.set.acl=true
sasl.mechanism=GSSAPI
2.In the same directory where server.properties file is edit the kafka_server_jaas.conf file so that it has this content:
//For connections to Kafka.
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab="true"
principal="kafka/**_HOST_**.fully_qualified@UPPERCASE_REALM"
storeKey="true"
serviceName="kafka"
keyTab="C:/keytabs/kafka.ktab";
};
//For connections to Zookeeper.
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab="true"
principal="kafka/**_HOST_**.fully_qualified@UPPERCASE_REALM"
serviceName="kafka"
keyTab="C:/keytabs/kafka.ktab";
};
3.Edit the kafka runner batch file (windows\kafka-server-start.bat) to have this:
-Djava.security.auth.login.config="%~dp0../../config/kafka_server_jaas.conf"
The updated command file should look like this:
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:"%~dp0../../config/log4j.properties" -Dkafka.logs.dir="%Dkafka_logs_dir%" -Djava.security.auth.login.config="%~dp0../../config/kafka_server_jaas.conf" -Dsun.security.krb5.debug=true
Note: the bold entry is optional and only required if you want to see detailed information from the security module.
If all works immediately then great, otherwise you to have a look at the console output with the following modifier in the jaas files (both Zookeeper and Kafka): debug="true"
4.When you run Kafka process make sure it runs as domain\Kafka_test.
Now you can try out the SASL_WIN32 kafka driver as updated by @zyzil. Please see here. For example:
rdkafka_example.exe -P -t SOME_TOPIC -b **_HOST_**.fully_qualified:9093 -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.service.name= kafka/**_HOST_**.fully_qualified@UPPERCASE_REALM -X sasl.kerberos.principal=kafka -d security,protocol,broker