Using SASL with librdkafka

Note: This guide assumes you have a Debian/Ubuntu system. Other OS and distributions will have the same packages but names and paths may be different.
For Windows please follow the guide SASL via SSPI (Windows).

Note: librdkafka must be built with SASL support (which is enabled by default if libsasl2-dev is installed at buildtime). You can query librdkafka for enabled feattures by getting the builtin.features configuration property.

#Overview

Kerberos keytabs (file-based pre-authenticated keys) are created for each broker in the cluster as well as for each client. They keytabs are distributed to the broker and client nodes respectively.

Decide on the following things:

• REALM - Your Kerberos realm, typically your operational domain in upper case. E.g., YOURDOMAIN.COM
• KDC_HOST - Host where kerberos runs, for simplicity we'll run it on the Kafka broker host. broker1
• SERVICENAME - The Kerberos service name, the service is Kafka so I suggest you use kafka.
• BROKER_HOST - Broker hostname, E.g., broker1. Quantify as necessary.
• CLIENT_NAME - Client name, e.g., "kafkaclient".
• CLIENT_HOST - Client hostname, i.e., where the client application runs, e.g., client1. Quantify as necessary.

#1. Set up Kerberos

NOTE: I strongly suggest reading Ubuntu's Kerberos guide.

Install the kerberos server if you do not already have a Kerberos server installed:

sudo apt-get install krb5-kdc krb5-admin-server

Answer the questions accordingly:

• Default Kerberos version 5 realm: insert your REALM.
• Kerberos servers for your realm: insert your KDC_HOST
• Administrative server for your Kerberos realm: insert your KDC_HOST

#2. Configure Kafka broker for SASL

Follow instructions here: http://docs.confluent.io/2.0.1/kafka/sasl.html

#3. Create client principal and keytab on Kerberos server

# kadmin.local -q 'addprinc -randkey ${CLIENT_NAME}/${CLIENT_HOST}@{REALM}'
# kadmin.local -q 'ktadd -k /etc/security/keytabs/${CLIENT_NAME}.keytab ${CLIENT_NAME}/${CLIENT_HOST}@{REALM}'

Securely copy the /etc/security/keytabs/${CLIENT_NAME}.keytab file to the ${CLIENT_HOST}, preferably in the same location. Set up permissions to secure the file accordingly, make sure the user that will run the Kafka client has access to read the keytab file.

#4. Install SASL modules on client host

sudo apt-get install libsasl2-modules-gssapi-mit

#5. Configure Kafka client on client host

The configuration listed below are standard librdkafka configuration properties (see CONFIGURATION.md), how these are actually set in a librdkafka based client depends on the application, for instance kafkacat uses -X <prop>=<val> command line arguments.

# Use SASL plaintext
security.protocol=SASL_PLAINTEXT

# Broker service name
sasl.kerberos.service.name=$SERVICENAME

# Client keytab location
sasl.kerberos.keytab=/etc/security/keytabs/${CLIENT_NAME}.keytab

# sasl.kerberos.principal
sasl.kerberos.principal=${CLIENT_NAME}/${CLIENT_HOST}

NOTE: Make sure to replace $... with the appropriate values above.

#6. Try it

NOTE: The ${BROKER_HOST} must exactly match the hostname part of the broker's principal. E.g., make sure to connect to broker abc123 if the Kerberos principal is kafka/[email protected].

# kafkacat -b ${BROKER_HOST} -L -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.keytab=/etc/security/keytabs/${CLIENT_NAME}.keytab -X sasl.kerberos.principal=${CLIENT_NAME}/${CLIENT_HOST}

##Hint

• If you are running kdc on a non-standard port or using a non-standard config file, make sure to export KRB5_KDC_PROFILE=/path/to/kdc.conf and possibly also KRB5_CONFIG=/path/to/krb5.conf