Kafka with multiple Listeners and SASL

This will quickly discuss how to configure multiple Listeners, with the intent of having a unique Listener for External/Client traffic and another for Internal/Inter-broker traffic (and how this can be done with Cloudera Manager which requires a slight work-around in the current versions pre-2021).

There’s several valid use cases for multiple Listeners. In this case, we have brokers with multiple NICs, where one NIC is designated for internal traffic, while the other NIC is for external traffic.

Example 1 - Vanilla Kafka, 2 NICs, different IPs, different Hostnames

Easy enough to do with Kafka on its own:

listeners=PUBLIC://<public_hostname>:9093, PRIVATE://<external_hostname>:19093
listener.security.protocol.map=PUBLIC:SASL_SSL,PRIVATE:SASL_SSL
advertised.listeners=PUBLIC_NIC://<public_hostname>:9093, SASL_SSL://<private_hostname>:19093
inter.broker.listener.name=PRIVATE

Example 2 - Cloudera Manager + Kafka, 2 NICs, different IPs, different Hostnames

Unfortunately, there is a pending issue with Cloudera Manager right now that does not allow us to remove the security.inter.broker.protocol setting - which, annoyingly, means we can’t use the same config as above - you’ll see the following error:

org.apache.kafka.common.config.ConfigException: Only one of inter.broker.listener.name and security.inter.broker.protocol should be set.

So we have a work around…

listeners=PUBLIC://<public_hostname>:9093, SASL_SSL://<external_hostname>:19093
listener.security.protocol.map=PUBLIC:SASL_SSL,SASL_SSL:SASL_SSL
advertised.listeners=PUBLIC_NIC://<public_hostname>:9093, SASL_SSL://<private_hostname>:19093

There are 2 things to note:

We have renamed the PRIVATE listener to SASL_SSL (same name as the protocol we want it to use)
We do not specify the inter.broker.listener.name

We also need to ensure that the Inter Broker Protocol setting in Cloudera Manager is set to match the Listener name for the private interface, i.e. SASL_SSL.

We’re basically tricking Cloudera Manager by naming the Listener exactly the same as its protocol.

Example 3 - Vanilla Kafka, 2 NICs, different IPs, same Hostnames

listeners=PUBLIC://<public_ip>:9093, PRIVATE://<external_ip>:19093
listener.security.protocol.map=PUBLIC:SASL_SSL,PRIVATE:SASL_SSL
advertised.listeners=PUBLIC_NIC://<hostname>:9093, SASL_SSL://<hostname>:19093
inter.broker.listener.name=PRIVATE

Example 4 - Cloudera Manager + Kafka, 2 NICs, different IPs, same Hostnames

Apply the same workaround as Example 2.

listeners=PUBLIC://<public_ip>:9093, SASL_SSL://<external_ip>:19093
listener.security.protocol.map=PUBLIC:SASL_SSL,SASL_SSL:SASL_SSL
advertised.listeners=PUBLIC_NIC://<hostname>:9093, SASL_SSL://<hostname>:19093