Connecting to Zookeeper to edit topic configs

[unfrgivn]

First of all, thank you so much for this awesome repo, but I've been working on this specific issue for awhile and I can't find a solution and figured it would be helpful to others.

I am running the basic setup using the latest version of this repo on GKE. I have 3 Kafka brokers running with 2 pzoo and 3 zoo pods and the zookeeper service to make it accessible internally on 2181 (all unmodified files from this repo except CPU and memory increased). I can access my brokers externally with the Nodeport config described in this repo no problem and my whole system works for producing and consuming and I can view records externally in Kadeck.

My problem is that I need to be able to configure individual topics like log retention days, etc... normally I do this using the Kafka CLI tools, but you have to pass the zookeeper address (not the brokers). I tried to make zookeeper/30service.yml into a Nodeport with a nodeport of 32181. I made sure this port was open on my firewall and it's configured the same as the brokers so I'm confident this isn't the issue. I still can't connect to Zookeeper externally though using kafka-topics --zookeeper <cluster_node_ip>:32181 --list which times out.

I then thought I would then try to use these tools to connect by kubectl exec into one of the brokers, but they don't have Kafka CLI tools installed.

I also tried to load up Yahoo Kafka Manager since I think you can configure topics using that but I am getting the same error as this issue #260. I've tried setting the env ZK_HOSTS to both the default zookeeper.kafka:2181 as well as zookeeper.kafka.svc.cluster.local:2181 which is more akin to the hostname I use for connecting to the brokers, but no luck.

This seems like a simple/common task so before I go making a custom deployment just to install Kafka CLI tools interact with Zookeeper instance internally, I'm wondering if I'm doing something wrong here? How are others editing topic configs?

Thanks for any insights!

[unfrgivn]

I should also caveat that I am able to access CLI tools using --bootstrap-server flag instead of Zookeeper, but I have always used the --zookeeper flag so is there a downside to connecting to a random broker instead?

I found the Kafka CLI tools by kubectl exec into one of the Zookeeper pods, however I'm still not sure how to connect to Zookeeper from inside this pod. I tried the following from the bin folders (again using the default zookeeper yamls from this repo):

./kafka-topics.sh --zookeeper zookeeper.kafka:2181 --list
./kafka-topics.sh --zookeeper zookeeper.kafka.svc.cluster.local:2181 --list
./kafka-topics.sh --zookeeper localhost:2181 --list

What address would I use internally to connect to Zookeeper? I get the same error for all of these.

2019-09-08 23:03:27,278] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-09-08 23:03:28,291] INFO [ZooKeeperClient] Initializing a new session to zookeeper.kafka:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-09-08 23:03:28,298] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,298] INFO Client environment:host.name=pzoo-0.pzoo.kafka.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,298] INFO Client environment:java.version=11.0.2 (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,299] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,299] INFO Client environment:java.home=/usr/lib/jvm/jdk-11 (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,300] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,300] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,300] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,301] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,301] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,301] INFO Client environment:os.version=4.14.91+ (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,301] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,301] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,301] INFO Client environment:user.dir=/opt/kafka/bin (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,304] INFO Initiating client connection, connectString=zookeeper.kafka:2181 sessionTimeout=30000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@7a362b6b (org.apache.zookeeper.ZooKeeper)
[2019-09-08 23:03:28,337] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-09-08 23:03:28,345] INFO Opening socket connection to server zookeeper.kafka/10.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-09-08 23:03:28,360] INFO Socket connection established to zookeeper.kafka/10.x.x.x:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-09-08 23:03:28,370] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)