Docker compose setup for async workflows with kafka queue

[taylanisikdemir]

What changed?

• Added new docker compose file which runs all typical Cadence dependencies plus Kafka/Zookeeper.
• Updated config_template.yaml with a new conditional asyncWorkflowQueues section. Only enabled by the new docker compose file via an env var.
• Updated Dockerfile to generate a new image that includes kafka binaries. Only need kafka-topics.sh to check during startup for now. The new image is tagged as ubercadence/server:master-auto-setup-with-kafka.

Why?
This setup will be handy to try async workflows locally and also to setup CI integration test.

How did you test it?
Ran docker-compose -f docker/docker-compose-async-wf-kafka.yml up and validated all containers up and running. Verified the topic is created.
A follow up PR will validate async workflows working e2e.

[coveralls]

Pull Request Test Coverage Report for Build 018daecf-4877-4be3-b0f0-e7bb2b04844e

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 98 unchanged lines in 15 files lost coverage.
• Overall coverage decreased (-0.03%) to 62.723%

---

Files with Coverage Reduction	New Missed Lines	%
client/history/client.go	2	38.1%
common/persistence/sql/sqlplugin/postgres/db.go	2	85.0%
common/persistence/sql/sqlplugin/postgres/task.go	2	73.4%
common/task/weighted_round_robin_task_scheduler.go	2	89.05%
service/history/handler.go	2	49.97%
service/matching/matcher.go	2	90.72%
service/matching/taskReader.go	2	84.88%
service/history/queue/timer_gate.go	3	95.83%
service/frontend/api/handler.go	4	62.11%
service/history/task/fetcher.go	4	85.05%

Totals
Change from base Build 018dae9f-1cbd-4682-94eb-c8ea3a325702:	-0.03%
Covered Lines:	92593
Relevant Lines:	147622

---

💛 - Coveralls

[shijiesheng]

tested

root@74af085dad6c:/# $KAFKA_HOME/bin/kafka-topics.sh --describe --topic async-wf-topic1 --zookeeper zookeeper:2181
Topic:async-wf-topic1   PartitionCount:10       ReplicationFactor:1     Configs:
        Topic: async-wf-topic1  Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 1    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 3    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 4    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 5    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 6    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 7    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 8    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: async-wf-topic1  Partition: 9    Leader: 1001    Replicas: 1001  Isr: 1001

Also another possible optimization is to add another kafka setup image like this stack suggests to avoid a bigger additional cadence server image.
https://stackoverflow.com/questions/47615751/docker-compose-run-a-script-after-container-has-started