Leaking connection

[anhdat]

Describe the bug
While running argo-events v0.9.2, after running for awhile, the gateway stops responding with error messages:

�[33mWARN�[0m[2019-05-15 12:14:06] failed to dispatch event to sensor watcher over http. communication error  �[33merror�[0m="Post http://calendar-sensor-svc:9300/: dial tcp: lookup calendar-sensor-svc on 172.20.0.10:53: dial udp 172.20.0.10:53: socket: too many open files" �[33msensor-name�[0m=calendar-sensor

Checking open files count in gateway-client container we see:

sh-4.2# lsof | wc -l
351
sh-4.2# lsof | wc -l
351
sh-4.2# lsof | wc -l
351
sh-4.2# lsof | wc -l
351
sh-4.2# lsof | wc -l
604
sh-4.2# lsof | wc -l
3059

To Reproduce
Steps to reproduce the behavior:

1. Deploy argo and argo-events in Kubernetes
2. Create calendar gateway and calendar sensor
3. Check opening files count inside gateway-client container:

# install lsof
yum install -y lsof
# check opening files
while sleep 1; do lsof | grep gateway-c | wc -l; done

Expected behavior

Opening files shouldn't exceed OS limit

Environment (please complete the following information):

• Kubernetes: 1.11 EKS
• Version: v0.9.2