Investigation: Running this as a Job rather than DaemonSet.

[Tatsinnit]

Currently we run this tool as Daemonset, it is being discussed few times that what will it take to make it run as a job https://kubernetes.io/docs/concepts/workloads/controllers/job/ : also, a great idea share by @arnaud-tincelin - we can scope this so that we can get pros and cons of this approach. At present I am not sure what all functionality we will not be able to deliver when run as a job because there are few logs which will run as direct commands in host etc.

A gentle ping to @JunSun17 : if you can please guide us with the daemonset option and was the job option was not done due to any specific reasons.

This is one of a good workitem to stock take and scope. Opened it as a place holder for discussion and ideas. Thanks heaps.

[Tatsinnit]

💡 Loop back for some thoughts, but since with DaemonSet we get assurance that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created. https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

Some really good discussion here: kubernetes/kubernetes#64623 - I believe that daemon set is the right way moving forward for this tool. I especially like this comment, which discuss some work-around patterns: kubernetes/kubernetes#64623 (comment)

There was an idea about some connectivity prober which is easily done is we use the current mechanism between the aks-periscope pods.

Prima facie, A Job is designed to run to completion but there’s no way to ensure one runs on every node. A DaemonSet is designed to run on every node but there’s no way to ensure that it runs to completion only once.

Thanks, 🙏☕️

[bcho]

Discussed with @Tatsinnit offline, seems like using a daemonset is still the way to collect logs from each agent node. We think in the long term, we might be able to come up a new design by using a controller to have fine grained control over the log process.

@Tatsinnit I think we can close it now?

[Tatsinnit]

Absolutely, thank you so much for chiming in @bcho , 🙏❤️☕️ I will close and pin this issue.