Event stream support for rkt

[yujuhong]

Docker has a watch feature to serve real-time events stream through /events. This is a nice feature to eliminate frequent polling. Is there any plan for rkt to support similar watch function? For example, rkt can use inotify to watch sysfs...

/cc @yifan-gu

[philips]

What events would you like to see? This is the list docker has: https://docs.docker.com/reference/api/docker_remote_api_v1.17/#monitor-docker-s-events

Most of these events are things that an init system does like notify you of process start, stop, etc events. If you are launching rkt containers under systemd you can get start/stop events via the systemd API. For example you could use this Go systemd API to get all changed state units from systemd:

[yifan-gu]

@yujuhong The use case is for syncPod to watch the event stream instead of doing pulling such as ListPod right?

[yujuhong]

Most of these events are things that an init system does like notify you of process start, stop, etc events. If you are launching rkt containers under systemd you can get start/stop events via the systemd API. For example you could use this Go systemd API to get all changed state units from systemd:

@philips, I'd like to see rkt support a pod lifecycle event stream. To give you more background (as @yifan-gu pointed out), I am exploring options to restructure kubelet for better performance and scale.

In kubernetes, kubelet is a daemon that manages containers on the node. Kubelet uses a pod abstraction, exposed through a container runtime interface that straddles both docker and rkt.

1. rkt supports the pod abstraction but doesn't expose a subscription stream for pod lifecycle
2. docker only supports containers but does expose a container event stream

This means a lot of the complexity is pushed down into the container runtime interface, while kubelet sits around poll-listing containers. Hence the latency of kubelet is predicated by the runtime's list/inspect performance, making it difficult to profile, optimize and reason about concurrently.

I'm trying to scope out if there is interest in supporting a pod (or container) event stream, and if there is, where it should live. Implementing a watch on systemd for rkt alone seems like a step in the wrong direction IMHO.

[dchen1107]

Any updates on this one?

[dchen1107]

Chatted with @yujuhong offline, and she proposed another option to achieve this by using cAdvisor event stream.

[yujuhong]

Chatted with @yujuhong offline, and she proposed another option to achieve this by using cAdvisor event stream.

To expand on this, we have two options for event stream as of now:

1. Use runtime-specific (docker/rkt) event stream
2. Subscribe to the cadvisor event stream, which uses inotify to watch /sys/fs/cgroup.

The former requires rkt to support event stream, but would provide richer event set for kubelet to consume. The latter requires cadvisor to support rkt, and provides only basic container events (start/termination). However, since Kubelet relies on cadvisor for resource monitoring, etc, it might be good for the long run if cadvisor can support rkt fullly. We'd appreciate if the rkt team can help with either option :-)

[yifan-gu]

@dchen1107 @yujuhong I am glad to help integrate with the cadvisor in short term. In longer term we are planning to provide an API for such things, as well as pod creation, image pulling, etc.

[alban]

I added a reference to this issue on systemd/systemd#1369 about EVENT_TYPE_APP_STARTED and EVENT_TYPE_APP_EXITED. Was the communication mechanism between the api service and the container decided/implemented?

[yifan-gu]

@alban Not yet.

[alban]

#1696 should go first before this issue.

[alban]

@yifan-gu what is the status on this?

[yifan-gu]

@alban Will try to come up with a design doc.

[yifan-gu]

@alban Still debugging network issues. Haven't worked on this today.

[tmrts]

Filed the issue in systemd to help rkt facilitate this feature systemd#3726