feat: Leverage OCI Hooks for Container Events

[akshay196]

Feature Request

Short Description

Use OCI hooks and get events in context to container start/stop: Currently KubeArmor mounts docker/containerd/crio UNIX domain socket file in KubeArmor to watch for container events. The aim is to use OCI hooks for getting such container events.

Is your feature request related to a problem? Please describe the use case.

Eliminate exposing docker/containerd/crio UNIX domain sockets inside a container.

Describe the solution you'd like

OCI hooks can be used to get containers create/stop events which does not require any access to runtime sockets.

Posix hooks documentation - https://github.com/opencontainers/runtime-spec/blob/v1.1.0/config.md#posix-platform-hooks
Inspektor-gadget runc hook example - https://github.com/inspektor-gadget/inspektor-gadget/tree/main/examples/runc-hook

Describe alternatives you've considered

Nothing.

Note

Created out of #1130. Kept earlier issue for OCI registry feature.

[akshay196]

Design proposal draft document - https://docs.google.com/document/d/1abw7IwlLCgS1VUQLHM5PtxbaYPwkkIZyQCzbcd7MLUg/edit
@daemon1024 @nyrahul @Ankurk99 @kranurag7 Please share your thoughts.

[Ankurk99]

Hey @akshay196, can we also talk about the reasons for not considering PodInformers, Fanotify? (anything else that we considered?)

[nyrahul]

Great document. We need to move this document to Wiki. Or we can just push the markdown here in the issue. I am worried about losing doc from the above link.

[akshay196]

Great document. We need to move this document to Wiki. Or we can just push the markdown here in the issue. I am worried about losing doc from the above link.

Thanks. I have shared the Google document because it is in WIP. It would be easy to get comment/suggestions in Google doc. Once it is ready, I will move it to issue here.

[akshay196]

can we also talk about the reasons for not considering PodInformers, Fanotify? (anything else that we considered?)

@Ankurk99 Added section that talks about different options considered.

[1awesomeJ]

Hello @akshay196,
Is this project still open for LFX term 1 2024?
I would like to work on it.

[1awesomeJ]

I noticed that create, start, and stop are the only events you listed.
Does that mean those are the priority events we'll be starting with?

[akshay196]

Is this project still open for LFX term 1 2024?

The project is added to LFX Mentorship, Term 01 - 2024 March - May. Do submit your application once mentee applications are open.

I noticed that create, start, and stop are the only events you listed.
Does that mean those are the priority events we'll be starting with?

KubeArmor monitors start, stop, destroy events as per the container runtime in use. Notice Monitor*Events() functions for different runtime handlers located at core package. All events that KubeArmor watch out for should considere in OCI hooks implementation.

[1awesomeJ]

Thank you so much.
I'll start prepping.

[1awesomeJ]

I have set up a minikube cluster, and have now cloned the repository.
I want to build the project and test out features.

I ran go build from the parent directory, but my compiler doesn't see go initialized.

Is there a guide I could follow to build, and test the project on a local k8s cluster(minikube)?

Thank you.

[1awesomeJ]

go build works now.
Just had to run it in the right directory.

[daemon1024]

Hey Folks, Thanks for the interest in the mentorship. We have certain prerequisites which we expect to be included in your application. Please include details or reference to a document for the said prerequisite in your Cover Letter / Mail to the mentors / Submit it in the issue thread / DM Mentors in KubeArmor Slack by 20 Feb.

Following are the details.

Leverage OCI Hooks for Container Events - https://mentorship.lfx.linuxfoundation.org/project/a604ba9c-565d-4e8c-aed2-dcd4ebedc85d
Prerequisite: Setup a sample OCI Hook.
Background: We have a full fledged design document around what the integration would look like thanks to the previous LFX Mentee. A basic requirement to understand the design document and implementing is that mentee knows about OCI Hooks. So we would like the mentee to setup a sample OCI Hook in a hands on way so we are sure that the mentee has entire context to understand the upstream issue and implement it.

[1awesomeJ]

Hey Folks, Thanks for the interest in the mentorship. We have certain prerequisites which we expect to be included in your application. Please include details or reference to a document for the said prerequisite in your Cover Letter. Following are the details.

Leverage OCI Hooks for Container Events - https://mentorship.lfx.linuxfoundation.org/project/70f1ef34-7ddd-466d-a5bc Prerequisite: Setup a sample OCI Hook. Background: We have a full fledged design document around what the integration would look like thanks to the previous LFX Mentee. A basic requirement to understand the design document and implementing is that mentee knows about OCI Hooks. So we would like the mentee to setup a sample OCI Hook in a hands on way so we are sure that the mentee has entire context to understand the upstream issue and implement it.

Thank you @daemon1024 .

For those of us who have already submitted our cover letter, can we provide the reference tobthe prerequisite here once we have it? Or do we have to absolutely submit another cover letter.

I would appreciate if we could provide the reference/document detailing our implementation here since the LFX portal will be closing tomorrow.

Thank you.

[daemon1024]

For those of us who have already submitted our cover letter, can we provide the reference tobthe prerequisite here once we have it? Or do we have to absolutely submit another cover letter.

Definitely it's okay to submit here, mail to us, or just sent it to us over DM in Slack. Apologies for not mentioning it. I have updated the original message

[1awesomeJ]

Thank you.

[1awesomeJ]

@daemon1024 @akshay196 , I'm trying to implement a prestart hook that just sets an environment variable.
I expect the script to run whenever I spin up a new pod, but so far it doesn't.

I am using an Ubuntu VM
Here are my steps so far:

1. I installed the docker packages
   sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

2. I installed kubectl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

3. I installed minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

4. I started minikube using docker as the driver
   minikube start --driver=docker

5. I logged into the minikube VM using minikube ssh

6. Inside the minikube VM, I created the directory /var/ocihooks

7. I created the script for the prestart hook ("prestarthook.sh") with the following content:

#!/bin/bash

echo "prestarthook.sh: Executed at $(date)" >> /var/ocihooks/hook_log.txt
echo "Setting DEFAULT_CONTAINER_RUNTIME for all pods..."
export DEFAULT_CONTAINER_RUNTIME="crio" 

8. I set the right permissions on the script, the log file(hook_log.txt) and the parent directories.

9. I edited my /etc/containerd/config.toml file and added the following lines:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.Runtime]
        RuntimeType = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.Hooks]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.Hooks.prestart]
            path = "/var/ocihooks/prestarthook.sh"

10. I did systemctl restart containerd.service

11. I exited the ssh connection to the minikube VM

12. I restarted minikube

13. Now inside my main VM, I created a new pod uisng this manifest

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: nginx:latest
    command: ["sleep", "3600"]

14. I expected my pre start hook to have run during the pod's creation, such that if I logged into the pod, I should see my environment variable in the output of env
    I did:
    kubectl exec -it test-container -- bash
    then env
    My environment variable DEFAULT_CONTAINER_RUNTIME was not set.
    Also the file /var/ocihooks/hooklog.txt didn't have anything appended to it.

I'm working on resolving this, but If you have spotted where I'm missing it, kindly guide me through, thank you sirs.

[ChucklesDroid]

Hey Folks, Thanks for the interest in the mentorship. We have certain prerequisites which we expect to be included in your application. Please include details or reference to a document for the said prerequisite in your Cover Letter / Mail to the mentors / Submit it in the issue thread / DM Mentors in KubeArmor Slack by 20 Feb.

Following are the details.

Leverage OCI Hooks for Container Events - https://mentorship.lfx.linuxfoundation.org/project/70f1ef34-7ddd-466d-a5bc Prerequisite: Setup a sample OCI Hook. Background: We have a full fledged design document around what the integration would look like thanks to the previous LFX Mentee. A basic requirement to understand the design document and implementing is that mentee knows about OCI Hooks. So we would like the mentee to setup a sample OCI Hook in a hands on way so we are sure that the mentee has entire context to understand the upstream issue and implement it.

How much time do we have to submit this? I just saw it mentioned

[daemon1024]

@1awesomeJ It's a bit not straightforward method to make OCI Hooks work with containerd, you can check the design doc it has detailed steps into registering OCI hooks with containerd. Primarily the difference in approach is modifying the base spec and setting the basespec path in containerd config, rather than trying to add hooks to containerd config directly.

@ChucklesDroid as mentioned in the comment. The deadline is 20 Feb.

[1awesomeJ]

@1awesomeJ It's a bit not straightforward method to make OCI Hooks work with containerd, you can check the design doc it has detailed steps into registering OCI hooks with containerd. Primarily the difference in approach is modifying the base spec and setting the basespec path in containerd config, rather than trying to add hooks to containerd config directly.

@ChucklesDroid as mentioned in the comment. The deadline is 20 Feb.

Wow, thank you so much. This guidance is significant.

[1awesomeJ]

@daemon1024 @akshay196 , I have SUCCESSFULLY implemented an OCI hook that runs with CRIO container runtime!

I am using an Ubuntu VM
Here are the steps I followed:

1. I installed the docker packages
   sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

2. I installed kubectl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

3. I installed minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

4. I started minikube using crio as the container runtime
   minikube start --container-runtime=crio

5. I logged into the minikube VM using minikube ssh

6. Inside the minikube VM, I created three files hook_log.txt, kubearmor-hook.json and prestarthook.sh inside the /usr/share/containers/oci/hooks.d/ directory.

prestarthook.sh has the following content:

#!/bin/bash

echo "Setting the desired default container runtime to 'crio'"
export DEFAULT_CONTAINER_RUNTIME="crio"
echo "OCI hook executed successfully! at $(date)" >> /usr/share/containers/oci/hooks.d/hook_log.txt

kubearmor-hook.json has the following content:

{
    "version": "1.0.0",
    "hook": {
        "path": "/usr/share/containers/oci/hooks.d/prestarthook.sh"
    },
    "when": {
        "always": true 
    },
    "stages": ["createContainer", "poststop"] 
}

hook_log.txt was an ampty file.

8. I set the right permissions on kubearmor-hook.json and hook_log.txt

sudo chmod a+x /usr/local/bin/prestarthook.sh
sudo chmod 777 /usr/share/containers/oci/hooks.d/hook_log.txt

9. I exited the ssh connection to the minikube VM

10. I created a new pod
    kubectl run test-pod --image=docker.io/nginx:latest

11. I logged back into the minkube VM
    minikube ssh

12. I checked the content of hook_log.txt, and it had the expected content like so:

docker@minikube:~$ cat /usr/share/containers/oci/hooks.d/hook_log.txt
OCI hook executed successfully! at Wed Feb 14 15:31:54 UTC 2024
docker@minikube:~$ 

Kindly review this task, and please let me know if there are other things I need to do.

In the meantime, I will work on creating and testing an OCI hook with containerd runtime.

Thank you!!!

[1awesomeJ]

@daemon1024 @akshay196,
I have now implemented OCI hooks in both crio and containerd runtimes.

I have updated my documentation at https://github.com/1awesomeJ/KubeArmor/blob/LFX_Mentorship/OCI_Hook_implementation.md

Kindly review this updated submission.
Thnak you.

[1awesomeJ]

For Docker runtime conatiners, What I have seen is that docker leverages containerd or runc, and there may not yet be an explicit way of integrating hooks with them?

What do you say to this? What would you recommend for studying? Thank you.

[AbdelrahmanElawady]

Here is a list of the tasks after discussion on KubeArmor slack:

• Configure Container Runtime and Hosts:
  • Setup the right runtime hooks with the right conditions. i.e. on createContainer, createRuntime or postStop invoke needed executable. (for all container runtimes).
  • In non-K8s mode, we will document how it is done (and maybe automate that too as an extension to the project).
  • Setup the hook executable on the host.
• Hook Executable:
  • Figure out which runtime it is being executed on.
  • Get containers details.
  • Send these details to KubeArmor through a socket KubeArmor is listening on.
• KubeArmor:
  • Add listening to hook logic which consists of figuring out which are new containers and which are deleted and apply security profile accordingly.
  • Remove Container Runtime polling gradually and replace it with listening to hook to update its data.
• Resource Definitions:
  • Remove Container Runtime socket from mounts and add socket for listening for hook to mounts.

We will start with figuring out which hook we will use from different container lifecycle hooks. Then, starting with CRI-O we will begin with KubeArmor changes, update resource definitions, configuring hook.