Installing NVIDIA GPU Operator - Does Bottlerocket come with container toolkit?

[aradwan777]

I am trying to install the nvidia gpu operator. It is not recommended to use according to BottleRocket and I wonder if that is because of the configuration settings we have to pass? I know we have to add flag to disable nvidia driver but what about container toolkit?

More info:

toolkit.enabled | By default, the Operator deploys the NVIDIA Container Toolkit (nvidia-docker2 stack) as a container on the system. Set this value to false when using the Operator on systems with pre-installed NVIDIA runtimes.

I am using EKS bottlerocket nvidia ami

[arnaldo2792]

We do include the NVIDIA container toolkit in Bottlerocket. The GPU operator will try to do more things than you expect it to do, that's why we don't recommend using it with Bottlerocket. I don't recall if there is a flag to pass to the GPU operator to prevent it from messing up with containerd's configurations. What's your use case for the GPU operator?

[ChristopherEdwardsNTC]

I'm also interested in using the GPU Operator with the use case being that I want to time share GPUs and have control over the number of GPU resources that the physical GPU gets shared out as.

I was able to disable the driver and toolkit installation following nvidia's guide, but the GPU Operator will not become healthy.

This is from the nvidia-operator-validator/driver-validation pod/container:

running command bash with args [-c stat /run/nvidia/validations/.driver-ctr-ready]                                                                                                                                                                                         stat: cannot statx '/run/nvidia/validations/.driver-ctr-ready': No such file or directory 

Is there a way to control the number of GPU resources on a per-node basis without using the GPU Operator I'd be interested.

[ChristopherEdwardsNTC]

I did try to work around this by touching the file it wants then it can't find nvidia-smi :(

[ChristopherEdwardsNTC]

i found nvidia-smi by searching the host (bottlerocket) file system but libnvidia-ml.so (or something similar I don't have the exact error handy) is missing

[arnaldo2792]

Hey @ChristopherEdwardsNTC , I'm sorry for the late response. I remember that I did make the GPU Operator work with a lot of workarounds to let it find the nvidia-smi and libnvidia-ml.so files, however what I did scale since I had to hard-code the NVIDIA driver version in the pod spec and this wouldn't work in a cluster where there are multiple Bottlerocket versions each with a different driver version. You mentioned you want to time share your GPUs, do you mean you want to use "Time Slicing" or a different feature?

[ChristopherEdwardsNTC]

"Time slicing" is the correct term :)

I wasn't able to generate enough overrides to get it to find nvidia-smi and at the moment I've given up. Time slicing is something I need to support for my users and use cases, however we are getting by with what we have.