Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: resolve Azure Linux nvidia-container-toolkit posttrans error #5321

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

fix: resolve Azure Linux nvidia-container-toolkit posttrans error #5321

wants to merge 1 commit into from

Conversation

henryli001
Copy link
Contributor

What type of PR is this?

What this PR does / why we need it:

  • This PR fixes an issue with Azure Linux AKS GPU where an error takes place within the posttrans phase when installing the nvidia-container-toolkit RPM package. that causes /etc/nvidia-container-runtime/config.toml to not be generated in the OS. The reason of the failure is because nvidia-ctk, the binary used to generate the config file is missing. This is because nvidia-ctk is provided by the nvidia-container-toolkit-base RPM package and this RPM is installed after nvidia-container-toolkit. This PR thus changes the RPM installation sequence to satisfy the package runtime requirements.

Which issue(s) this PR fixes:

Fixes #

Requirements:

Special notes for your reviewer:

Release note:

none

@cameronmeissner
Copy link
Collaborator

lgtm I think, is this currently causing wide impact in prod? if so then it'd probably worth to cherry-pick this to master so it can roll out in next week's release

dallasd1
dallasd1 reviewed Nov 21, 2024
View reviewed changes
parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh
@@ -82,7 +82,7 @@ installNvidiaContainerToolkit() {
MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION="1.16.2"
fi

for nvidia_package in nvidia-container-toolkit-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION} nvidia-container-toolkit-base-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION} libnvidia-container-tools-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION} libnvidia-container1-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION}; do
for nvidia_package in libnvidia-container1-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION} libnvidia-container-tools-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION} nvidia-container-toolkit-base-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION} nvidia-container-toolkit-${MARINER_NVIDIA_CONTAINER_TOOLKIT_VERSION}; do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a comment to show this ordering is necessary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment added.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to find official Nvidia docs that mentioned about this ordering dependency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not find any official doc that directly explains the dependency relationship between these packages. But a good source to refer to would be the official spec file for nvidia container toolkit: https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/packaging/rpm/SPECS/nvidia-container-toolkit.spec where the runtime requirements explain the dependency.

@henryli001 henryli001 force-pushed the lihl/fix-azurelinux-nvidia-container-toolkit branch from 472e8a1 to 0616c06 Compare November 22, 2024 21:59
@henryli001
Copy link
Contributor Author

lgtm I think, is this currently causing wide impact in prod? if so then it'd probably worth to cherry-pick this to master so it can roll out in next week's release

This has not been causing impact in prod yet since no customer bug requests have been noticed, but there's a possibility that customer workloads might be impacted by this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dallasd1 dallasd1 dallasd1 left review comments

@juan-lee juan-lee Awaiting requested review from juan-lee juan-lee is a code owner

@cameronmeissner cameronmeissner Awaiting requested review from cameronmeissner cameronmeissner is a code owner

@UtheMan UtheMan Awaiting requested review from UtheMan UtheMan is a code owner

@ganeshkumarashok ganeshkumarashok Awaiting requested review from ganeshkumarashok ganeshkumarashok is a code owner

@anujmaheshwari1 anujmaheshwari1 Awaiting requested review from anujmaheshwari1 anujmaheshwari1 is a code owner

@AlisonB319 AlisonB319 Awaiting requested review from AlisonB319 AlisonB319 is a code owner

@Devinwong Devinwong Awaiting requested review from Devinwong Devinwong is a code owner

@lilypan26 lilypan26 Awaiting requested review from lilypan26 lilypan26 is a code owner

@AbelHu AbelHu Awaiting requested review from AbelHu AbelHu is a code owner

@junjiezhang1997 junjiezhang1997 Awaiting requested review from junjiezhang1997 junjiezhang1997 is a code owner

@jason1028kr jason1028kr Awaiting requested review from jason1028kr jason1028kr is a code owner

@djsly djsly Awaiting requested review from djsly djsly is a code owner

@phealy phealy Awaiting requested review from phealy phealy is a code owner

@r2k1 r2k1 Awaiting requested review from r2k1 r2k1 is a code owner

@timmy-wright timmy-wright Awaiting requested review from timmy-wright timmy-wright is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@henryli001 @cameronmeissner @ganeshkumarashok @dallasd1