Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce FVT disk usage #368

Merged
merged 14 commits into from
May 9, 2023
Merged

Conversation

ckadner
Copy link
Member

@ckadner ckadner commented May 5, 2023

FVT

Motivation

https://github.com/kserve/modelmesh-serving/actions/runs/4831653258/jobs/8609373544#step:11:367

failed to register layer: ApplyLayer exit status 1 stdout:  stderr: write /usr/local/lib64/python3.8/site-packages/tensorflow/libtensorflow_cc.so.2: no space left on device
Error: Process completed with exit code 1.

Follow best practices to reduces disk usage for GitHub action runners by deleting binaries that are not needed.

sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

Modifications

      - name: Free up disk space
        run: |
          # https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
          sudo rm -rf /opt/ghc
          sudo rm -rf /usr/share/dotnet
          sudo rm -rf /usr/local/share/boost
          # also delete tools/libs for Android (12G), PowerShell (1.3G), Swift (1.7G)
          sudo rm -rf /usr/local/lib/android
          sudo rm -rf /usr/local/share/powershell
          sudo rm -rf /usr/share/swift

Result

Increased available disk space by 25 GB (plus an additional reclaimed space: 2.6 GB after docker prune).

Resolves #367

Resolves kserve#367

Signed-off-by: Christian Kadner <[email protected]>
ckadner added 2 commits May 4, 2023 19:46
Signed-off-by: Christian Kadner <[email protected]>
Signed-off-by: Christian Kadner <[email protected]>
@ckadner ckadner force-pushed the debug_fvt_no_space_error branch from 26baa96 to 92f35ec Compare May 5, 2023 02:58
@ckadner ckadner marked this pull request as draft May 5, 2023 06:59
@ckadner ckadner changed the title Debug FVT disk usage Reduce FVT disk usage May 5, 2023
@ckadner ckadner force-pushed the debug_fvt_no_space_error branch from 124d787 to 4e7325e Compare May 8, 2023 20:43
@ckadner ckadner marked this pull request as ready for review May 8, 2023 22:04
@ckadner ckadner added the test testing related bugs and fixes label May 8, 2023
@ckadner ckadner requested review from njhill and removed request for joerunde May 9, 2023 00:32
Copy link
Contributor

@tjohnson31415 tjohnson31415 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me!
Another problem with FVTs solved!

@kserve-oss-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner, tjohnson31415

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ckadner ckadner merged commit 1b3d30a into kserve:main May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved test testing related bugs and fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FVTs don't run due to "No space left on device" error
3 participants