Skip to content

Commit

Permalink
chore: Fix no space left on device error in `predictor-runtime-buil…
Browse files Browse the repository at this point in the history
…d` (kubeflow#3049)

* Create a separate action to free-up disk space on GH action runner
* Move 'free-up-disk-space' step outside 'minikube-setup' action
* Free-up disk space before running 'predictor-runtime-build'

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
  • Loading branch information
ckadner authored Jul 25, 2023
1 parent a294e6e commit a8b9046
Show file tree
Hide file tree
Showing 4 changed files with 50 additions and 31 deletions.
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Please delete options that are not relevant.

**Feature/Issue validation/testing**:

Please describe the tests that you ran to verify your changes and relevent result summary. Provide instructions so it can be reproduced.
Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

- [ ] Test A
Expand Down
34 changes: 34 additions & 0 deletions .github/actions/free-up-disk-space/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: 'Free-up disk space action'
description: 'Removes non-essential tools, libraries and cached files from GitHub action runner node'

runs:
using: "composite"
steps:
- name: Free up disk space
shell: bash
run: |
echo "Disk usage before cleanup:"
df -h
# remove non-essential tools and libraries, see:
# https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
sudo rm -rf /opt/ghc
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/share/boost
# delete libraries for Android (12G), CodeQL (5.3G), PowerShell (1.3G), Swift (1.7G)
sudo rm -rf /usr/local/lib/android
sudo rm -rf "${AGENT_TOOLSDIRECTORY}/CodeQL"
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/share/swift
echo "Disk usage after cleanup:"
df -h
- name: Prune docker images
shell: bash
run: |
echo "Pruning docker images"
docker image prune -a -f
docker system df
df -h
29 changes: 0 additions & 29 deletions .github/actions/minikube-setup/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,6 @@ description: 'Sets up minikube on the github runner'
runs:
using: "composite"
steps:
- name: Free up disk space
shell: bash
run: |
echo "Disk usage before cleanup:"
df -h
# remove non-essential tools and libraries, see:
# https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
sudo rm -rf /opt/ghc
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/share/boost
# delete libraries for Android (12G), CodeQL (5.3G), PowerShell (1.3G), Swift (1.7G)
sudo rm -rf /usr/local/lib/android
sudo rm -rf "${AGENT_TOOLSDIRECTORY}/CodeQL"
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/share/swift
echo "Disk usage after cleanup:"
df -h
- name: Install kubectl
uses: azure/setup-kubectl@v3
with:
Expand All @@ -42,11 +21,3 @@ runs:
- name: Check Kubernetes pods
shell: bash
run: kubectl get pods -n kube-system

- name: Prune docker images
shell: bash
run: |
echo "Pruning docker images"
docker image prune -a -f
docker system df
df -h
16 changes: 15 additions & 1 deletion .github/workflows/e2e-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Free-up disk space
uses: ./.github/actions/free-up-disk-space

- name: Build runtime server images
run: |
./test/scripts/gh-actions/build-server-runtimes.sh predictor,transformer
Expand Down Expand Up @@ -147,6 +150,9 @@ jobs:
with:
python-version: '3.9'

- name: Free-up disk space
uses: ./.github/actions/free-up-disk-space

- name: Setup Minikube
uses: ./.github/actions/minikube-setup

Expand Down Expand Up @@ -227,6 +233,7 @@ jobs:
with:
python-version: '3.9'

- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -285,6 +292,7 @@ jobs:
with:
python-version: '3.9'

- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -337,7 +345,8 @@ jobs:
- uses: actions/setup-python@v4
with:
python-version: '3.9'


- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -390,6 +399,7 @@ jobs:
- uses: actions/setup-python@v4
with:
python-version: '3.9'
- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -442,6 +452,7 @@ jobs:
with:
python-version: '3.9'

- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -503,6 +514,7 @@ jobs:
with:
python-version: '3.9'

- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -541,6 +553,7 @@ jobs:
with:
python-version: '3.9'

- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
- uses: ./.github/actions/base-download
Expand Down Expand Up @@ -584,6 +597,7 @@ jobs:
with:
python-version: '3.9'

- uses: ./.github/actions/free-up-disk-space
- uses: ./.github/actions/minikube-setup
- uses: ./.github/actions/kserve-dep-setup
with:
Expand Down

0 comments on commit a8b9046

Please sign in to comment.