Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests for the NVIDIA GPU tests #433

Merged
merged 2 commits into from
Jun 16, 2023
Merged

Add tests for the NVIDIA GPU tests #433

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8c39dd5
misc: Add tests to the test installing of NVIDIA on instances
sayanchowdhury Apr 26, 2023
4cd37d7
CHANGELOG: Add changelog for adding nvidia tests
sayanchowdhury Jun 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- `DefaultUser` parameter when registering a test to use a user different from `core` ([#424](https://github.com/flatcar/mantle/pull/424))
- `systemd.sysext.custom-oem` for testing the activation of the OEM sysext image ([#423](https://github.com/flatcar/mantle/pull/423))
- Kubernetes 1.27 tests ([#441](https://github.com/flatcar/mantle/pull/441))
- Add tests for testing the installation/integrity of the NVIDIA drivers ([#433](https://github.com/flatcar/mantle/pull/433))

### Changed

Expand Down Expand Up @@ -67,7 +68,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Ignition v3 support and tests ([#301](https://github.com/flatcar-linux/mantle/pull/301), [#311](https://github.com/flatcar-linux/mantle/pull/311))
- Butane config support ([#318](https://github.com/flatcar-linux/mantle/pull/318))
- GCP: support testing with GVNIC ([#322](https://github.com/flatcar-linux/mantle/pull/322))
- `networkd` Ignition translation test ([#344](https://github.com/flatcar-linux/mantle/pull/334))
- `networkd` Ignition translation test ([#344](https://github.com/flatcar-linux/mantle/pull/334))
- kola test `cl.misc.falco` that tests falco kmod building ([#339](https://github.com/flatcar-linux/mantle/pull/339))
- Kubernetes test for release 1.24.1 ([#337](https://github.com/flatcar-linux/mantle/pull/337))
- Added storage abstraction for Equinix Metal tests (SSH can be used in addition of Google Cloud Storage) ([#340](https://github.com/flatcar-linux/mantle/pull/340))
Expand Down
52 changes: 52 additions & 0 deletions kola/tests/misc/nvidia.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
package misc

import (
"bytes"
"fmt"
"time"

"github.com/coreos/pkg/capnslog"
"github.com/flatcar/mantle/kola"
"github.com/flatcar/mantle/kola/cluster"
"github.com/flatcar/mantle/kola/register"
"github.com/flatcar/mantle/util"
)

const (
CmdTimeout = time.Second * 300
)

var plog = capnslog.NewPackageLogger("github.com/flatcar/mantle", "kola/tests/misc")

func init() {
register.Register(&register.Test{
Name: "cl.misc.nvidia",
Run: verifyNvidiaInstallation,
ClusterSize: 1,
Distros: []string{"cl"},
// This test is to test the NVIDIA installation, limited to AZURE for now
Platforms: []string{"azure"},
Architectures: []string{"amd64"},
Flags: []register.Flag{register.NoEnableSelinux},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance to track the avc message? (for issue tracking and/or documentation purpose?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I will need to search the logs, if it's not there then I can trigger a local/Jenkins build to track the avc messages for tracking on a issue.

Copy link
Contributor

@tormath1 tormath1 Jun 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should see it in the journal logs (before disabling the SELinux flags)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I meant searching the journal logs if they are not deleted on bincache.

})
}

func verifyNvidiaInstallation(c cluster.TestCluster) {
if kola.AzureOptions.Size != "Standard_NC6s_v3" {
c.Skip("skipping due to wrong instance size")
}
m := c.Machines()[0]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With

Suggested change
m := c.Machines()[0]
if kola.AzureOptions.Size != "Standard_NC6s_v3" {
c.Skip("skipping due to wrong instance size")
}
m := c.Machines()[0]

I think we could do this in scripts:

diff --git a/ci-automation/vendor-testing/azure.sh b/ci-automation/vendor-testing/azure.sh
index 17081b3598..c57210d651 100755
--- a/ci-automation/vendor-testing/azure.sh
+++ b/ci-automation/vendor-testing/azure.sh
@@ -74,7 +74,7 @@ query_kola_tests() {
     kola list --platform=azure --filter "${@}"
 }
 
-other_instance_types=()
+other_instance_types=('Standard_NC6s_v3')
 if [[ "${CIA_ARCH}" = 'amd64' ]]; then
     other_instance_types+=('V1')
 fi
@@ -85,6 +85,6 @@ run_kola_tests_on_instances \
     "${CIA_FIRST_RUN}" \
     "${other_instance_types[@]}" \
     '--' \
-    'cl.internet' \
+    'cl.internet' 'cl.misc.nvidia' \
     '--' \
     "${@}"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to keep this check, we could even move it to the SkipFunc (as kola.AzureOptions.Size should be accessible)


nvidiaStatusRetry := func() error {
out, err := c.SSH(m, "systemctl is-active nvidia.service")
if !bytes.Contains(out, []byte("inactive")) {
return fmt.Errorf("nvidia.service: %q: %v", out, err)
}
Comment on lines +41 to +44
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI, you could do this way:

Suggested change
out, err := c.SSH(m, "systemctl is-active nvidia.service")
if !bytes.Contains(out, []byte("inactive")) {
return fmt.Errorf("nvidia.service: %q: %v", out, err)
}
_, err := c.SSH(m, "systemctl --quiet is-active nvidia.service")
return err

we just want to assert if the unit is active or not, if the unit is active with --quiet it will simply return 0.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also _ = c.MustSSH(…) which fails the test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct but in this case we don't want to fail the test as we are in a retry loop no?

return nil
}

if err := util.Retry(40, 15*time.Second, nvidiaStatusRetry); err != nil {
c.Fatal(err)
}
c.AssertCmdOutputContains(m, "/opt/bin/nvidia-smi", "Tesla")
}