Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

config-daemon: Restart all instances of device-plugin #783

Merged

Conversation

zeeke
Copy link
Member

@zeeke zeeke commented Oct 4, 2024

When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing.
If the config-daemon executes restartDevicePluginPod() at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

Copy link

github-actions bot commented Oct 4, 2024

Thanks for your PR,
To run vendors CIs, Maintainers can use one of:

  • /test-all: To run all tests for all vendors.
  • /test-e2e-all: To run all E2E tests for all vendors.
  • /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs, Maintainers can use one of:

  • /skip-all: To skip all tests for all vendors.
  • /skip-e2e-all: To skip all E2E tests for all vendors.
  • /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
    Best regards.

@zeeke zeeke force-pushed the us/multiple-device-plugins branch from f5f470b to f2fdee5 Compare October 4, 2024 17:13
@coveralls
Copy link

coveralls commented Oct 4, 2024

Pull Request Test Coverage Report for Build 11255003478

Details

  • 11 of 25 (44.0%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.006%) to 45.017%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/daemon/daemon.go 11 25 44.0%
Totals Coverage Status
Change from base Build 11254641885: 0.006%
Covered Lines: 6663
Relevant Lines: 14801

💛 - Coveralls

pkg/daemon/daemon.go Outdated Show resolved Hide resolved
@zeeke zeeke force-pushed the us/multiple-device-plugins branch from f2fdee5 to c318a89 Compare October 7, 2024 09:39
Copy link
Collaborator

@SchSeba SchSeba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zeeke
Copy link
Member Author

zeeke commented Oct 8, 2024

Failing test

Summarizing 1 Failure:
  [FAIL] [sriov] Metrics Exporter When Prometheus operator is available [It] Metrics should have the correct labels
  /root/opr-ocp2-1/data/sriov-network-operator/sriov-network-operator/test/conformance/tests/test_exporter_metrics.go:162

is addressed in

Copy link
Collaborator

@ykulazhenkov ykulazhenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen
that there are two device plugin pods for a given node, one that is terminating, the other that is
initializing.
If the config-daemon executes `restartDevicePluginPod()` at the same time, it may kill the terminating
pod, while the initializing one will run with the old dp configuration. This may cause one or more resources
to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

Signed-off-by: Andrea Panattoni <apanatto@redhat.com>
@zeeke zeeke force-pushed the us/multiple-device-plugins branch from c318a89 to f286a04 Compare October 9, 2024 12:31
Copy link
Collaborator

@adrianchiris adrianchiris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@zeeke zeeke merged commit a85ab70 into k8snetworkplumbingwg:master Oct 9, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants