Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add network stats for Windows nodes and containers #74788

Merged
merged 3 commits into from
Mar 8, 2019

Conversation

feiskyer
Copy link
Member

@feiskyer feiskyer commented Mar 1, 2019

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Add network stats for Windows nodes and containers.

Which issue(s) this PR fixes:

Fixes #74101

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add network stats for Windows nodes and containers

/sig windows
/sig node
/assign @yujuhong @PatrickLang @michmike

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/windows Categorizes an issue or PR as relevant to SIG Windows. labels Mar 1, 2019
@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Mar 1, 2019
@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Mar 1, 2019
@feiskyer feiskyer added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 1, 2019
@feiskyer
Copy link
Member Author

feiskyer commented Mar 1, 2019

/milestone v1.14

@k8s-ci-robot k8s-ci-robot added this to the v1.14 milestone Mar 1, 2019
@michmike
Copy link
Contributor

michmike commented Mar 1, 2019

@benmoss, @aserdean can you PTAL

pkg/kubelet/stats/BUILD Outdated Show resolved Hide resolved
@michmike
Copy link
Contributor

michmike commented Mar 1, 2019

looks good overall, even though there is a lot of duplication of code that would have been better in an abstraction. i noted a small bug in there.

have you tested this and how does performance relate to the previous code?
thanks for all the work!

@feiskyer
Copy link
Member Author

feiskyer commented Mar 4, 2019

Fixed the bug now. PTAL.

have you tested this and how does performance relate to the previous code?

Yep, kubelet consumes much less CPU now (kubelet 1-4% from my cluster).

@feiskyer
Copy link
Member Author

feiskyer commented Mar 4, 2019

/retest

// listContainerNetworkStats returns the network stats of all the running containers.
func (p *criStatsProvider) listContainerNetworkStats() (map[string]*statsapi.NetworkStats, error) {
// Always return nil for unsupported platforms.
return nil, nil
Copy link
Member

@benmoss benmoss Mar 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to yield a nil pointer exception on unsupported platforms on cri_stats_provider.go:192. Not sure if it matters, but maybe this could return an error like https://github.com/kubernetes/kubernetes/blob/48def74e641f8dff50fcf7c33c2a412593e2fede/pkg/util/mount/exec_mount_unsupported.go ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In such case, we still want to continue other stats logic in callers. So nil is returned here instead of errors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way the code is written right now, it's hard to tell whether nil is expected or not. @feiskyer is there any existing e2e test that can be modified to validate whether networkstats is present? Maybe the summary API test?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Linx, there always have network stats, but for Windows, there doesn't before this PR. After this, both Windows and Linux containers would have network stats.

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 6, 2019
@michmike
Copy link
Contributor

michmike commented Mar 6, 2019

/lgtm

@michmike
Copy link
Contributor

michmike commented Mar 6, 2019

/approve

@michmike
Copy link
Contributor

michmike commented Mar 6, 2019

/assign @yujuhong

@yujuhong
Copy link
Contributor

yujuhong commented Mar 6, 2019

/hold
Reviewing and I will some questions (will post soon). Holding this PR to avoid it from being merged accidentally.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 6, 2019
@yujuhong
Copy link
Contributor

yujuhong commented Mar 6, 2019

/test pull-kubernetes-e2e-containerd-gce

@PatrickLang
Copy link
Contributor

PatrickLang commented Mar 6, 2019

I'm only seeing one network now.

    "network": {
      "time": "2019-03-06T19:42:34Z",
      "name": "e76e4a10-eth0",
      "rxBytes": 146563,
      "txBytes": 491768,
      "interfaces": [
        {
          "name": "e76e4a10-eth0",
          "rxBytes": 146563,
          "txBytes": 491768
        }
      ]
    },

That matches the adapter name from the Azure-CNI logs:

azure-vnet.log:704:2019/03/06 19:38:59 [cni-net] Creating endpoint e76e4a10-eth0.
azure-vnet.log:705:2019/03/06 19:38:59 [net] Creating endpoint &{Id:e76e4a10-eth0
ContainerID:e76e4a108a64edbac93f89f286262fea94221c3ecb1f7854909fc7ddd275e96d
NetNsPath:none IfName:eth0 SandboxKey: IfIndex:0 MacAddress:
DNS:{Suffix:default.svc.cluster.local Servers:[10.0.0.10 168.63.129.16]} ...

Looks good!

@k8s-ci-robot
Copy link
Contributor

@feiskyer: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-containerd-gce ee5fd0a9f4d1b797a72c9109b63acef2e1bfbbbe link /test pull-kubernetes-e2e-containerd-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Copy link
Contributor

@yujuhong yujuhong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CRI stats path is not tested by dockershim+linux AFAIK. I can't get the presubmit containerd job to run right now, so we'll need to check if post-submit CI will be affected by this change.

Other than that, this mostly looks good with questions/comments.

The summary api test is in the node e2e suite (cannot be run by windows yet). I think we should port the test over to a regular e2e, and do some sanity check to see what metrics are present. I don't think it should block this PR, but that should be addressed afterwards.

pkg/kubelet/stats/cri_stats_provider_others.go Outdated Show resolved Hide resolved
pkg/kubelet/stats/cri_stats_provider_windows.go Outdated Show resolved Hide resolved
pkg/kubelet/stats/cri_stats_provider_windows.go Outdated Show resolved Hide resolved
pkg/kubelet/stats/cri_stats_provider_windows.go Outdated Show resolved Hide resolved
pkg/kubelet/stats/cri_stats_provider.go Show resolved Hide resolved
pkg/kubelet/stats/cri_stats_provider.go Show resolved Hide resolved
// listContainerNetworkStats returns the network stats of all the running containers.
func (p *criStatsProvider) listContainerNetworkStats() (map[string]*statsapi.NetworkStats, error) {
// Always return nil for unsupported platforms.
return nil, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way the code is written right now, it's hard to tell whether nil is expected or not. @feiskyer is there any existing e2e test that can be modified to validate whether networkstats is present? Maybe the summary API test?

pkg/kubelet/stats/helper.go Show resolved Hide resolved
pkg/kubelet/winstats/network_stats.go Outdated Show resolved Hide resolved
go wait.Forever(func() {
p.collectMetricsData(cpuCounter, memWorkingSetCounter, memCommittedBytesCounter)
p.collectMetricsData(cpuCounter, memWorkingSetCounter, memCommittedBytesCounter, networkAdapterCounter)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the perfCounterUpdatePeriod (1s) still reasonable after adding more metrics?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1s should still work. And for network stats, there are some metrics per second. To ensure the cumulative data correctly, 1s should be used.

@PatrickLang
Copy link
Contributor

/retest pull-kubernetes-e2e-containerd-gce

@yujuhong
Copy link
Contributor

yujuhong commented Mar 7, 2019

pull-kubernetes-e2e-containerd-gce

We haven't triggered this job for a while. Seems like it doesn't work anymore. We'll watch the post-submit CI instead.

@feiskyer
Copy link
Member Author

feiskyer commented Mar 7, 2019

The summary api test is in the node e2e suite (cannot be run by windows yet). I think we should port the test over to a regular e2e, and do some sanity check to see what metrics are present. I don't think it should block this PR, but that should be addressed afterwards.

@PatrickLang Could you open an issue for tracking this? We should enable node e2e tests for Windows.

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2019
@PatrickLang
Copy link
Contributor

/LGTM
@yujuhong can you approve?

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2019
@yujuhong
Copy link
Contributor

yujuhong commented Mar 7, 2019

/hold cancel
/lgtm
/approve

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 7, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, michmike, yujuhong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2019
@k8s-ci-robot k8s-ci-robot merged commit fbc5150 into kubernetes:master Mar 8, 2019
@feiskyer feiskyer deleted the windows-network-stats branch March 8, 2019 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/windows Categorizes an issue or PR as relevant to SIG Windows. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

kubelet on windows consuming 100% cpu running multiple powershell process withGet-NetAdapterStatistics
7 participants