Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values #3412

Closed
sofat1989 opened this issue Jul 15, 2019 · 0 comments · Fixed by containerd/cgroups#93 or moby/moby#39713
Labels

Comments

@sofat1989
Copy link

When we get the metrics of containerd, we get this kind of error

collected metric container_blkio_io_serviced_recursive_total label:<name:"container_id" value:"7d31271cc4fe549a789d0c6a2b3070e2eca1b5123776315a1d78e55ff40f5c84" > label:<name:"device" value:"/dev/sda" > label:<name:"major" value:"8" > label:<name:"minor" value:"0" > label:<name:"namespace" value:"k8s.io" > label:<name:"op" value:"Total" > gauge:<value:83 > was collected before with the same name and label values

After adding more logs, we find there are duplicated blkio entries returned by the func https://github.com/containerd/containerd/blob/master/vendor/github.com/containerd/cgroups/blkio.go#L75

[&BlkIOEntry{Op:Read,Device:/dev/sda,Major:8,Minor:0,Value:0,}
 &BlkIOEntry{Op:Write,Device:/dev/sda,Major:8,Minor:0,Value:1,}
 &BlkIOEntry{Op:Sync,Device:/dev/sda,Major:8,Minor:0,Value:0,} 
 &BlkIOEntry{Op:Async,Device:/dev/sda,Major:8,Minor:0,Value:1,}
 &BlkIOEntry{Op:Total,Device:/dev/sda,Major:8,Minor:0,Value:1,} 
   &BlkIOEntry{Op:Read,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
    &BlkIOEntry{Op:Write,Device:/dev/sdb,Major:8,Minor:16,Value:0,} 
    &BlkIOEntry{Op:Sync,Device:/dev/sdb,Major:8,Minor:16,Value:35,} 
    &BlkIOEntry{Op:Async,Device:/dev/sdb,Major:8,Minor:16,Value:0,} 
    &BlkIOEntry{Op:Total,Device:/dev/sdb,Major:8,Minor:16,Value:35,} 
    &BlkIOEntry{Op:Read,Device:/dev/dm-0,Major:253,Minor:0,Value:35,}
     &BlkIOEntry{Op:Write,Device:/dev/dm-0,Major:253,Minor:0,Value:0,} 
     &BlkIOEntry{Op:Sync,Device:/dev/dm-0,Major:253,Minor:0,Value:35,} 
     &BlkIOEntry{Op:Async,Device:/dev/dm-0,Major:253,Minor:0,Value:0,} 
     &BlkIOEntry{Op:Total,Device:/dev/dm-0,Major:253,Minor:0,Value:35,}
     &BlkIOEntry{Op:Read,Device:/dev/sda,Major:8,Minor:0,Value:0,} 
     &BlkIOEntry{Op:Write,Device:/dev/sda,Major:8,Minor:0,Value:1,} 
    &BlkIOEntry{Op:Sync,Device:/dev/sda,Major:8,Minor:0,Value:0,} 
    &BlkIOEntry{Op:Async,Device:/dev/sda,Major:8,Minor:0,Value:1,} 
    &BlkIOEntry{Op:Total,Device:/dev/sda,Major:8,Minor:0,Value:1,} 
    &BlkIOEntry{Op:Read,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
     &BlkIOEntry{Op:Write,Device:/dev/sdb,Major:8,Minor:16,Value:0,} 
     &BlkIOEntry{Op:Sync,Device:/dev/sdb,Major:8,Minor:16,Value:35,} 
     &BlkIOEntry{Op:Async,Device:/dev/sdb,Major:8,Minor:16,Value:0,} 
     &BlkIOEntry{Op:Total,Device:/dev/sdb,Major:8,Minor:16,Value:35,}]

The root cause:
we have both throttle.io_serviced and io_serviced_recursive, so the IoServicedRecursive will be added twice

{
			name:  "throttle.io_serviced",
			entry: &stats.Blkio.IoServicedRecursive,
		},

{
				name:  "io_serviced_recursive",
				entry: &stats.Blkio.IoServicedRecursive,
			},
@sofat1989 sofat1989 changed the title ollected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values Jul 15, 2019
thaJeztah added a commit to thaJeztah/docker that referenced this issue Aug 26, 2019
full diff:  github.com/containerd/cgroups containerd/cgroups@4994991...c4b9ac5

changes included:

  - containerd/cgroups#81 Add network stats
    - addresses containerd/cgroups#80 Add network metrics
  - containerd/cgroups#85 Fix cgroup hugetlb size prefix for kB
    - addresses kubernetes/kubernetes#77169 Permission denied on hugetlb due to wrong filename
    - relates to opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB
  - containerd/cgroups#88 cgroups: fix MoveTo function fail problem
  - containerd/cgroups#92 fixed an issue with invalid soft memory limits
  - containerd/cgroups#93 avoid adding io_serviced and io_service_bytes duplicately
    - fixes containerd/containerd#3412 collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
solsson added a commit to Yolean/ystack that referenced this issue Oct 1, 2019
containerd/containerd#3412

k3s has upgraded to 1.3.0 in master since 834f7df7ecb8eb8be27ebb300f2bcb23f24397b2
and ofc we don't know when GKE's _CONTAINERD nodes will upgrade.
dmcgowan pushed a commit to thaJeztah/docker that referenced this issue Oct 4, 2019
full diff:  github.com/containerd/cgroups containerd/cgroups@4994991...c4b9ac5

changes included:

  - containerd/cgroups#81 Add network stats
    - addresses containerd/cgroups#80 Add network metrics
  - containerd/cgroups#85 Fix cgroup hugetlb size prefix for kB
    - addresses kubernetes/kubernetes#77169 Permission denied on hugetlb due to wrong filename
    - relates to opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB
  - containerd/cgroups#88 cgroups: fix MoveTo function fail problem
  - containerd/cgroups#92 fixed an issue with invalid soft memory limits
  - containerd/cgroups#93 avoid adding io_serviced and io_service_bytes duplicately
    - fixes containerd/containerd#3412 collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this issue Oct 8, 2019
full diff:  github.com/containerd/cgroups containerd/cgroups@4994991...c4b9ac5

changes included:

  - containerd/cgroups#81 Add network stats
    - addresses containerd/cgroups#80 Add network metrics
  - containerd/cgroups#85 Fix cgroup hugetlb size prefix for kB
    - addresses kubernetes/kubernetes#77169 Permission denied on hugetlb due to wrong filename
    - relates to opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB
  - containerd/cgroups#88 cgroups: fix MoveTo function fail problem
  - containerd/cgroups#92 fixed an issue with invalid soft memory limits
  - containerd/cgroups#93 avoid adding io_serviced and io_service_bytes duplicately
    - fixes containerd/containerd#3412 collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 0af1099a81861dd0269adad53bdfb387b5c78f39
Component: engine
burnMyDread pushed a commit to burnMyDread/moby that referenced this issue Oct 21, 2019
full diff:  github.com/containerd/cgroups containerd/cgroups@4994991...c4b9ac5

changes included:

  - containerd/cgroups#81 Add network stats
    - addresses containerd/cgroups#80 Add network metrics
  - containerd/cgroups#85 Fix cgroup hugetlb size prefix for kB
    - addresses kubernetes/kubernetes#77169 Permission denied on hugetlb due to wrong filename
    - relates to opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB
  - containerd/cgroups#88 cgroups: fix MoveTo function fail problem
  - containerd/cgroups#92 fixed an issue with invalid soft memory limits
  - containerd/cgroups#93 avoid adding io_serviced and io_service_bytes duplicately
    - fixes containerd/containerd#3412 collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: zach <Zachary.Joyner@linux.com>
solsson added a commit to Yolean/ystack that referenced this issue Nov 19, 2019
containerd/containerd#3412

k3s has upgraded to 1.3.0 in master since 834f7df7ecb8eb8be27ebb300f2bcb23f24397b2
and ofc we don't know when GKE's _CONTAINERD nodes will upgrade.
thaJeztah added a commit to thaJeztah/docker that referenced this issue Dec 3, 2019
full diff:  github.com/containerd/cgroups containerd/cgroups@4994991...c4b9ac5

changes included:

  - containerd/cgroups#81 Add network stats
    - addresses containerd/cgroups#80 Add network metrics
  - containerd/cgroups#85 Fix cgroup hugetlb size prefix for kB
    - addresses kubernetes/kubernetes#77169 Permission denied on hugetlb due to wrong filename
    - relates to opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB
  - containerd/cgroups#88 cgroups: fix MoveTo function fail problem
  - containerd/cgroups#92 fixed an issue with invalid soft memory limits
  - containerd/cgroups#93 avoid adding io_serviced and io_service_bytes duplicately
    - fixes containerd/containerd#3412 collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0af1099)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this issue Jan 23, 2020
full diff:  github.com/containerd/cgroups containerd/cgroups@4994991...c4b9ac5

changes included:

  - containerd/cgroups#81 Add network stats
    - addresses containerd/cgroups#80 Add network metrics
  - containerd/cgroups#85 Fix cgroup hugetlb size prefix for kB
    - addresses kubernetes/kubernetes#77169 Permission denied on hugetlb due to wrong filename
    - relates to opencontainers/runc#2065 Fix cgroup hugetlb size prefix for kB
  - containerd/cgroups#88 cgroups: fix MoveTo function fail problem
  - containerd/cgroups#92 fixed an issue with invalid soft memory limits
  - containerd/cgroups#93 avoid adding io_serviced and io_service_bytes duplicately
    - fixes containerd/containerd#3412 collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0af1099a81861dd0269adad53bdfb387b5c78f39)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: fe00613d0686a47519048a190f1138504a2cdedc
Component: engine
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
1 participant