Inconsistent & stale metrics reported for a docker container which was restarted

[amolfound]

Using:
cadvisor v36
docker image: debian8

While querying the /metrics endpoint i receive inconsistent entries for a container which was restarted a while back. For eg. if i query the metrics endpoint 10 times. 5 times i would receive 1st one and other 5 times the second one
`49444.206645393 1607337449406 ---> timestamp corresponds to 7 December 2020

40299.534580353 1600434184085 ---> timestamp corresponds to 18 September 2020`

the first one corresponds to entry from the latest new container. The second is from the instance which was shutdown before the restart

cgroup parent of the container
"CgroupParent": "/system.slice/<service-name>",

in the code we use 2minutes as maxage while creating the inmemorycache here : https://github.com/google/cadvisor/blob/master/cache/memory/memory.go
which is used in the cadvisor manger.

The above config setting should ideally not allow entries older than 2minutes. Here the entry is months old.

[iwankgb]

@amolfound, can you check if #2800 fixed your problem, please?

[amolfound]

@iwankgb : i believe #2800 is a different issue.

My issue i infered is caused by stale cgroups inside cgroup parent of the service.
"CgroupParent": "/system.slice/",

So randomly one of these entries is picked up by cadvisor at a time which is causing the issue

[mrliptontea]

I wonder if #2844 is related? It's a bit different problem though.