Stackdriver metrics invalid and don't match values in heapster

[acondrat]

Hello,

We upgraded k8s from 1.7.1 to 1.7.6 and started to see strange cpu/memory usage being reported in stackdriver.

Heapster seems to report accurate values however the graph shows something else (please see bellow)

root@busybox-3618177290-dsqz4:/# curl http://heapster/api/v1/model/namespaces/production/pods/integration-ftp-1819114418-9g32f/metrics/memory/usage
{
  "metrics": [
   {
    "timestamp": "2017-09-26T07:29:00Z",
    "value": 1008308224
   },
   {
    "timestamp": "2017-09-26T07:30:00Z",
    "value": 1008709632
   },
   {
    "timestamp": "2017-09-26T07:31:00Z",
    "value": 1009090560
   },
   {
    "timestamp": "2017-09-26T07:32:00Z",
    "value": 1009471488
   },
   {
    "timestamp": "2017-09-26T07:33:00Z",
    "value": 1009848320
   },
   {
    "timestamp": "2017-09-26T07:34:00Z",
    "value": 1010237440
   },
   {
    "timestamp": "2017-09-26T07:35:00Z",
    "value": 1010618368
   },
   {
    "timestamp": "2017-09-26T07:36:00Z",
    "value": 1010999296
   },
   {
    "timestamp": "2017-09-26T07:37:00Z",
    "value": 1011396608
   },
   {
    "timestamp": "2017-09-26T07:38:00Z",
    "value": 1011798016
   },
   {
    "timestamp": "2017-09-26T07:39:00Z",
    "value": 1012166656
   },
   {
    "timestamp": "2017-09-26T07:40:00Z",
    "value": 1008861184
   },
   {
    "timestamp": "2017-09-26T07:41:00Z",
    "value": 1008861184
   },
   {
    "timestamp": "2017-09-26T07:42:00Z",
    "value": 1008861184
   },
   {
    "timestamp": "2017-09-26T07:43:00Z",
    "value": 1008861184
   }
  ],
  "latestTimestamp": "2017-09-26T07:43:00Z"
 }

[piosz]

@acondrat thanks for reporting. We're looking into this.

cc @loburm @x13n

[piosz]

Indeed there is an issue: we export evictable memory to Stackdriver instead of non-evictable one. @loburm is going to fix this.

[piosz]

https://github.com/kubernetes/heapster/blob/master/metrics/sinks/stackdriver/stackdriver.go#L412

[loburm]

This should be fixed by #1822. Soon we will cherrypick 1.4 release and in the week it's going to released (both heapster and new kubernetes version).