Export image name and env variables as prometheus labels #780

Marmelatze · 2015-06-23T16:42:29Z

I had the same issue as described in #546. The names from containers created with Mesos/Marathon are not telling something about the app this container is running. With this PR any env variable can be exported as prometheus label. I also added the current image a container is running for monitoring the performance of some images.

I hope this use case isn't to specific.

cadvisorJenkinsBot · 2015-06-23T16:43:18Z

Can one of the admins verify this patch?

vmarmol · 2015-06-29T18:13:22Z

ok to test

vmarmol · 2015-06-29T21:40:35Z

@Marmelatze this looks good for the most part. I know image has been requested for a while. Env variables I'm not so sure we want to do however...what is the intended usecase? Just afraid that we're exposing data that is tricky to use and easy to abuse.

Marmelatze · 2015-06-30T11:28:22Z

I have the same use case as the guy in #546. I have some Mesos/Marathon hosts and Mesos assigns some random id as container name (I expect Mesos/kubernetes and Mesos/Swarm to do the same).

In my dashboard (cadvisor on every host -> prometheus -> grafana) I can see that a container uses 100% CPU but I only can see the host its running on and the name (mesos-1234-5678...). To determine what application this container is running I have to SSH into that host and inspect the container. There I can see the mesos task-id, the marathon app name and version.

The reverse way is also time consuming: I have more than one instance of this app running and I want to know how these are performing (CPU, RAM, Network). So I have to check Marathon or Mesos to get a list of hosts, that app is deployed to. Than I have to ssh into these. List all containers and look which could be my app. Inspect these to be sure. Go back to the dashboard and filter for the names.

I know some sensitive data, like passwords and access tokens, are stored in env variables, so I made this optional and the exposed variables configurable.

For Mesos/Marathon it would be:
--docker_metadata_env=MESOS_TASK_ID,MARATHON_APP_ID,MARATHON_APP_VERSION

vmarmol · 2015-07-07T01:08:24Z

container/docker/handler.go

@@ -71,6 +71,10 @@ type dockerContainerHandler struct {

 	// Metadata labels associated with the container.
 	labels map[string]string
+
+	image string


nit: add comments for these

Marmelatze · 2015-07-07T10:07:23Z

Thanks for the feedback. With all container labels exported as prometheus labels it would also fix #688

vmarmol · 2015-07-08T06:17:24Z

LGTM, thanks @Marmelatze! Can you rebase and repush to kick the CI?

googlebot · 2015-07-08T07:35:18Z

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

googlebot · 2015-07-08T07:37:55Z

CLAs look good, thanks!

MrMMorris · 2015-07-18T06:36:50Z

@Marmelatze thanks so much for taking this on. It is basically a requirement when using Prometheus with Consul if I want to populate graphs dynamically. Now I just have to LABEL my images and it will Just Work™ 😀

Marmelatze · 2015-07-21T09:34:28Z

Rebased to current master and tests seem to pass

gtmtech · 2015-07-27T17:24:44Z

+1 I have exactly the same usecase (mesos, classifying containers by appname for aggregation and averaging reasons across a cluster). Hopefully this can be merged ASAP

jimmidyson · 2015-08-25T09:58:29Z

Would really like to see this merged, although only want the image name personally.

rjnagal · 2015-08-25T16:18:39Z

@Marmelatze Let's split this up. I think image name makes a lot of sense. We can merge that quickly.

Do the containers in Mesos use labels - something more structured than env variables and intended for exporting? We can pick up runtime image labels too for identifying the apps.

jimmidyson · 2015-08-28T21:49:45Z

Image name added in #869.

Marmelatze · 2015-09-02T22:23:04Z

OK thanks, but unfortunately mesos doesn't seem to set labels or something else. Just the env variables.

jzaefferer · 2015-09-15T14:40:24Z

@Marmelatze could you rebase this to remove the changes already introduced by #869? Unless you've found an alternative solution by now; if so, could you share?

@rjnagal was the answer to your question satisfying? We've got the same usecase as @Marmelatze and would like to see this addressed.

MikeMichel · 2015-09-15T14:44:37Z

+1 for rebase

koboltmarky · 2015-09-15T14:46:29Z

+1 for rebase

jzaefferer · 2015-09-15T15:00:12Z

I actually tried rebasing myself, but with my very limited understanding of Go, I couldn't figure out how to resolve the conflicts in metrics/prometheus.go.

Marmelatze · 2015-09-16T15:24:59Z

Rebased to current master and removed the image name parts.

jzaefferer · 2015-09-16T15:35:54Z

The commit message still mentions the image, but that should be trivial to fix when landing this.

MikeMichel · 2015-09-17T10:57:13Z

@Marmelatze awesome, works perfect.

thx

jimmidyson · 2015-09-24T08:55:24Z

metrics/prometheus.go

@@ -493,6 +493,11 @@ func (c *PrometheusCollector) collectContainersInfo(ch chan<- prometheus.Metric)
 		}
 		baseLabelValues := []string{id, name, image}[:len(baseLabels)]

+		for labelKey, labelValue := range container.Spec.Labels {
+			baseLabels = append(baseLabels, labelKey)


labelKey will need to be sanitized as Prometheus labels need to match this regexp.

You can copy the sanitize method from here & please add a test for this.

vishh · 2015-10-08T18:32:04Z

@Marmelatze: Can you rebase?

dqminh · 2015-10-27T10:02:49Z

@vishh @Marmelatze im also interested to see this and ability to specify container labels as tags. Let me know if you want some help with this ( rebase / take over etc. )

deedubs · 2015-11-04T21:24:02Z

Is anyone running this yet?

MikeMichel · 2015-11-04T21:47:43Z

yes https://hub.docker.com/r/mikemichel/cadvisor/tags/

docker run --name cadvisor --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys
:ro --volume=/mnt/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true   mikemichel/cadvisor:metaenvs --docker_metadata_env=MESOS_TASK_ID,MARATHO
N_APP_ID,MARATHON_APP_VERSION

deedubs · 2015-11-13T20:23:18Z

@rjnagal feels like the security concerns are minimized by only exporting whitelisted ENV vars no?

scalp42 · 2015-11-17T20:48:57Z

@jimmidyson or @vmarmol Any chance to see this merged now that we have to whitelist the env vars?

jimmidyson · 2015-11-17T20:55:52Z

@scalp42 If you can rebase I'll try to find time to review tomorrow. I have no security concerns now that you have added the whitelist for env vars.

scalp42 · 2015-11-17T20:57:47Z

cc @Marmelatze 🍶

rjnagal · 2015-11-17T20:58:07Z

This looks fine with the whitelist. Can you rebase? Thanks.

k8s-bot · 2015-12-11T21:46:16Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

scalp42 · 2015-12-11T21:49:01Z

@Marmelatze any chance ? 🚨

k8s-bot · 2015-12-12T01:35:05Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

dqminh · 2015-12-15T18:29:57Z

@vishh @rjnagal @jimmidyson i like to use this patch. Should i take over and do a rebase on behalf of @Marmelatze ?

vishh · 2015-12-15T18:49:03Z

Yeah. Feel free to work on top of existing patches.

On Tue, Dec 15, 2015 at 10:34 AM, Daniel, Dao Quang Minh <
notifications@github.com> wrote:

@vishh https://github.com/vishh @rjnagal https://github.com/rjnagal
@jimmidyson https://github.com/jimmidyson i like to use this patch.
Should i take over and do a rebase on behalf of @Marmelatze
https://github.com/Marmelatze ?

—
Reply to this email directly or view it on GitHub
#780 (comment).

Carry #780: Export env variables as prometheus labels

jimmidyson · 2016-01-14T15:19:26Z

Fixed by #780.

r0fls · 2017-10-17T03:36:04Z

Fixed by #780.

Well hey now, that's a circular reference! 😄 Do you mean #1023 ?

vmarmol reviewed Jul 7, 2015
View reviewed changes

Marmelatze force-pushed the metadata_env branch 2 times, most recently from 6a160c8 to 6e404da Compare July 7, 2015 09:52

Marmelatze force-pushed the metadata_env branch from 6e404da to f3ac8d8 Compare July 8, 2015 07:35

Marmelatze force-pushed the metadata_env branch from f3ac8d8 to 588a127 Compare July 8, 2015 07:37

Marmelatze force-pushed the metadata_env branch from 588a127 to 57e526d Compare July 21, 2015 09:14

Marmelatze force-pushed the metadata_env branch 2 times, most recently from cc5b3e0 to 19967f0 Compare August 4, 2015 12:53

jimmidyson mentioned this pull request Aug 28, 2015

Add image name to container spec & prometheus metrics #869

Merged

Export image name and env variables as prometheus labels

8940433

Marmelatze force-pushed the metadata_env branch from 19967f0 to 8940433 Compare September 16, 2015 15:18

jimmidyson reviewed Sep 24, 2015
View reviewed changes

dqminh mentioned this pull request Dec 16, 2015

Carry #780: Export env variables as prometheus labels #1023

Merged

vishh added a commit that referenced this pull request Jan 13, 2016

Merge pull request #1023 from dqminh/metadata_env

9cfc8ce

Carry #780: Export env variables as prometheus labels

jimmidyson closed this Jan 14, 2016

Export image name and env variables as prometheus labels #780

Export image name and env variables as prometheus labels #780

Conversation

Marmelatze commented Jun 23, 2015

cadvisorJenkinsBot commented Jun 23, 2015

vmarmol commented Jun 29, 2015

vmarmol commented Jun 29, 2015

Marmelatze commented Jun 30, 2015

vmarmol Jul 7, 2015

Choose a reason for hiding this comment

Marmelatze commented Jul 7, 2015

vmarmol commented Jul 8, 2015

googlebot commented Jul 8, 2015

googlebot commented Jul 8, 2015

MrMMorris commented Jul 18, 2015

Marmelatze commented Jul 21, 2015

gtmtech commented Jul 27, 2015

jimmidyson commented Aug 25, 2015

rjnagal commented Aug 25, 2015

jimmidyson commented Aug 28, 2015

Marmelatze commented Sep 2, 2015

jzaefferer commented Sep 15, 2015

MikeMichel commented Sep 15, 2015

koboltmarky commented Sep 15, 2015

jzaefferer commented Sep 15, 2015

Marmelatze commented Sep 16, 2015

jzaefferer commented Sep 16, 2015

MikeMichel commented Sep 17, 2015

jimmidyson Sep 24, 2015

Choose a reason for hiding this comment

vishh commented Oct 8, 2015

dqminh commented Oct 27, 2015

deedubs commented Nov 4, 2015

MikeMichel commented Nov 4, 2015

deedubs commented Nov 13, 2015

scalp42 commented Nov 17, 2015

jimmidyson commented Nov 17, 2015

scalp42 commented Nov 17, 2015

rjnagal commented Nov 17, 2015

k8s-bot commented Dec 11, 2015

scalp42 commented Dec 11, 2015

k8s-bot commented Dec 12, 2015

dqminh commented Dec 15, 2015

vishh commented Dec 15, 2015

jimmidyson commented Jan 14, 2016

r0fls commented Oct 17, 2017