Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check that metric_relabel_configs has non-nil configuration in runtime config #3868

Merged
merged 2 commits into from
Jan 9, 2023

Conversation

pstibrany
Copy link
Member

@pstibrany pstibrany commented Jan 6, 2023

What this PR does

This PR fixes panic in distributor which happens when user writes invalid metric_relabel_configs into overrides like this:

overrides:
  anonymous:
    metric_relabel_configs:
      -

This was accepted as valid runtime config file, but metric relabel configs translated to []*relabel.Config{nil}. When distributor uses this, it panics (line numbers match main at ef085cd):

2023/01/06 09:42:40 http: panic serving 172.18.0.5:42300: runtime error: invalid memory address or nil pointer dereference
goroutine 15715 [running]:
net/http.(*conn).serve.func1()
	/opt/homebrew/Cellar/go/1.19.2/libexec/src/net/http/server.go:1850 +0x120
panic({0x1fbfd80, 0x37a5770})
	/opt/homebrew/Cellar/go/1.19.2/libexec/src/runtime/panic.go:890 +0x264
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
	/Users/peter/Grafana/mimir/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:150 +0x1a0
panic({0x1fbfd80, 0x37a5770})
	/opt/homebrew/Cellar/go/1.19.2/libexec/src/runtime/panic.go:890 +0x264
github.com/prometheus/prometheus/model/relabel.relabel({0x40019301c0, 0xb, 0xe}, 0x0, 0x400082f540)
	/Users/peter/Grafana/mimir/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go:222 +0x7c
github.com/prometheus/prometheus/model/relabel.Process({0x40019301c0, 0xb, 0xe}, {0x4001fb40d8, 0x1, 0x1})
	/Users/peter/Grafana/mimir/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go:211 +0xb0
github.com/grafana/mimir/pkg/distributor.(*Distributor).prePushRelabelMiddleware.func1({0x2945388, 0x40005c5ad0}, 0x4000f97680)
	/Users/peter/Grafana/mimir/pkg/distributor/distributor.go:801 +0x2ac
...

Which issue(s) this PR fixes or relates to

No issue created.

Checklist

  • Tests updated
  • [na] Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

…e nil configs.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
@pstibrany pstibrany requested a review from a team as a code owner January 6, 2023 10:00
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
@pstibrany pstibrany changed the title limits-validation Check that metric_relabel_configs has non-nil configuration in runtime config Jan 6, 2023
@pracucci pracucci merged commit 228503a into main Jan 9, 2023
@pracucci pracucci deleted the limits-validation branch January 9, 2023 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants