Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki mixin: make labelsSelector in loki chunks dashboards configurable #5536

Merged
merged 1 commit into from
Mar 4, 2022
Merged

Loki mixin: make labelsSelector in loki chunks dashboards configurable #5536

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## Main


* [5536](https://github.com/grafana/loki/pull/5536) **jiachengxu**: Loki mixin: make labelsSelector in loki chunks dashboards configurable
* [5535](https://github.com/grafana/loki/pull/5535) **jiachengxu**: Loki mixins: use labels selector for loki chunks dashboard
* [5507](https://github.com/grafana/loki/pull/5507) **MichelHollands**: Remove extra param in call for inflightRequests metric.
* [5356](https://github.com/grafana/loki/pull/5356) **jbschami**: Enhance lambda-promtail to support adding extra labels from an environment variable value
Expand Down
44 changes: 22 additions & 22 deletions production/loki-mixin/dashboards/loki-chunks.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ local utils = import 'mixin-utils/utils.libsonnet';
(import 'dashboard-utils.libsonnet') {
grafanaDashboards+: {
local dashboards = self,
local labelsSelector = 'cluster="$cluster", job="$namespace/ingester"',
'loki-chunks.json': {
local cfg = self,
labelsSelector:: 'cluster="$cluster", job="$namespace/ingester"',
} +
$.dashboard('Loki / Chunks', uid='chunks')
.addCluster()
Expand All @@ -16,14 +16,14 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.row('Active Series / Chunks')
.addPanel(
$.panel('Series') +
$.queryPanel('sum(loki_ingester_memory_chunks{%s})' % labelsSelector, 'series'),
$.queryPanel('sum(loki_ingester_memory_chunks{%s})' % dashboards['loki-chunks.json'].labelsSelector, 'series'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these can all reference self

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dannykopping for the review.
I just tried:

$mixtool lint dashboards.libsonnet                                                                                                           
RUNTIME ERROR: Field does not exist: labelsSelector
	dashboards/loki-chunks.libsonnet:19:83-102
	<std>:237:21-22	thunk from <function <anonymous>>
	<std>:776:20-24	thunk from <function <anonymous>>
	<std>:32:25-26	thunk from <function <anonymous>>
	<std>:32:16-27	function <anonymous>
	<std>:776:8-25	function <anonymous>
	<std>:237:7-23	function <anonymous>
	dashboards/loki-chunks.libsonnet:19:42-102
	/Users/jiachengxu/projects/github.com/grafana/loki/production/loki-mixin/vendor/grafana-builder/grafana.libsonnet:234:19-26	thunk from <object <anonymous>>
	/Users/jiachengxu/projects/github.com/grafana/loki/production/loki-mixin/vendor/grafana-builder/grafana.libsonnet:234:10-27	object <anonymous>
	...
	/Users/jiachengxu/projects/github.com/grafana/loki/production/loki-mixin/vendor/grafana-builder/grafana.libsonnet:255:17-24	object <anonymous>
	/Users/jiachengxu/projects/github.com/grafana/loki/production/loki-mixin/vendor/grafana-builder/grafana.libsonnet:(246:15)-(256:6)
	/Users/jiachengxu/projects/github.com/grafana/loki/production/loki-mixin/vendor/grafana-builder/grafana.libsonnet:(246:15)-(256:6)	+:
	Field "targets"
	Array element 0
	Field "panels"
	Array element 0
	Field "rows"
	Field "loki-chunks.json"
	During manifestation

I think the reason is that the labelsSelector doesn't belong to the current object(https://github.com/grafana/loki/pull/5536/files#diff-d15d89e43869bd43b09a788d084921eff62a763bbb4c7df33c95cb919a7bd644R7-R10)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I think it'd be best to make it a local then; I'm not sure what value there is in having this as a field

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but this PR is trying to switch from local to a field, to make it configurable for different use cases. local basically cannot be overridden if you want to use a different labelsSelector.
I saw similar use cases in other dashboards: https://github.com/grafana/loki/blob/main/production/loki-mixin/dashboards/loki-operational.libsonnet#L35 for example, in this dashboard, user can override matchers for difference use cases

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

local basically cannot be overridden if you want to use a different labelsSelector

Ah, right. Got it.
I think we need to take a more deliberate approach with the labelsSelector, then. This implementation feels a little hacky.

For now, I think let's merge it and we should try split this file up and provide a cleaner way to override these labelsSelectors in the future.

Thanks 👍

)
.addPanel(
$.panel('Chunks per series') +
$.queryPanel(
'sum(loki_ingester_memory_chunks{%s}) / sum(loki_ingester_memory_streams{%s})' % [
labelsSelector,
labelsSelector,
dashboards['loki-chunks.json'].labelsSelector,
dashboards['loki-chunks.json'].labelsSelector,
],
'chunks'
),
Expand All @@ -33,27 +33,27 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.row('Flush Stats')
.addPanel(
$.panel('Utilization') +
$.latencyPanel('loki_ingester_chunk_utilization', '{%s}' % labelsSelector, multiplier='1') +
$.latencyPanel('loki_ingester_chunk_utilization', '{%s}' % dashboards['loki-chunks.json'].labelsSelector, multiplier='1') +
{ yaxes: $.yaxes('percentunit') },
)
.addPanel(
$.panel('Age') +
$.latencyPanel('loki_ingester_chunk_age_seconds', '{%s}' % labelsSelector),
$.latencyPanel('loki_ingester_chunk_age_seconds', '{%s}' % dashboards['loki-chunks.json'].labelsSelector),
),
)
.addRow(
$.row('Flush Stats')
.addPanel(
$.panel('Log Entries Per Chunk') +
$.latencyPanel('loki_ingester_chunk_entries', '{%s}' % labelsSelector, multiplier='1') +
$.latencyPanel('loki_ingester_chunk_entries', '{%s}' % dashboards['loki-chunks.json'].labelsSelector, multiplier='1') +
{ yaxes: $.yaxes('short') },
)
.addPanel(
$.panel('Index Entries Per Chunk') +
$.queryPanel(
'sum(rate(loki_chunk_store_index_entries_per_chunk_sum{%s}[5m])) / sum(rate(loki_chunk_store_index_entries_per_chunk_count{%s}[5m]))' % [
labelsSelector,
labelsSelector,
dashboards['loki-chunks.json'].labelsSelector,
dashboards['loki-chunks.json'].labelsSelector,
],
'Index Entries'
),
Expand All @@ -63,22 +63,22 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.row('Flush Stats')
.addPanel(
$.panel('Queue Length') +
$.queryPanel('cortex_ingester_flush_queue_length{%s}' % labelsSelector, '{{pod}}'),
$.queryPanel('cortex_ingester_flush_queue_length{%s}' % dashboards['loki-chunks.json'].labelsSelector, '{{pod}}'),
)
.addPanel(
$.panel('Flush Rate') +
$.qpsPanel('loki_ingester_chunk_age_seconds_count{%s}' % labelsSelector,),
$.qpsPanel('loki_ingester_chunk_age_seconds_count{%s}' % dashboards['loki-chunks.json'].labelsSelector,),
),
)
.addRow(
$.row('Flush Stats')
.addPanel(
$.panel('Chunks Flushed/Second') +
$.queryPanel('sum(rate(loki_ingester_chunks_flushed_total{%s}[$__rate_interval]))' % labelsSelector, '{{pod}}'),
$.queryPanel('sum(rate(loki_ingester_chunks_flushed_total{%s}[$__rate_interval]))' % dashboards['loki-chunks.json'].labelsSelector, '{{pod}}'),
)
.addPanel(
$.panel('Chunk Flush Reason') +
$.queryPanel('sum by (reason) (rate(loki_ingester_chunks_flushed_total{%s}[$__rate_interval])) / ignoring(reason) group_left sum(rate(loki_ingester_chunks_flushed_total{%s}[$__rate_interval]))' % [labelsSelector, labelsSelector], '{{reason}}') + {
$.queryPanel('sum by (reason) (rate(loki_ingester_chunks_flushed_total{%s}[$__rate_interval])) / ignoring(reason) group_left sum(rate(loki_ingester_chunks_flushed_total{%s}[$__rate_interval]))' % [dashboards['loki-chunks.json'].labelsSelector, dashboards['loki-chunks.json'].labelsSelector], '{{reason}}') + {
stack: true,
yaxes: [
{ format: 'short', label: null, logBase: 1, max: 1, min: 0, show: true },
Expand All @@ -102,7 +102,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
).addTargets(
[
grafana.prometheus.target(
'sum by (le) (rate(loki_ingester_chunk_utilization_bucket{%s}[$__rate_interval]))' % labelsSelector,
'sum by (le) (rate(loki_ingester_chunk_utilization_bucket{%s}[$__rate_interval]))' % dashboards['loki-chunks.json'].labelsSelector,
legendFormat='{{le}}',
format='heatmap',
),
Expand All @@ -127,7 +127,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
).addTargets(
[
grafana.prometheus.target(
'sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[$__rate_interval])) by (le)' % labelsSelector,
'sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[$__rate_interval])) by (le)' % dashboards['loki-chunks.json'].labelsSelector,
legendFormat='{{le}}',
format='heatmap',
),
Expand All @@ -141,9 +141,9 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.panel('Chunk Size Quantiles') +
$.queryPanel(
[
'histogram_quantile(0.99, sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[1m])) by (le))' % labelsSelector,
'histogram_quantile(0.90, sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[1m])) by (le))' % labelsSelector,
'histogram_quantile(0.50, sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[1m])) by (le))' % labelsSelector,
'histogram_quantile(0.99, sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[1m])) by (le))' % dashboards['loki-chunks.json'].labelsSelector,
'histogram_quantile(0.90, sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[1m])) by (le))' % dashboards['loki-chunks.json'].labelsSelector,
'histogram_quantile(0.50, sum(rate(loki_ingester_chunk_size_bytes_bucket{%s}[1m])) by (le))' % dashboards['loki-chunks.json'].labelsSelector,
],
[
'p99',
Expand All @@ -161,11 +161,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.panel('Chunk Duration hours (end-start)') +
$.queryPanel(
[
'histogram_quantile(0.5, sum(rate(loki_ingester_chunk_bounds_hours_bucket{%s}[5m])) by (le))' % labelsSelector,
'histogram_quantile(0.99, sum(rate(loki_ingester_chunk_bounds_hours_bucket{%s}[5m])) by (le))' % labelsSelector,
'histogram_quantile(0.5, sum(rate(loki_ingester_chunk_bounds_hours_bucket{%s}[5m])) by (le))' % dashboards['loki-chunks.json'].labelsSelector,
'histogram_quantile(0.99, sum(rate(loki_ingester_chunk_bounds_hours_bucket{%s}[5m])) by (le))' % dashboards['loki-chunks.json'].labelsSelector,
'sum(rate(loki_ingester_chunk_bounds_hours_sum{%s}[5m])) / sum(rate(loki_ingester_chunk_bounds_hours_count{%s}[5m]))' % [
labelsSelector,
labelsSelector,
dashboards['loki-chunks.json'].labelsSelector,
dashboards['loki-chunks.json'].labelsSelector,
],
],
[
Expand Down