Dashboard: Add panels for ingest storage to Writes and Queries dashboards #7670
Conversation
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
…cy panels. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
| ) + $.aliasColors({ failed: '#FF0000', 'read errors': '#FF0000' }) + $.stack, | ||
| ) | ||
| .addPanel( | ||
| $.timeseriesPanel('Records per fetch') + |
There was a problem hiding this comment.
Instead of this panel I would add "Kafka records / sec", which similar to "Kafka fetches / sec" but based on cortex_ingest_storage_reader_records_* metrics. I think it's important to show the rate of records / s, as much as the rate of fetches / s.
There was a problem hiding this comment.
Great idea, this is much more useful than records per fetch.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
| ) + $.aliasColors({ failed: '#FF0000', 'read errors': '#FF0000' }) + $.stack, | ||
| ) | ||
| .addPanel( | ||
| $.timeseriesPanel('Records per fetch') + |
There was a problem hiding this comment.
Great idea, this is much more useful than records per fetch.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
| $._config.show_ingest_storage_panels, | ||
| ($.row('Ingester (ingest storage: strong consistency)')) | ||
| .addPanel( | ||
| $.timeseriesPanel('Requests with strong consistency') + |
There was a problem hiding this comment.
[nit] WDYT about being more specific about the "strong consistency"?
| $.timeseriesPanel('Requests with strong consistency') + | |
| $.timeseriesPanel('Requests with strong read consistency / sec') + |
If you agree, same comment applies in other places.
| @@ -649,6 +649,8 @@ | |||
| 'debug_pprof', | |||
| ], | |||
|
|
|||
| ingester_read_path_routes_regex: '/cortex.Ingester/Query(Stream)?|/cortex.Ingester/MetricsForLabelMatchers|/cortex.Ingester/LabelValues|/cortex.Ingester/MetricsMetadata', | |||
There was a problem hiding this comment.
I know this is what we use in the Reads dashboard but I think it's missing several stuff (e.g. ActiveSeries, UserStats, ...).
There was a problem hiding this comment.
Will send separate PR to fix, since it's unrelated to this PR.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
pstibrany
changed the title
What this PR does
This PR adds panels for monitoring distributor and ingester when using ingest-storage. These panels can be enabled by setting
show_ingest_storage_panelsoption (defaults tofalse) in dashboard config. Irrelevant panels used for monitoring distributors and ingesters when using gRPC communication internally can be hidden by settingshow_grpc_ingestion_panels: false(defaults to true).Write dashboard has new "Ingester (ingest storage)" section:
Queries dashboard has new ingest-storage related sections:
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.