Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add monitoring to objects cleaner #3608

Merged
merged 1 commit into from
Nov 17, 2023

Conversation

opudrovs
Copy link
Contributor

@opudrovs opudrovs commented Nov 8, 2023

Closes #3043

What changed?

To enhance the troubleshooting experience of the Explorer's management path for users, we have added RED metrics to the objects cleaner component of Explorer:

  1. Added recording of objects cleaner status and object removal operations as metrics.

  2. Introduced Grafana dashboard panels: "Objects Cleaner Request Rate" and "Objects Cleaner Requests Duration" for the objects cleaner.

  3. Updated Explorer's monitoring documentation to include the new metrics.

Release Notes

(Include in the next release notes under the Highlights section)

Highlights

RED metrics and corresponding Grafana dashboard panels have been added to the objects cleaner component of Explorer, enhancing the troubleshooting capabilities in Explorer's management path.

@opudrovs opudrovs added the enhancement New feature or request label Nov 8, 2023
@opudrovs opudrovs self-assigned this Nov 8, 2023
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch 14 times, most recently from d1754b9 to 64a1aa5 Compare November 14, 2023 10:18
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from 64a1aa5 to 600218a Compare November 14, 2023 12:53
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch 8 times, most recently from f92e3bd to d36e578 Compare November 15, 2023 12:40
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch 3 times, most recently from 324c0f5 to f8361fc Compare November 16, 2023 17:27
opudrovs added a commit that referenced this pull request Nov 17, 2023
Add setting objects cleaner status and recording it as a metric.

Add cleaner metrics for removing old objects and corresponding test.

Fix object cleaner not starting.

Rename `reterr` to `retErr` for consistency.
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from f8361fc to dabf1f2 Compare November 17, 2023 00:28
@opudrovs opudrovs marked this pull request as ready for review November 17, 2023 10:11
@opudrovs
Copy link
Contributor Author

opudrovs commented Nov 17, 2023

@enekofb @jpellizzari please do not merge this PR until two other PRs for this issue are merged (in quickstart and OSS).

@opudrovs
Copy link
Contributor Author

As discussed, only added metrics for calling removeOldObjects in general, without counting total objects removed (and without skipping recording metrics if no objects were removed). But it can be changed if/when needed.

Copy link
Contributor

@enekofb enekofb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! minor comments suggested

Tested that i could see metrics and dashboard

Screenshot 2023-11-17 at 11 32 02

pkg/query/cleaner/cleaner.go Outdated Show resolved Hide resolved
pkg/query/cleaner/metrics/recorder.go Show resolved Hide resolved
pkg/query/cleaner/metrics/recorder.go Show resolved Hide resolved
opudrovs added a commit that referenced this pull request Nov 17, 2023
Add setting objects cleaner status and recording it as a metric.

Add cleaner metrics for removing old objects and corresponding test.

Fix object cleaner not starting.

Rename `reterr` to `retErr` for consistency.
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from 8e69f61 to c0a5c57 Compare November 17, 2023 14:41
opudrovs added a commit that referenced this pull request Nov 17, 2023
Add setting objects cleaner status and recording it as a metric.

Add cleaner metrics for removing old objects and corresponding test.

Fix object cleaner not starting.

Rename `reterr` to `retErr` for consistency.
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from c0a5c57 to 758431a Compare November 17, 2023 14:42
Copy link

@jpellizzari jpellizzari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice work on cleaning up the enabled/disabled discrepancy

@opudrovs
Copy link
Contributor Author

LGTM. Nice work on cleaning up the enabled/disabled discrepancy

That's @enekofb 's doing. Yeah, enabled/disabled was somewhat confusing.

@jpellizzari
Copy link

LGTM. Nice work on cleaning up the enabled/disabled discrepancy

That's @enekofb 's doing. Yeah, enabled/disabled was somewhat confusing.

The original intent was to have people opt out of the cleaner by exception. I kinda messed up the naming, so glad that is cleaned up.

@opudrovs
Copy link
Contributor Author

Yes, if I did it myself, I would have renamed it to enabled first thing. 😅 So, glad that Eneko picked it up, knowledge of the whole context is very important here.

opudrovs added a commit that referenced this pull request Nov 17, 2023
Add setting objects cleaner status and recording it as a metric.

Add cleaner metrics for removing old objects and corresponding test.

Fix object cleaner not starting.

Rename `reterr` to `retErr` for consistency.
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from 758431a to 8e1331c Compare November 17, 2023 17:40
opudrovs added a commit that referenced this pull request Nov 17, 2023
Add setting objects cleaner status and recording it as a metric.

Add cleaner metrics for removing old objects and corresponding test.

Fix object cleaner not starting.

Rename `reterr` to `retErr` for consistency.

Remove unneeded comment.
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from 8e1331c to 5f9bf4b Compare November 17, 2023 18:20
Add setting objects cleaner status and recording it as a metric.

Add cleaner metrics for removing old objects and corresponding test.

Fix object cleaner not starting.

Rename `reterr` to `retErr` for consistency.

Remove unneeded comment.
@opudrovs opudrovs force-pushed the 3043-add-monitoring-to-object-cleaner branch from 5f9bf4b to 6ad9524 Compare November 17, 2023 19:22
@opudrovs opudrovs merged commit 928f2a3 into main Nov 17, 2023
10 checks passed
@opudrovs opudrovs deleted the 3043-add-monitoring-to-object-cleaner branch November 17, 2023 19:37
@opudrovs
Copy link
Contributor Author

Closes #3043

@opudrovs opudrovs changed the title Add monitoring to object cleaner Add monitoring to objects cleaner Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add monitoring to object cleaner
3 participants