Skip to content
This repository has been archived by the owner on Jul 12, 2023. It is now read-only.

Simplify latency alerting #1510

Merged
merged 5 commits into from
Jan 5, 2021
Merged

Simplify latency alerting #1510

merged 5 commits into from
Jan 5, 2021

Conversation

yuriatgoogle
Copy link
Contributor

Fixes #

Proposed Changes

  • loosened default latency SLO target
  • added latency-alert module
  • added per-service toggles for latency alerts
  • added per-service toggles for latency SLOs

Release Note

Latency alerting done via threshold, rather than SLO

@googlebot googlebot added the cla: yes Auto: added by CLA bot when all committers have signed a CLA. label Jan 5, 2021

variable "duration" {
type = number
description = "Duration of alert evaluation"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you added the unit for threshold in ms, what is the unit for duration? same ms?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added unit, thank you.

latency_threshold = 60000 # in ms
enable_alert = false
availability_goal = 0.995
enable_latency_slo = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment that we are going to disable latency SLO in favor of the latency alert for services with low number of requests? Just to clarify since this can be confusing in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -71,11 +94,12 @@ module "latency-slos" {

project = var.project

enabled = var.https-forwarding-rule != ""
# enabled = var.https-forwarding-rule != ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this line?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you.

{ enable_alert = true,
{ enable_alert = true,
enable_latency_alert = true,
enable_latency_slo = false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems you don't need to set this here, since it is false by default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thank you.

Copy link
Member

@sethvargo sethvargo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sethvargo, yuriatgoogle

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mariliamelo
Copy link
Contributor

/lgtm

@google-oss-robot google-oss-robot merged commit f133368 into main Jan 5, 2021
@google-oss-robot google-oss-robot deleted the update-latency branch January 5, 2021 20:57
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla: yes Auto: added by CLA bot when all committers have signed a CLA.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants