Centralized monitoring for rate of reported unauthorized logins

[rfk]

One of our Q4 objectives is to avoid regressing the rate of unauthorized logins while we add news ways to connect to the account. We've started collecting a few different ways for users to report unauthorized login attempts, and I count four in total:

• Reporting an unexepected "unblock code" email
• Reporting an unexpected "sign-in confirmation" email
• Reporting an unexpected "new device connected" email
• Disconnecting an unrecognized device through the devices view

For monitoring purposes, we should display all of these on a single graph, stacked to give an overall "rate at which bad things are being reported" measure, and set up alerts if it starts to increase significantly. Here's a quick mock of what it might look like:

I believe we have events for all four of these things already, although they may not all be going to the same place. We might also consider making a more purpose-build "report this to us" page for the sign-in confirmation and new-device-notification emails, along the same lines as what we did for sign-in unblock.

/cc @shane-tomlinson @davismtl @philbooth

[davismtl]

This looks good to me. This should work perfectly for monitoring our quarter (and the following). Seems like it should be pretty simple to do with events.

[rfk]

@shane-tomlinson @vbudhram @vladikoff - as a starting point, what metrcs events do we currently have that correspond to these four ways or reporting unauthorized activity, and where are they currently available for viewing?

[vbudhram]

@rfk I could see the sign-in confirmation stack being built from the server-side dashboard.

This dashboard tracks the attempts, successes and errors when performing sign-in confirmation. Errors for sign-in confirmation could be a little tricky because they encompass invalid tokens and already used tokens.

[rfk]

Digging into the events, I think we can build a first cut of this in datadog:

• Reporting an unexepected "unblock code" email ->
  • fxa.content.report_signin.submit
• Reporting an unexpected "sign-in confirmation" email ->
  • fxa.content.reset_password.submit with utm_campaign=fx-new-signin
• Reporting an unexpected "new device connected" email ->
  • fxa.content.reset_password.submit with utm_campaign=fx-new-device-login
  • (although we don't have any of these, because noone is currently getting this email, because signin confirmation)
• Disconnecting an unrecognized device through the devices view ->
  • fxa.content.settings.clients.disconnect.submit.suspicious

I've added it here:

https://app.datadoghq.com/dash/116995/fxa-content---security?live=true&page=0&fullscreen=130208654

And it has a (thankfully) low baseline rate.

[rfk]

Although since they're client-side datadog metrics, are these all sampled at 10% @vladikoff?

[vladikoff]

Although since they're client-side datadog metrics, are these all sampled at 10% @vladikoff?

Yea 10%

[rfk]

OK thanks...anyway it's still pretty low :-)