Skip to content

US Digital Services Playbook Play 12

Jump to bottom
mattkwong-kpmg edited this page Mar 3, 2017 · 10 revisions

#Play 12: Use Data to Drive Decisions At every stage of a project, we should measure how well our service is working for our users. This includes measuring how well a system performs and how people are interacting with it in real-time. Our teams and agency leadership should carefully watch these metrics to find issues and identify which bug fixes and improvements should be prioritized. Along with monitoring tools, a feedback mechanism should be in place for people to report issues directly.

##Checklist

  • 1. Monitor system-level resource utilization in real time

We are using Nagios and Azure built-in tools to monitor system-level resource utilization in real time.

  • 2. Monitor system performance in real-time (e.g. response time, latency, throughput, and error rates)

We are using Nagios to monitor system performance in real time.

  • 3. Ensure monitoring can measure median, 95th percentile, and 98th percentile performance

Nagios is capable of monitoring performance in this way, but it was not necessary for this prototype.

  • 4. Create automated alerts based on this monitoring

We are using Nagios to perform automated alerts based on monitoring results. Nagios can send alerts when critical infrastructure components fail and recover, providing administrators with notice of important events via email, SMS, or custom script.

  • 5. Track concurrent users in real-time, and monitor user behaviors in the aggregate to determine how well the service meets user needs

We have a backlog item to track integration with Google Analytics for this capability (Issue #73)

  • 6. Publish metrics internally

Nagios has comprehensive monitoring dashboard for those with access, and reports and alerts for sending metrics to internal users as necessary.

  • 7. Publish metrics externally

Nagios has comprehensive monitoring dashboard for those with access, and reports and alerts for sending metrics to external users as necessary.

  • 8. Use an experimentation tool that supports multivariate testing in production

Not applicable for this prototype.

##Key Questions

  1. What are the key metrics for the service?

Key metrics include Memory, CPU, Latency, etc.

  1. How have these metrics performed over the life of the service?

Not applicable for this prototype.

  1. Which system monitoring tools are in place?

We are using Nagios and Azure built-in tools.

  1. What is the targeted average response time for your service? What percent of requests take more than 1 second, 2 seconds, 4 seconds, and 8 seconds?

The majority of requests take less than 2 seconds.

  1. What is the average response time and percentile breakdown (percent of requests taking more than 1s, 2s, 4s, and 8s) for the top 10 transactions?

See Nagios.

  1. What is the volume of each of your service’s top 10 transactions? What is the percentage of transactions started vs. completed?

Not applicable for this prototype.

  1. What is your service’s monthly uptime target?

99%

  1. What is your service’s monthly uptime percentage, including scheduled maintenance? Excluding scheduled maintenance?

Not applicable for prototype.

  1. How does your team receive automated alerts when incidents occur?

We set up automatic email notifications in Nagios.

  1. How does your team respond to incidents? What is your post-mortem process?

Not applicable for this prototype.

  1. Which tools are in place to measure user behavior?

Not applicable for this prototype.

  1. What tools or technologies are used for A/B testing?

Not applicable for this prototype.

  1. How do you measure customer satisfaction?

Not applicable for this prototype.

#US Digital Services Playbook

  1. Play 1 Understand what people need
  2. Play 2 Address the whole experience, from start to finish
  3. Play 3 Make it simple and intuitive
  4. Play 4 Build the service using agile and iterative practices
  5. Play 5 Structure budgets and contracts to support delivery
  6. Play 6 Assign one leader and hold that person accountable
  7. Play 7 Bring in experienced teams
  8. Play 8 Choose a modern technology stack
  9. Play 9 Deploy in flexible hosting environment
  10. Play 10 Automate testing and deployments
  11. Play 11 Manage security and privacy through reusable processes
  12. Play 12 Use data to drive decisions
  13. Play 13 Default to open
Clone this wiki locally