Users don’t complete all the things needed to go live

[quis]

User need

As a Notify user
I need to easily find information about how to request to go live,
So that I’m able to see what I need to go before I can go live and navigate back to it again once I’m ready

Insight

The user journey to request to go live is frustrating for users. In lab testing we saw users get confused when they were directed to the ‘Using Notify’ page. They commented that they didn’t know where they’d been linked to and tried to understand where the ‘Using Notify’ page was in the top nav. There were concerns that they wouldn’t know how to navigate back to the page again if they wanted to.

Once users had performed an action on the go-live checklist, they frequently struggled to navigate back to the request to go live page. In some cases, the only way they could work out how to find the ‘Request to go live’ page was to try sending a message again in order to encounter the error message about trial mode.

In general, users struggled to find the information about trial mode/going live on the ‘Settings’ page. They typically clicked around for quite a while before finding it in ‘Settings’.

---

The process is not clear to users. In support tickets they say stuff like:

Hi, we'll soon be looking to go live with this service. Just wanted to check the activation process beforehand, timescales. And if there's anything you need from us in advance.

We are currently using Notify in Trial mode, but ideally we would like to do some testing on the process of sending letters. Please can you tell us how we move out of trial mode in order for us to start the testing on sending letters?

We're wanting to use Notify as part of our access management solution. We've created an account already, but want to know what the process is to make that account usable in a production environment without the restrictions. We plan to 'go live' in end of June / July time.

User need

As a Notify user,
I need to understand what I must do before I’m allowed to go live,
So that I can go live as soon as possible after making the request, without needing additional support from the Notify team.

Insight

In lab testing, understanding of the ‘request to go live’ checklist was mixed. Most understood it was a list of things they need to do before they can request to go live. But one person didn’t pay attention to the checklist at all and just clicked the green button:

‘I saw the big green button saying continue….I guess continue suggests next action….these links up here look optional’

‘The team member hyperlink looks like it might tell me what a team member is generally, not that I have to do something’

Although most understood they needed to add a team member to their service, they typically assigned the wrong level of permission. This was either because they hadn’t read this on the previous page or couldn’t remember the level of permission mentioned.

Support ticket analysis findings May-July 2018

• 46% of requests to go live are 'incomplete'.
• 30% have completed the checklist, but hadn't signed the MOU
• 32.5% had signed the MOU, but not completed checklist
• 37% hadn't done either

Needing to add or amend the email-reply-to address and/or text sender name were most common things missed off checklist:

• add or change email-reply-to address (15)
• add or change text message sender name (12)
• Add team member with manage service permission (4)
• Add or change templates (9)

Some people use their personal email address as a reply-to by mistake. Others use an inappropriate shared email that we think their users won't recognise, or an invalid address.
When it comes to text message sender names, in most cases we're letting them know they can change it from the default to something more relevant for their service.

[quis]

This comment is a placeholder to summarise the work in these pull requests:

alphagov/notifications-admin#508

The original request to go live page:

alphagov/notifications-admin#1887

Before	After

alphagov/notifications-admin#1909

Before	After

alphagov/notifications-admin#1950

Before	After

• Clean up request to go live page notifications-admin#1955

[quis]

From analysis of support tickets about 25% of users who request to go live aren’t completing all the items on the checklist

We did some guerrilla usability testing of this journey and in 1 of 6 (17%) of sessions we saw someone skip straight past the checklist page because of big green button syndrome. While 1 in 6 people would normally be a small number¹ in the context of a usability testing session, it’s enough to cause a big workload for our team (assuming it is the sole cause of people not completing the items on the checklist).

The initial reason for using the tick cross pattern for the checklist was:

• it was coherent with the rest of Notify
• the task list pattern didn’t have a way of showing that something still needed doing – it put more visual emphasis on the things the user had already done

There’s been some interesting discussion on the GOV.UK Design System backlog² about users failing to complete items in the task list. A few people have tried different patterns for communicating that items in the task list still need ‘completing’.

So we tried using the task list pattern on the request to go live page. The task list is adapted from the one in the design system in that:

• the ‘completed’ label has a black, not blue background (because Notify often uses blocks of blue to indicate something that’s clickable)
• it adds an explicit ‘not complete’ label which is visually not filled in (sort of how ticked/unticked radio buttons work)

Add text message sender to the task list where appropriate

We often check that a service has an appropriate text message sender as a condition of them going live. We don’t mention this anywhere.

The services for whom GOVUK is definitely not an appropriate sender are those in local government. As we have more of these teams starting to use Notify, we should streamline the process by making this check automated.

So we also added a check for this to the task list, for teams who:

• have text message templates
• have self-declared as NHS or local government

Before	After

alphagov/notifications-admin#2250

---

1. With the caveat that looking only at task completion and quantifying qualitative research can both be problematic and the intention here is to show that the numbers are close enough to say that they could be symptoms of the same problem. Leisa Reichelt’s Mind the Product talk is good on this stuff.

2. Task list govuk-design-system-backlog#72

[quis]

Add line about the data sharing and financial agreement to the checklist

Placeholder comment for the work in this pull request:

• Add line about the data sharing and financial agreement to the checklist notifications-admin#2327

---

Before	After

[quis]

Let people add details of how many messages they’re sending at any time

Placeholder comment for the work in:

• Frontload estimate usage notifications-admin#2774

---

Before	After

This is the page behind the new item in the task list:

[quis]

Placeholder comment for the work done in:

• When going live, make reply-to address and text message sender dependent on estimated sending volumes notifications-admin#2827

[quis]

Something about how we’re managing organisations now…

[quis]

Don’t let users make a request to go live until they’ve completed all the required steps

Dealing with users who request to go live but haven’t completed all the steps still represents a significant support overhead for our team. We’ve made some improvements to the percentage of incomplete requests with a better page design, but ultimately because it still shows the button people think it’s OK to press the button while some of the items on the page still say [Not completed].

We can do this now because organisations are in the database, which means we can mark the agreement signed as soon as we get it back, without having to deploy code.

—alphagov/notifications-admin#2907

---

Before	After

[quis]

Let users accept the data sharing and financial agreement online

At the moment, accepting the data sharing and financial agreement looks like this:

The full process is:

1. download a pdf

• print it out
• get someone to sign it
• scan it
• email it back to us
• we rename the file and save it in Google Drive
• we then update the organisation to say the MOU is signed
• sometimes we also:
• print it out and get it counter-signed
• scan it again
• email it back to the service

Let's not do that any more.

When the first service for an organisation that doesn't have the agreement in place is in the process of going live, then they should be able to accept the agreement online as part of the go live flow.

Where the checklist shows the agreement as [not completed] then they can follow a link to download it (as happens now). From here, they should then also be able to provide some info to accept it. The info that we need is:

• Version - because we version the agreements occasionally, we need to know which version they are accepting. It may not be the latest one if they downloaded it a while ago and it took time to be signed off

• Who is accepting the agreement - this will often be someone in the finance team, and not necessarily a team member, so we should let the person either accept as themselves, or on behalf of someone else. If it's on behalf of someone else we need to the name and email address of that person so we have that on record. Obvs if it's them accepting it themselves, we have that already.

We then need to replay the collected info back in a sort of legally binding kind of way pulling in the organisation name too.

---

Google docs sketch:

---

After

Further reading

• Allow online acceptance of the contract/memorandum of understanding notifications-admin#3019
• https://www.pivotaltracker.com/story/show/166497058