UX Explorations: Common Error States

[savannahostrowski]

Background

One of the biggest key findings from our usability studies and experience audit is that the current experience does not allow for developers to easily self-correct issues they encounter. This is most frequently observed when users encounter an error or a warning message in the CLI. Often the error message will result in a failed user journey, ending the experience abruptly without the option to take action to resolve the error in context.

Specific error state examples

• Changing Azure location and re-deploying results in deployment name conflict #293
• https://github.com/Azure/azure-dev-pr/issues/1207 - this issue is now closed. Created for when we were using environment name for naming Azure resources.
• [onboarding experience improvement] Region capacity check #191
• [onboarding experience improvement] Region support check #305
• https://github.com/Azure/azure-dev-pr/issues/610
• az dev deploy reporting misleading errors when I tried to deploy to an environment that has not been provisioned #211
• [Issue] user study error message guide on deploying service #168
• [Issue] When customizing container resources, error message is generic #107
• Reword message for not having .NET installed #1371
• No error message if run from root of user directory (.azure file conflict)  #1400

Hypothesis

We believe we can reduce the amount of end-user journeys that end in failure by providing more actionable and consistent error messages.

• Introduce a common and consistent visual pattern for all error messaging in the CLI (bonus: Warning & Success states)
• Update error messaging to be more human readable, precise, and include instructive advice
• Educate users on why the error occurred so that they can prevent future occurrences
• Wherever possible provide the ability for a user to take action on the error in context

Initial task list

• Explore treatment for errors outside the control of azd (example: az error or npm error)
• Triage and re-produce error state examples listed above (puicchan)
• Discuss error messages captured - identify potential solutions
• Complete design pass at error messages ( @Austinauth )
• Complete design pass at "Design Language" (colors, success, error, warning, hint patterns)
• Feedback session to review error state designs and Design Language
• Update and finalize designs based on feedback provided
• Demo to devs for implementation (get pulse check from Wallace before we demo the designs)

[Austinauth]

Update: 08/08/2022

Updated @savannahostrowski's initial comment above with more context, including a background, hypothesis, and current observed error examples.

[Austinauth]

Update: 08/16/2022

• Review issue that Matt captured ( @Austinauth )
• Explore generic error styles ( @Austinauth )
• Attempt to apply error styles ( @Austinauth )
• Review error styles and attempts to apply to in context next week ( @Austinauth )
• Need flow chart for errors to show what happens
• Need data on top error messages we get ( @savannahostrowski )

[savannahostrowski]

Added #107 to the list as well.

[Austinauth]

Update: 08/30/2022

• Review feedback from Savannah & PC
• Share with the dev for feedback
• Update the design with feedback
• Schedule time on the next dev demo day to review

[karolz-ms]

Thanks for sharing @Austinauth Here is my feedback--HTH!

Color usage: none of the examples show the IDE with a light theme 😞 . I would like to see how the color output will like when the background is white or very bright gray.
Also, I would assume we want to use primary text color ONLY if NO_COLOR is set?

Progress indications look great

Regarding the "actionable hint": I am afraid of issues that might arise from the fact, that we are changing the execution model of the tool from "single command per invocation" to "multiple commands per invocation, with rollbacks". This significantly raises the complexity level, both internal, as well as from the perspective of users understanding what the tool will do/has done. Perhaps instead of trying to automatically execute the "hint", we can stop at just giving the hint and letting the user decide what next command to issue.

Also, a hint in the form of a single-sentence hint with a yes/no decision might not be the most effective. E.g in the "busy region" example there are many things that the user might want to do, such as, trying the deployment again later, tweaking the deployment to use a flavor of a resource that is less in demand and trying again, trying a different region, deleting/purging the deployment and trying again, and possibly others. So maybe a short list of suggestions, with a URL for "more info" might be a better option.

No strong preference on whether to show full command help, or just the error message, when an unrecognized parameter/flag is used. I think both can be made to work.

For the "misspelled template" problem I would like to sort of combine option 1 and 2: I actually think it is a very useful piece of information that AZD template is just a GitHub repo, but I would also like to be told how I can get the list of available templates. Note: I prefer to be told that there is azd template list, instead of forced to select a template from a template list.

Hope this helps!

[Austinauth]

@karolz-ms

Color usage: none of the examples show the IDE with a light theme 😞 . I would like to see how the color output will like when the background is white or very bright gray.

Good idea. I can definitely mock this up for us to see how it looks.

Also, I would assume we want to use primary text color ONLY if NO_COLOR is set?

I'm honestly pretty new when it comes to working with the CLI and IDE, so I don't know how this works. Are you saying that if the user already has a color selected for their default text we would use that instead of our primary text color variable? If so, I think that makes sense.

Progress indications look great

🥳

Regarding the "actionable hint": I am afraid of issues that might arise from the fact, that we are changing the execution model of the tool from "single command per invocation" to "multiple commands per invocation, with rollbacks". This significantly raises the complexity level, both internal, as well as from the perspective of users understanding what the tool will do/has done. Perhaps instead of trying to automatically execute the "hint", we can stop at just giving the hint and letting the user decide what next command to issue.

Also, a hint in the form of a single-sentence hint with a yes/no decision might not be the most effective. E.g in the "busy region" example there are many things that the user might want to do, such as, trying the deployment again later, tweaking the deployment to use a flavor of a resource that is less in demand and trying again, trying a different region, deleting/purging the deployment and trying again, and possibly others. So maybe a short list of suggestions, with a URL for "more info" might be a better option.

This is really good feedback. It's something @savannahostrowski and @puicchan were also discussing. At what point does the hint become too complicated, overbearing, or even restrictive (limiting the user to a certain path)? We briefly explored options that were text only hints (like you describe) that allow the user to decide what command they want to enter and do it themselves.

We also admittedly did not totally know how complex the interactive hints would be to develop. Very glad to have your input there. Would also love to hear @savannahostrowski and @puicchan's follow up thoughts on how to approach this -- should we try going back to text only hints? I remember it was very challenging for us to be succinct given all the different next steps a user might want to take.

No strong preference on whether to show full command help, or just the error message, when an unrecognized parameter/flag is used. I think both can be made to work.

Okay cool! I've received feedback from others that the full command help could be overwhelming, so we may just start by showing the error message and let the user know they can run --help to see more.

For the "misspelled template" problem I would like to sort of combine option 1 and 2: I actually think it is a very useful piece of information that AZD template is just a GitHub repo, but I would also like to be told how I can get the list of available templates. Note: I prefer to be told that there is azd template list, instead of forced to select a template from a template list.

Awesome! That is a good idea. I will work on mocking this up ASAP so we can further assess.

Seriously, thank you so much for the feedback @karolz-ms! Its very helpful for me :)

[karolz-ms]

@Austinauth I am super happy that I was able to help a bit

Also, I would assume we want to use primary text color ONLY if NO_COLOR is set?

I'm honestly pretty new when it comes to working with the CLI and IDE, so I don't know how this works. Are you saying that if the user already has a color selected for their default text we would use that instead of our primary text color variable? If so, I think that makes sense.

No, what I am getting at is that some users may prefer not to use color at all. There is an informal, but very popular standard for expressing such preference, and I would love to see us support it, see https://no-color.org/ for more information.

[Austinauth]

@karolz-ms Ah gotcha. I have no problem if we want to support a low/no color option. I'm not sure what the technical implications are, but I'm attempting to design everything in such a way that the color is only there to enhance an already accessible experience.

For example:

1. Logs will always be prefixed with the log type (ex: ERROR: Error message), so even if there is no color there's still an indicator of what type of log message you're seeing.
2. Individual status of items as they're provisioned or deployed will not only feature the status in plain text but also a visual indicator (ex: (✓) or (×) )
3. I'm exploring a consistent way for suggested commands as part of error/warning messages to stand out visually using tick marks (ex: Try running azd up --help to see all available flags)

The only thing I can think of that might be an issue is hyperlinks? They may not be as easily identifiable as something you can click.

If there's something specific we need to do to make sure we support "no color" technically, let me know and we can work through it together! If its as simple as this, I say we do it:

Command-line software which adds ANSI color to its output by default should check for a NO_COLOR environment variable that, when present and not an empty string (regardless of its value), prevents the addition of ANSI color.

• https://no-color.org/

I'm also still planning on mocking up a version of the experience for users who have their terminal in light mode! Just need to carve out some time to do it 😅

[karolz-ms]

@Austinauth perfect! Thank you for hearing me out.

For hyperlinks we could differentiate them in NO_COLOR mode by underlining the text. I have seen it done by other terminal commands.

Other than that, NO_COLOR support is just a question of including it in the design. The libraries that we use support NO_COLOR intrinsically (at least the faith/color one), but it does not quite work today, we probably use it incorrectly.

[Austinauth]

Update 09/06/2022

• Hard to measure design success yet until we have telemetry
• Demo the errors today - see where we stand with the team (what feedback do we get)

[Austinauth]

Solution proposals for each of the error states can be found in this prototype under the Error Messages section. Team decided to hold off on the interactive error states for the time being.

[Austinauth]

@rajeshkamal5050 is there someone I should be pairing with on the dev side for implementation?

[rajeshkamal5050]

@rajeshkamal5050 is there someone I should be pairing with on the dev side for implementation?

Please work with @vhvb1989 on this.

[savannahostrowski]

@Austinauth Was this issue also considered
#468?

[rajeshkamal5050]

Moving it to October iteration.

[Austinauth]

Will review this in the next couple days and provide guidance.

[Austinauth]

@wbreza @vhvb1989 can I assign this issue to either of you since design is complete? Happy to review or provide input if anything else is needed.

[Austinauth]

Update 12/05/2022

• Need to follow up with Developers on the status of implementation of initial UX fixes.

[rajeshkamal5050]

@Austinauth do we have the design completed on the error handling/reporting? Please let @vhvb1989 know.

[Austinauth]

@rajeshkamal5050

@savannahostrowski, we agreed to break these out 1 by 1 right? I think a lot of these have been addressed or can be addressed by the error state styling guidance.

That being said, I am happy to sit sync with Victor or anyone else working on these on a case by case basis if more guidance/discussion is needed. Especially on things like messaging and hint text.

[savannahostrowski]

I think we can implement however best works for you and the eng team. I'll leave it to you to coordinate!

[Austinauth]

@rajeshkamal5050, I think we can close this then. Implementation can reference the guidance above, and then feel free to tag/assign me on any specific error issues that require further input.

[rajeshkamal5050]

Thanks @Austinauth @savannahostrowski

@vhvb1989 lets use the guidance from #243 (comment) on the specific error handling items we implement.