Skip to content

Commit

Permalink
Expose Text Attributes to UI Automation (#10336)
Browse files Browse the repository at this point in the history
## Summary of the Pull Request
This implements `GetAttributeValue` and `FindAttribute` for `UiaTextRangeBase` (the shared `ITextRangeProvider` for Conhost and Windows Terminal). This also updates `UiaTracing` to collect more useful information on these function calls. 

## References
#7000 - Epic
[Text Attribute Identifiers](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-textattribute-ids)
[ITextRangeProvider::GetAttributeValue](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-getattributevalue)
[ITextRangeProvider::FindAttribute](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-findattribute)

## PR Checklist
* [X] Closes #2161 
* [X] Tests added/passed

## Detailed Description of the Pull Request / Additional comments
- `TextBuffer`:
   - Exposes a new `TextBufferCellIterator` that takes in an end position. This simplifies the logic drastically as we can now use this iterator to navigate through the text buffer. The iterator can also expose the position in the buffer.
- `UiaTextRangeBase`:
   - Shared logic & helper functions:
      - Most of the text attributes are stored as `TextAttribute`s in the text buffer. To extract them, we generate an attribute verification function via `_getAttrVerificationFn()`, then use that to verify if a given cell has the desired attribute.
      - A few attributes are special (i.e. font name, font size, and "is read only"), in that they are (1) acquired differently and (2) consistent across the entire text buffer. These are handled separate from the attribute verification function.
   - `GetAttributeValue`: Retrieve the attribute verification of the first cell in the range. Then, verify that the entire range has that attribute by iterating through the text range. If a cell does not have that attribute, return the "reserved mixed attribute value".
   - `FindAttribute`: Iterate through the text range and leverage the attribute verification function to find the first contiguous range with that attribute. Then, make the end exclusive and output a `UiaTextRangeBase`. This function must be able to perform a search backwards, so we abstract the "start" and "end" into `resultFirstAnchor` and `resultSecondAnchor`, then perform post processing to output a valid `UiaTextRangeBase`.
- `UiaTracing`:
   - `GetAttributeValue`: Log uia text range, desired attribute, resulting attribute metadata, and the type of the result.
   - `FindAttribute`: Log uia text range, desired attribute and attribute metadata, if we were searching backwards, the type of the result, and the resulting text range.
   - `AttributeType` is a nice way to understand/record if the result was either of the reserved UIA values, a normal result, or an error.
- `UiaTextRangeTests`:
   - `GetAttributeValue`:
      - verify that we know which attributes we support
      - test each of the known text attributes (expecting 100% code coverage for `_getAttrVerificationFn()`)
   - `FindAttribute`: 
      - test each of the known _special_ text attributes
      - test `IsItalic`. NOTE: I'm explicitly only testing one of the standard text attributes because the logic is largely the same between all of them and they leverage `_getAttrVerificationFn()`.

## Validation Steps Performed
- @codeofdusk has been testing this Conhost build
- Tests added for Conhost and shared implementation
- Windows Terminal changes were manually verified using accessibility insights and NVDA
  • Loading branch information
carlos-zamora committed Jul 9, 2021
1 parent d57fb84 commit a0e5085
Show file tree
Hide file tree
Showing 17 changed files with 866 additions and 46 deletions.
5 changes: 5 additions & 0 deletions src/buffer/out/textBufferCellIterator.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -265,3 +265,8 @@ const OutputCellView* TextBufferCellIterator::operator->() const noexcept
{
return &_view;
}

COORD TextBufferCellIterator::Pos() const noexcept
{
return _pos;
}
2 changes: 2 additions & 0 deletions src/buffer/out/textBufferCellIterator.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ class TextBufferCellIterator
const OutputCellView& operator*() const noexcept;
const OutputCellView* operator->() const noexcept;

COORD Pos() const noexcept;

protected:
void _SetPos(const COORD newPos);
void _GenerateView();
Expand Down
2 changes: 2 additions & 0 deletions src/cascadia/TerminalControl/ControlCore.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,8 @@ namespace winrt::Microsoft::Terminal::Control::implementation
const int newDpi = static_cast<int>(static_cast<double>(USER_DEFAULT_SCREEN_DPI) *
_compositionScale);

_terminal->SetFontInfo(_actualFont);

// TODO: MSFT:20895307 If the font doesn't exist, this doesn't
// actually fail. We need a way to gracefully fallback.
_renderer->TriggerFontChange(newDpi, _desiredFont, _actualFont);
Expand Down
50 changes: 46 additions & 4 deletions src/cascadia/TerminalControl/XamlUiaTextRange.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
#include "XamlUiaTextRange.h"
#include "../types/TermControlUiaTextRange.hpp"
#include <UIAutomationClient.h>
#include <UIAutomationCoreApi.h>

// the same as COR_E_NOTSUPPORTED
// we don't want to import the CLR headers to get it
Expand Down Expand Up @@ -89,12 +90,52 @@ namespace winrt::Microsoft::Terminal::Control::implementation

winrt::Windows::Foundation::IInspectable XamlUiaTextRange::GetAttributeValue(int32_t textAttributeId) const
{
// Copied functionality from Types::UiaTextRange.cpp
if (textAttributeId == UIA_IsReadOnlyAttributeId)
// Call the function off of the underlying UiaTextRange.
VARIANT result;
THROW_IF_FAILED(_uiaProvider->GetAttributeValue(textAttributeId, &result));

// Convert the resulting VARIANT into a format that is consumable by XAML.
switch (result.vt)
{
case VT_BSTR:
{
return box_value(result.bstrVal);
}
case VT_I4:
{
// Surprisingly, `long` is _not_ a WinRT type.
// So we have to use `int32_t` to make sure this is output properly.
// Otherwise, you'll get "Attribute does not exist" out the other end.
return box_value<int32_t>(result.lVal);
}
case VT_R8:
{
return box_value(result.dblVal);
}
case VT_BOOL:
{
return winrt::box_value(false);
return box_value<bool>(result.boolVal);
}
else
case VT_UNKNOWN:
{
// This one is particularly special.
// We might return a special value like UiaGetReservedMixedAttributeValue
// or UiaGetReservedNotSupportedValue.
// Some text attributes may return a real value, however, none of those
// are supported at this time.
// So we need to figure out what was actually intended to be returned.

com_ptr<IUnknown> mixedAttributeVal;
UiaGetReservedMixedAttributeValue(mixedAttributeVal.put());

if (result.punkVal == mixedAttributeVal.get())
{
return Windows::UI::Xaml::DependencyProperty::UnsetValue();
}

[[fallthrough]];
}
default:
{
// We _need_ to return XAML_E_NOT_SUPPORTED here.
// Returning nullptr is an improper implementation of it being unsupported.
Expand All @@ -103,6 +144,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
// Magically, this doesn't affect other forms of navigation...
winrt::throw_hresult(XAML_E_NOT_SUPPORTED);
}
}
}

void XamlUiaTextRange::GetBoundingRectangles(com_array<double>& returnValue) const
Expand Down
1 change: 0 additions & 1 deletion src/cascadia/TerminalCore/Terminal.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
#include "../../terminal/parser/OutputStateMachineEngine.hpp"
#include "TerminalDispatch.hpp"
#include "../../inc/unicode.hpp"
#include "../../inc/DefaultSettings.h"
#include "../../inc/argb.h"
#include "../../types/inc/utils.hpp"
#include "../../types/inc/colorTable.hpp"
Expand Down
8 changes: 7 additions & 1 deletion src/cascadia/TerminalCore/Terminal.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

#include <conattrs.hpp>

#include "../../inc/DefaultSettings.h"
#include "../../buffer/out/textBuffer.hpp"
#include "../../types/inc/sgrStack.hpp"
#include "../../renderer/inc/BlinkingState.hpp"
Expand Down Expand Up @@ -68,6 +69,7 @@ class Microsoft::Terminal::Core::Terminal final :

void UpdateSettings(winrt::Microsoft::Terminal::Core::ICoreSettings settings);
void UpdateAppearance(const winrt::Microsoft::Terminal::Core::ICoreAppearance& appearance);
void SetFontInfo(const FontInfo& fontInfo);

// Write goes through the parser
void Write(std::wstring_view stringView);
Expand Down Expand Up @@ -160,6 +162,7 @@ class Microsoft::Terminal::Core::Terminal final :
COORD GetTextBufferEndPosition() const noexcept override;
const TextBuffer& GetTextBuffer() noexcept override;
const FontInfo& GetFontInfo() noexcept override;
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;

void LockConsole() noexcept override;
void UnlockConsole() noexcept override;
Expand All @@ -168,7 +171,6 @@ class Microsoft::Terminal::Core::Terminal final :
#pragma region IRenderData
// These methods are defined in TerminalRenderData.cpp
const TextAttribute GetDefaultBrushColors() noexcept override;
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;
COORD GetCursorPosition() const noexcept override;
bool IsCursorVisible() const noexcept override;
bool IsCursorOn() const noexcept override;
Expand Down Expand Up @@ -276,6 +278,10 @@ class Microsoft::Terminal::Core::Terminal final :
size_t _hyperlinkPatternId;

std::wstring _workingDirectory;

// This default fake font value is only used to check if the font is a raster font.
// Otherwise, the font is changed to a real value with the renderer via TriggerFontChange.
FontInfo _fontInfo{ DEFAULT_FONT_FACE, TMPF_TRUETYPE, 10, { 0, DEFAULT_FONT_SIZE }, CP_UTF8, false };
#pragma region Text Selection
// a selection is represented as a range between two COORDs (start and end)
// the pivot is the COORD that remains selected when you extend a selection in any direction
Expand Down
20 changes: 6 additions & 14 deletions src/cascadia/TerminalCore/terminalrenderdata.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,23 +27,15 @@ const TextBuffer& Terminal::GetTextBuffer() noexcept
return *_buffer;
}

// Creating a FontInfo can technically throw (on string allocation) and this is noexcept.
// That means this will std::terminate. We could come back and make there be a default constructor
// backup to FontInfo that throws no exceptions and allocates a default FontInfo structure.
#pragma warning(push)
#pragma warning(disable : 26447)
const FontInfo& Terminal::GetFontInfo() noexcept
{
// TODO: This font value is only used to check if the font is a raster font.
// Otherwise, the font is changed with the renderer via TriggerFontChange.
// The renderer never uses any of the other members from the value returned
// by this method.
// We could very likely replace this with just an IsRasterFont method
// (which would return false)
static const FontInfo _fakeFontInfo(DEFAULT_FONT_FACE, TMPF_TRUETYPE, 10, { 0, DEFAULT_FONT_SIZE }, CP_UTF8, false);
return _fakeFontInfo;
return _fontInfo;
}

void Terminal::SetFontInfo(const FontInfo& fontInfo)
{
_fontInfo = fontInfo;
}
#pragma warning(pop)

const TextAttribute Terminal::GetDefaultBrushColors() noexcept
{
Expand Down
3 changes: 1 addition & 2 deletions src/host/renderData.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ class RenderData final :
COORD GetTextBufferEndPosition() const noexcept override;
const TextBuffer& GetTextBuffer() noexcept override;
const FontInfo& GetFontInfo() noexcept override;
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;

std::vector<Microsoft::Console::Types::Viewport> GetSelectionRects() noexcept override;

Expand All @@ -37,8 +38,6 @@ class RenderData final :
#pragma region IRenderData
const TextAttribute GetDefaultBrushColors() noexcept override;

std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;

COORD GetCursorPosition() const noexcept override;
bool IsCursorVisible() const noexcept override;
bool IsCursorOn() const noexcept override;
Expand Down
Loading

0 comments on commit a0e5085

Please sign in to comment.