Add an AI chat experience to Windows Terminal, powered by the user's Azure OpenAI resource #16285

PankajBhojwani · 2023-11-09T20:14:34Z

Summary of the Pull Request

Adds an AI chatbot to Windows Terminal. Currently we do not ship with our own LLM, the user needs to provide their own Azure OpenAI subscription to be able to use this feature.

A new settings page has been added where the user can input their Azure OpenAI endpoint and key
A new palette has been added to the dropdown, called Terminal Chat
From Terminal Chat, the user can make queries to the provided endpoint for assistance with shell commands
We let the endpoint know about the user's current shell so that (hopefully) the commands received are relevant to the user's context
Received commands can be clicked from within the palette to be input into the user's active shell
The system prompt, alongside Azure OpenAI's in-built safeguards, should prevent strange/inappropriate replies from the LLM

Validation Steps Performed

References and Relevant Issues

#15000

PR Checklist

Closes #xxx
Tests added/passed
Documentation updated
- If checked, please file a pull request on our docs repo and link it here: #xxx
Schema updated (if necessary)

zadjii-msft

31/56 and I've obviously only done the easy ones

src/cascadia/TerminalApp/App.cpp

src/renderer/atlas/AtlasEngine.cpp

src/cascadia/TerminalSettingsEditor/init.cpp

src/cascadia/TerminalSettingsEditor/MainPage.xaml

src/cascadia/TerminalApp/TerminalPage.h

src/cascadia/QueryExtension/pch.h

src/cascadia/QueryExtension/init.cpp

src/cascadia/QueryExtension/ExtensionPalette.cpp

…minal_chat

…e the partC data

nguyen-dows · 2023-11-16T01:11:24Z

@check-spelling-bot Report

🔴 Please review

See the 📂 files view or the 📜action log for details.

Unrecognized words (6)

CVS desploy endpt gpt hyperlinked openai

Previously acknowledged words that are now absent
To accept ✔️ these unrecognized words as correct and remove the previously acknowledged and now absent words, run the following commands
Pattern suggestions ✂️ (1)
Warnings (1)
✏️ Contributor please read this

We definitely got to fix that "desploy", I think everything else that the spell-bot detected is fine though.

DHowett · 2023-11-16T01:16:44Z

doc/cascadia/profiles.schema.json

@@ -393,6 +393,7 @@
        "addMark",
        "adjustFontSize",
        "adjustOpacity",
+        "AIChat",


nts: make this openTerminalChat

DHowett · 2023-11-16T03:01:45Z

Notes:

Issues:
- We hardcoded the colors so light mode doesn't work
- The only assets we have are dark mode, and xaml doesn't support the theme-light qualifier. I had to switch to SVGs for the menu and settings.
- Our resource string structure, with strings split into two or more pieces, does not work well for localization.

I forced us into dark mode to fix 1.
I removed all Audit and Fuzzing configurations for this project from the SLN.
I removed all AnyCPU configurations from the SLN.
The tracelogging was emitting different payloads (EndpointSet / EndpointStored)
The tracelogging didn't know endpoints were set because it didn't request the endpoint from the app...
We no longer crash inserting an answer into the settings UI

DHowett · 2023-11-16T19:22:12Z

src/cascadia/QueryExtension/ExtensionPalette.xaml

+                                Padding="16,8,16,8"
+                                CornerRadius="8">
+                        <StackPanel.Background>
+                            <SolidColorBrush Color="#005EB7" />


In the future, we need to use existing xaml color ramp items. Otherwise, we lose theme support.

src/cascadia/TerminalApp/TerminalPage.cpp

src/cascadia/QueryExtension/ExtensionPalette.cpp

DHowett · 2023-11-16T19:33:06Z

src/cascadia/QueryExtension/ExtensionPalette.cpp

+    {
+        _AIEndpoint = endpoint;
+        _AIKey = key;
+        _httpClient = winrt::Windows::Web::Http::HttpClient{};


probably don't need to recreate this every time these change, but I'm OK with it for now

DHowett · 2023-11-16T19:35:03Z

src/cascadia/QueryExtension/ExtensionPalette.cpp

+        {
+            result = RS_(L"UnknownErrorMessage");
+
+            TraceLoggingWrite(


too many copies of error handling code. streamline.

src/cascadia/QueryExtension/ExtensionPalette.cpp

DHowett

fixed stuff

zadjii-msft

🤏👃,

Let's get these tracked somewhere. There's lots

zadjii-msft · 2023-11-16T20:04:58Z

src/cascadia/QueryExtension/Microsoft.Terminal.Query.Extension.vcxproj

+    <SubSystem>Console</SubSystem>
+    <!-- sets a bunch of Windows Universal properties -->
+    <OpenConsoleUniversalApp>true</OpenConsoleUniversalApp>
+    <PgoTarget>false</PgoTarget>


Eh, it's mostly web request bound.

zadjii-msft · 2023-11-16T20:07:03Z

src/cascadia/QueryExtension/Resources/en-US/Resources.resw

+  <data name="AIContentDisclaimerPartOne.Text" xml:space="preserve">
+    <value>AI can make mistakes —</value>
+    <comment>Part one of the disclaimer presented to the user within the chat UI element.</comment>
+  </data>
+  <data name="AIContentDisclaimerHyperlink" xml:space="preserve">
+    <value>send feedback</value>
+    <comment>The portion of the disclaimer presented to the user as a hyperlink within the chat UI element.</comment>
+  </data>
+  <data name="AIContentDisclaimerPartTwo.Text" xml:space="preserve">
+    <value>to help us improve.</value>
+    <comment>Part two of the disclaimer presented to the user within the chat UI element.</comment>
+  </data>


I'd bet that localization eats it on this - breaking the sentence across resources is surely gonna blow up.

Given that we only want the relevant parts of the sentence to be hyperlinked, I'm not entirely sure how we can solve this without breaking it up in the resource file... will file a work item to investigate if/how this messes with localization

zadjii-msft · 2023-11-16T20:09:15Z

src/cascadia/TerminalApp/TerminalPage.cpp

+                    "0.4001 0 0.2209 0.17909 0.4 0.40003 0.4h3.2003zm-3.2003 1.6001h1.6001c0.221 "
+                    "0 0.4001-0.1791 0.4001-0.4s-0.1791-0.4-0.4001-0.4h-1.6001c-0.22094 0-0.40003 "
+                    "0.1791-0.40003 0.4s0.17909 0.4 0.40003 0.4z"
+                };


that's on me, sorry XD

zadjii-msft · 2023-11-16T20:16:40Z

src/cascadia/TerminalApp/TerminalPage.cpp

+            }
+
+            AIChatFlyout.Click({ this, &TerminalPage::_AIChatButtonOnClick });
+            newTabFlyout.Items().Append(AIChatFlyout);


do we append this item always?

I suppose the feature isn't behind velocity, it's in a separate branch. So this will always be in the menu for branches where it's checked in.

Honestly, don't love that, but 🤷

We can make Velocitizing it one of the go live requirements

zadjii-msft · 2023-11-16T20:17:18Z

src/cascadia/TerminalApp/TerminalPage.cpp

@@ -880,7 +949,7 @@ namespace winrt::TerminalApp::implementation
        });
        // Necessary for fly-out sub items to get focus on a tab before collapsing. Related to #15049
        newTabFlyout.Closing([this](auto&&, auto&&) {
-            if (!_commandPaletteIs(Visibility::Visible))
+            if (!_commandPaletteIs(Visibility::Visible) && (ExtensionPresenter().Visibility() != Visibility::Visible))


added the extension presenter here, but we totally still forgot the sxnui

zadjii-msft · 2023-11-16T22:50:18Z

src/cascadia/QueryExtension/ExtensionPalette.cpp

+    void ExtensionPalette::_backdropPointerPressed(const Windows::Foundation::IInspectable& /*sender*/,
+                                                   const Windows::UI::Xaml::Input::PointerRoutedEventArgs& e)
+    {
+        e.Handled(true);


This was for light-dismiss yea? can we just toss it entirely now?

This is to prevent the palette from dismissing when the user clicks within it, we only want light dismiss if they click out of it. So we still need this here

src/cascadia/QueryExtension/ExtensionPalette.cpp

zadjii-msft · 2023-11-16T22:53:07Z

src/cascadia/QueryExtension/ExtensionPalette.cpp

+        else if (key == VirtualKey::C && ctrlDown)
+        {
+            _queryBox().CopySelectionToClipboard();
+            e.Handled(true);
+        }
+        else if (key == VirtualKey::V && ctrlDown)
+        {
+            _queryBox().PasteFromClipboard();
+            e.Handled(true);
+        }


shouldn't this... just work?

or is this for when the focus isn't in the text box and you want to copy/paste into the text box. okay I think I get it

Nope, NOPE. NOPE. TermControl's Preview Key Handler gets ^V before anyone else. Remember, it is awful.

zadjii-msft · 2023-11-16T22:54:03Z

src/cascadia/QueryExtension/ExtensionPalette.xaml

+                                               Width="64"
+                                               Height="64"
+                                               Margin="0,0,0,20"
+                                               Source="ms-appx:///ProfileIcons/terminalChatLogo.png" />


does this one work with light & dark theme?

oh wait no this is the thing that has to be in dark mode isn't it

zadjii-msft · 2023-11-16T22:54:22Z

src/cascadia/QueryExtension/ExtensionPalette.xaml

+            </TextBlock>
+        </Grid>
+    </Grid>
+</UserControl>


yep this is XAML

DHowett · 2023-11-16T23:01:21Z

Kept the branch so Pankaj can see my changes

Docs for microsoft/terminal#16285

- [x] Remove the ESC handler that clears the query message - [x] Streamline error handling code - [x] Fix `this` and `&` in several lambdas - [x] Fix getting the [active commandline properly](#16285 (comment)) - [x] Use XAML color ramp resource names instead of hardcoded colors

squash commit of terminal chat

f15b596

This comment has been minimized.

Sign in to view

zadjii-msft reviewed Nov 9, 2023

View reviewed changes

zadjii-msft mentioned this pull request Nov 9, 2023

[Scenario]: Terminal Chat #15000

Open

Merge remote-tracking branch 'origin/feature/llm' into dev/pabhoj/ter…

44a1125

…minal_chat

This comment has been minimized.

Sign in to view

DHowett added 2 commits November 15, 2023 18:52

Fix the tracelog issue where KeyAndEndpointSet always equals 0; renam…

cd5f7ee

…e the partC data

Stop crashing when inserting a suggestion into the settings UI (heh)

5c2b199

DHowett reviewed Nov 16, 2023

View reviewed changes

Back out the changes to atlas and common files

03abed8

nguyen-dows mentioned this pull request Nov 16, 2023

Added Terminal Chat to the docs MicrosoftDocs/terminal#728

Merged

DHowett added 6 commits November 15, 2023 19:25

Propagate the Terminal Chat name change to the settings model and editor

bdee2c9

Privacy Policy -> Privacy Statement

28c4df3

That is not where BitmapIcon lives

b17c465

gpt-3.5-turbo -> gpt-35-turbo

a2c9080

Force the chat into dark theme

60132b3

We can't use PNGs. It's so bad.

cd827d7

This comment has been minimized.

Sign in to view

DHowett added 6 commits November 15, 2023 20:42

Clean up the includes

dd63791

Clean up the solution

179dd76

Spelling: cvs and desploy

2a7a03c

Spelling: endpt and hyperlinked

cf69e50

Spelling: openai and gpt

6bc79f1

Disable metadata provider gen; we are not an app

878fd89

This comment has been minimized.

Sign in to view

Mike's comment about pch

9b4133b

This comment has been minimized.

Sign in to view

Another Mike pch

b07df92

This comment has been minimized.

Sign in to view

Spelling: [H]yperlinked

19616e2

Lazy load the whole extension... properly.

95ef16c

DHowett requested changes Nov 16, 2023

View reviewed changes

Rename the thing named test to have a better name

6637bee

DHowett approved these changes Nov 16, 2023

View reviewed changes

microsoft-github-policy-service bot added the Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something label Nov 16, 2023

zadjii-msft approved these changes Nov 16, 2023

View reviewed changes

DHowett merged commit 32cfa5a into feature/llm Nov 16, 2023
15 of 17 checks passed

DHowett deleted the dev/pabhoj/terminal_chat branch November 16, 2023 23:00

DHowett restored the dev/pabhoj/terminal_chat branch November 16, 2023 23:01

pull bot pushed a commit to rbleattler/terminal that referenced this pull request Nov 18, 2023

Added Terminal Chat to the docs (MicrosoftDocs#728)

44d4036

Docs for microsoft/terminal#16285

microsoft-github-policy-service bot removed the Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something label Nov 27, 2023

PankajBhojwani mentioned this pull request Nov 27, 2023

Resolve several nits in the Terminal AI code #16382

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an AI chat experience to Windows Terminal, powered by the user's Azure OpenAI resource #16285

Add an AI chat experience to Windows Terminal, powered by the user's Azure OpenAI resource #16285

PankajBhojwani commented Nov 9, 2023 •

edited

Loading

This comment has been minimized.

zadjii-msft left a comment

This comment has been minimized.

nguyen-dows commented Nov 16, 2023

@check-spelling-bot Report

🔴 Please review

Unrecognized words (6)

DHowett Nov 16, 2023

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

DHowett commented Nov 16, 2023

DHowett Nov 16, 2023

DHowett Nov 16, 2023

DHowett Nov 16, 2023

DHowett left a comment

zadjii-msft left a comment

zadjii-msft Nov 16, 2023

DHowett Nov 16, 2023

zadjii-msft Nov 16, 2023

PankajBhojwani Nov 27, 2023

zadjii-msft Nov 16, 2023

DHowett Nov 16, 2023

zadjii-msft Nov 16, 2023

DHowett Nov 16, 2023

zadjii-msft Nov 16, 2023

zadjii-msft Nov 16, 2023

PankajBhojwani Nov 27, 2023 •

edited

Loading

zadjii-msft Nov 16, 2023

DHowett Nov 16, 2023

zadjii-msft Nov 16, 2023

zadjii-msft Nov 16, 2023

DHowett commented Nov 16, 2023

Add an AI chat experience to Windows Terminal, powered by the user's Azure OpenAI resource #16285

Add an AI chat experience to Windows Terminal, powered by the user's Azure OpenAI resource #16285

Conversation

PankajBhojwani commented Nov 9, 2023 • edited Loading

Summary of the Pull Request

Validation Steps Performed

References and Relevant Issues

PR Checklist

This comment has been minimized.

zadjii-msft left a comment

Choose a reason for hiding this comment

This comment has been minimized.

nguyen-dows commented Nov 16, 2023

@check-spelling-bot Report

🔴 Please review

Unrecognized words (6)

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

DHowett commented Nov 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DHowett left a comment

Choose a reason for hiding this comment

zadjii-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PankajBhojwani Nov 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DHowett commented Nov 16, 2023

PankajBhojwani commented Nov 9, 2023 •

edited

Loading

PankajBhojwani Nov 27, 2023 •

edited

Loading