Merge pull request #107 from BethanyJep/main

Docs: debugging prompty and minor changes

web/docs/getting-started/debugging-prompty/page.mdx

@@ -0,0 +1,156 @@
+---
+title: Debuggging Prompty
+authors:
+  - bethanyjep
+  - nitya
+date: 2024-10-22
+tags:
+  - getting-started
+  - documentation
+  - quickstart
+index: 6
+---
+
+_In the last section, we converted our Prompty asset into code and successfully executed the application. In this section, we will cover how we can use Observability in Prompty to debug our application._
+
+<br/>
+
+## 1. What we will cover
+
+For observability in Prompty, we will use the tracer to visualize and debug the execution of the Prompty asset through the following steps:
+
+- Understand observability in Prompty
+- Build and debug a Prompty asset
+- Understand how observability works in your code
+- Analyze the trace output to debug and fix the bug
+
+<br/>
+
+## 2. Understandfing Observability in Prompty
+
+Observability refers to the ability to monitor and understand the behavior of a system based on the data it produces, such as logs, metrics, and traces.. It is important as it provides you with insights on your LLM workflows and provides a way to debug your prompt inputs and outputs. 
+
+In Prompty, you can easily trace and visualize flow, which helps you to understand and debug your code using the built-in tracer. Additionally, you can create your own tracer by adding your own hook. By default, Prompty has two traces built-in:
+
+- **Console Tracer**: This tracer logs the output to the console.
+- **Prompty Tracer**: This tracer logs the output to a JSON file.
+
+<br/>
+
+## 3. Modify our Prompty
+In our `shakespeare.prompty` asset we will update the prompt to request for different variations of the same message. The new prompt will be: `"Can you create 5 different versions of a short message inviting friends to a Game Night?"`. Additionally, change the `max_tokens:` value from `3000` to `150`.
+
+Head over to the `shakespeare.py` file as well and update the question to: `"Can you create 5 different versions of a short message inviting friends to a Game Night?"`.
+
+<br/>
+
+<details>
+<summary>☑ **Function that executes the Prompty asset** (click to expand)</summary>
+```markdown
+---
+name: Shakespearean Writing Prompty
+description: A prompt that answers questions in Shakespearean style using Cohere Command-R model from GitHub Marketplace.
+authors:
+  - Bethany Jepchumba
+model:
+  api: chat
+  configuration:
+    type: azure_openai
+    azure_endpoint: ${env:AZURE_OPENAI_ENDPOINT}
+    azure_deployment: gpt-4o
+  parameters:
+    max_tokens: 150
+sample:
+  question: Can you create 5 different versions of a short message inviting friends to a Game Night?
+---
+
+system:
+You are a Shakespearean writing assistant who speaks in a` Shakespearean style. You help people come up with creative ideas and content like stories, poems, and songs that use Shakespearean style of writing style, including words like "thou" and "hath”.
+Here are some example of Shakespeare's style:
+- Romeo, Romeo! Wherefore art thou Romeo?
+- Love looks not with the eyes, but with the mind; and therefore is winged Cupid painted blind.
+- Shall I compare thee to a summer's day? Thou art more lovely and more temperate.
+
+example:
+user: Please write a short text turning down an invitation to dinner.
+assistant: Dearest,
+  Regretfully, I must decline thy invitation.
+  Prior engagements call me hence. Apologies.
+
+user:
+{{question}}
+```
+</details>
+
+<br/>
+
+## 4. Adding observability to your code
+To add a tracer, we have the following in our previously generated code snippet:
+
+```python
+from prompty.tracer import trace, Tracer, console_tracer, PromptyTracer
+
+Tracer.add("console", console_tracer)
+json_tracer = PromptyTracer()
+Tracer.add("PromptyTracer", json_tracer.tracer)
+
+@trace
+def run(    
+      question: any
+) -> str:
+
+  # execute the prompty file
+  result = prompty.execute(
+    "shakespeare.prompty", 
+    inputs={
+      "question": question
+    }
+  )
+```
+<br/>
+
+- **Tracer.add("console", console_tracer)**: logs tracing information to the console, useful for real-time debugging.
+- **json_tracer = PromptyTracer()**: Creates an instance of the PromptyTracer class, which is a custom tracer.
+- **Tracer.add("PromptyTracer", json_tracer.tracer)**: logs tracing in a `.tracy` JSON file for more detailed inspection after runs, providing you with an interactive UI.
+- **@trace**: Decorator that traces the execution of the run function.
+
+<br/>
+
+## 5: Analyzing and debugging the trace output
+
+The output from the tracer is displayed in the console and in a `.tracy` file. A new `.tracy` file is created in a new `.runs` folder. 
+
+The trace output is divided into three: _load, prepare_ and _run_. Load refers to the loading of the Prompty asset, prepare refers to the preparation of the Prompty asset, and run refers to the execution of the Prompty asset. Below is a sample of the trace output, showing the inputs, outputs, and metrics, such as execution time and token count:
+
+> **Note:** it may take a while for the trace output to appear, and you may need to click several runs before seeing the full trace.
+
+![Trace Output](trace-output.png)
+
+<br/>
+
+From the trace output, you can see the inputs, outputs and metrics such as time to execute the prompt and tokens. This information can be used to debug and fix any issues in your code. For example, we can see output has been truncated and the `Completion Tokens` count is less than 1000, which might not be sufficent for the prompt to generate different outputs. We can increase the `max_tokens` in our Prompty to 1000 to generate more tokens. Once done, run the code again and confirm you get 5 examples of the short message inviting friends to a Game Night.
+
+![updated trace output](trace-bug-fixed.png)
+
+You can continue experimenting with different parameters such as `temperature` and observe how it affects the model outputs.
+
+<br/>
+
+## 6. Using observability for Model Selection
+
+Another way to make the most of observability is in Model Selection. You can switch between models and observe their performance such as completion tokens, latency and accuracy for different tasks. For example, you can switch between the `gpt-4o` and `gpt-35-turbo` models and observe the performance of each model. You can also leverage on GitHub Models, Azure OpenAI and other models to observe the performance of each model. Below is a comparison of the trace output for the `gpt-4o` and `gpt-35-turbo` models:
+
+![grpt-35-turbo output](gpt-35-turbo-trace.png)
+
+<br/>
+
+From the output, you can see the difference in the completion tokens and the time taken to execute the prompt. This information can be used to select the best model for your use case.
+
+
+## 7. Building a Custom Tracer in Prompty
+
+In the guides section, we will provide a deep dive into [Observability in Prompty](docs/guides/prompty-observability) and how you can create your own tracer.
+
+---
+<br/>
+[Want to Contribute To the Project?](/docs/contributing/) - _Updated Guidance Coming Soon_.

web/docs/getting-started/debugging-prompty/shakespeare.prompty

@@ -0,0 +1,32 @@
+---
+name: Shakespearean Writing Prompty
+description: A prompt that answers questions in Shakespearean style using Cohere Command-R model from GitHub Marketplace.
+authors:
+  - Bethany Jepchumba
+model:
+  api: chat
+  configuration:
+    type: azure_openai
+    azure_endpoint: ${env:AZURE_OPENAI_ENDPOINT}
+    azure_deployment: gpt-4o
+  parameters:
+    max_tokens: 150
+sample:
+  question: Can you create 5 different versions of a short message inviting friends to a Game Night?
+---
+
+system:
+You are a Shakespearean writing assistant who speaks in a` Shakespearean style. You help people come up with creative ideas and content like stories, poems, and songs that use Shakespearean style of writing style, including words like "thou" and "hath”.
+Here are some example of Shakespeare's style:
+- Romeo, Romeo! Wherefore art thou Romeo?
+- Love looks not with the eyes, but with the mind; and therefore is winged Cupid painted blind.
+- Shall I compare thee to a summer's day? Thou art more lovely and more temperate.
+
+example:
+user: Please write a short text turning down an invitation to dinner.
+assistant: Dearest,
+  Regretfully, I must decline thy invitation.
+  Prior engagements call me hence. Apologies.
+
+user:
+{{question}}

web/docs/getting-started/debugging-prompty/shakespheare.py

@@ -0,0 +1,43 @@
+import json
+import prompty
+# to use the azure invoker make 
+# sure to install prompty like this:
+# pip install prompty[azure]
+import prompty.azure
+from prompty.tracer import trace, Tracer, console_tracer, PromptyTracer
+
+# add console and json tracer:
+# this only has to be done once
+# at application startup
+Tracer.add("console", console_tracer)
+json_tracer = PromptyTracer()
+Tracer.add("PromptyTracer", json_tracer.tracer)
+
+# if your prompty file uses environment variables make
+# sure they are loaded properly for correct execution
+from dotenv import load_dotenv
+load_dotenv()
+
+@trace
+def run(    
+      question: any
+) -> str:
+
+  # execute the prompty file
+  result = prompty.execute(
+    "shakespeare.prompty", 
+    inputs={
+      "question": question
+    }
+  )
+
+  return result
+
+if __name__ == "__main__":
+   json_input = '''{
+  "question": "Can you create 5 different versions of a short message inviting friends to a Game Night?"
+}'''
+   args = json.loads(json_input)
+
+   result = run(**args)
+   print(result)

web/docs/getting-started/page.mdx

@@ -3,7 +3,7 @@ title: Getting Started
 authors:
   - bethanyjep
   - nitya
-date: 2024-08-14
+date: 2024-10-22
 tags:
   - getting-started
   - documentation
@@ -17,6 +17,7 @@ _In this section we take you from core concepts to code, covering the following
 - **Setup**: Install the Prompty developer tooling (VS Code Extension and SDK)
 - **First Prompty**: Build and run your first Prompty from VS Code
 - **First App**: Convert your Prompty to code (with SDK) and execute it.
+- **Debugging**: Use Observability in Prompty to debug your application
 
 <br/>
 

web/docs/getting-started/prompty-to-code/page.mdx

@@ -3,7 +3,7 @@ title: Prompty To Code
 authors:
   - bethanyjep
   - nitya
-date: 2024-10-09
+date: 2024-10-21
 tags:
   - getting-started
   - documentation
@@ -17,7 +17,7 @@ _In the last section, we learned how to create and run a Prompty asset. In this
 
 To convert a Prompty asset to code and execute your first app, you need to have the following installed:
 
-- [Python 3.6 or higher](https://www.python.org/downloads/)
+- [Python 3.10 or higher](https://www.python.org/downloads/)
 - [Prompty Package (Python library)](https://pypi.org/project/prompty/)
 
 <br/>
@@ -124,17 +124,7 @@ Faithfully thine,
 
 <br/>
 
-## 6. Prompty invokers
-
-The Prompty runtime comes with a set of built-in invokers that can be used to execute external models and APIs. 
-Invokers trigger a call to a the different models and return its output ensuring standardization when it comes to handling models. The invokers currently supported are:
-
-1. **azure**: Invokes the Azure OpenAI API
-2. **openai**: Invokes the OpenAI API
-3. **serverless**: Invokes serverless models (e.g. GitHub Models) using the Azure AI Inference client library (currently only key based authentication is supported with more managed identity support coming soon)
-
-<br/>
-## 7. How Python code works
+## 6. How Python code works
 
 1. The ``.py`` code generated first imports the necessary modules and libraries. 
 
@@ -217,37 +207,11 @@ if __name__ == "__main__":
 
 <br/>
 
-## 6. Observability in Prompty
-
-Observability refers to the ability to monitor and understand the behavior of a system based on the data it produces such as logs, metrics and traces. In Prompty, you can easily trace and visualize flow, which helps you to understand and debug your code using the built-in tracer. You can create your own tracer by adding your own hook. By default, Prompty has two traces built-in:
-
-- **Console Tracer**: This tracer logs the output to the console.
-- **Prompty Tracer**: This tracer logs the output to a JSON file.
 
-<br/>
-
-To add a tracer, we have the following code snippet:
-
-```python
-from prompty.tracer import trace, Tracer, console_tracer, PromptyTracer
-
-Tracer.add("console", console_tracer)
-json_tracer = PromptyTracer()
-Tracer.add("PromptyTracer", json_tracer.tracer)
-```
-<br/>
-
-The output from the tracer is displayed in the console and in a `.tracy` file. A new `.tracy` file is created in a new `.runs` folder. 
-The trace output is divided into three: _load, prepare_ and _run_. Load refers to the loading of the Prompty asset, prepare refers to the preparation of the Prompty asset, and run refers to the execution of the Prompty asset. Below is a sample of the trace output showing the inputs, outputs and metrics i.e. time to execute the prompt and tokens:
-
-![Sample trace output](tracy_output.png)
-
-{/* to add: output of the  tracer */}
-<br/>
 
 ## 7. Additional supported runtimes
 
-The Prompty runtime supports additional runtimes, including frameworks such as LangChain, and Semantic Kernel. In the tutorials section, we will cover how to generate code from Prompty assets using these runtimes. (coming soon)
+The Prompty runtime supports additional runtimes, including frameworks such as [LangChain](/docs/tutorials/using-langchain), and [Semantic Kernel](/docs/tutorials/using-semantic-kernel). In the [tutorials](/docs/tutorials) section, we will cover how to generate code from Prompty assets using these runtimes. (coming soon)
 
 
 ---

web/docs/guides/prompty-invoker/page.mdx

@@ -12,6 +12,19 @@ tags:
 index: 1
 ---
 
+_In this section, we cover the different built-in Prompty Invokers and walk you through how you can build your own custom invoker._
+
+## 1. Prompty invokers
+
+The Prompty runtime comes with a set of built-in invokers that can be used to execute external models and APIs. 
+Invokers trigger a call to a the different models and return its output ensuring standardization when it comes to handling models. The invokers currently supported are:
+
+1. **azure**: Invokes the Azure OpenAI API
+2. **openai**: Invokes the OpenAI API
+3. **serverless**: Invokes serverless models (e.g. GitHub Models) using the Azure AI Inference client library (currently only key based authentication is supported with more managed identity support coming soon)
+
+<br/>
+
 TODO: Explain how invokers work and how to build a custom invoker
 
 <br/>

web/docs/guides/prompty-observability/page.mdx

@@ -4,15 +4,17 @@ authors:
   - bethanyjep
   - nitya
   - sethjuarez
-date: 2024-06-10
+date: 2024-10-22
 tags:
   - tutorials
   - runtime
 index: 4
 ---
 
-TODO: Explain how to trace Prompty execution from input to response
+Get started with Observability at [debugging Prompty](/docs/getting-started/debugging-prompty), check out additional documentation at [Observability](https://github.com/microsoft/prompty/tree/main/runtime/prompty#using-tracing-in-prompty).
+
 
+TODO: Explain how to trace Prompty execution from input to response
 <br/>
 ---
 <br/>