From e610180e6329169ba945f2ed1675da616e88ba87 Mon Sep 17 00:00:00 2001 From: Guangya Liu Date: Fri, 6 Dec 2024 11:48:00 -0500 Subject: [PATCH] AI Agent Span Semantic Convention --- docs/gen-ai/gen-ai-agent-spans.md | 95 +++++++++++++++++++++ model/gen-ai/registry.yaml | 122 +++++++++++++++++++++++++++ model/gen-ai/spans.yaml | 132 ++++++++++++++++++++++++++++++ 3 files changed, 349 insertions(+) create mode 100644 docs/gen-ai/gen-ai-agent-spans.md diff --git a/docs/gen-ai/gen-ai-agent-spans.md b/docs/gen-ai/gen-ai-agent-spans.md new file mode 100644 index 0000000000..82d69d2028 --- /dev/null +++ b/docs/gen-ai/gen-ai-agent-spans.md @@ -0,0 +1,95 @@ + + +# Semantic Conventions for GenAI agent and framework spans + +**Status**: [Experimental][DocumentStatus] + + + + + +- [SPAN KIND](#span-kind) +- [Name](#name) +- [AI Agent attributes](#ai-agent-attributes) + - [Workflow Attributes](#workflow-attributes) + - [Agent Attributes](#agent-attributes) + - [Tools Attributes](#tools-attributes) + - [Task Attributes](#task-attributes) + - [Interaction Attributes](#interaction-attributes) +- [Examples](#examples) + - [Workflow: Market Analysis Pipeline](#workflow-market-analysis-pipeline) + - [Agents Involved](#agents-involved) + - [DataCollectorAgent](#datacollectoragent) + - [DataAnalystAgent](#dataanalystagent) + - [Tasks and Tools](#tasks-and-tools) + - [Task: Data Collection](#task-data-collection) + - [Tool Utilized: WebScraperTool](#tool-utilized-webscrapertool) + - [Task: Data Analysis](#task-data-analysis) + - [Tool Utilized: DataAnalyzerTool](#tool-utilized-dataanalyzertool) + + + +Generative AI scenarios may involve multiple steps. For example RAG pattern implementation usually involves +the following steps: + +- initial request to the GenAI model to rewrite user query +- getting embeddings for the rewritten query +- searching for similar documents in the vector store using embeddings +- getting completion from the GenAI model based on the similar documents. + +These steps may be done by the application itself using GenAI models and vector databases. +It's also common to leverage client framework such as [LangGraph](https://github.com/langchain-ai/langgraph) +or service offerings such as [OpenAI Assistants](https://platform.openai.com/docs/assistants), +[Azure AI Agents](TODO link), or [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/). + +This document defines semantic conventions for GenAI agent calls that are +handled by the remote agent service. +It MAY be applicable to agent operations that are performed by the GenAI +framework locally. + +The semantic conventions for GenAI agents extend and override the semantic conventions +for [Gen AI Spans](gen-ai-spans.md). + +## Spans + +### Create Workflow Span + +### Create Agent Span + +Describes GenAI agent creation and is usually applicable when working with remote agent +services. + +The `gen_ai.operation.name` MUST be `create_agent`. + +The **span name** SHOULD be `create_agent {gen_ai.agent.name}`. +Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + + + + +### Create Task Span + + + + +### Create Tool Span + + + + +### Run Tool Span + + + + +### Run Task Span + + + + +### Run Workflow Span + + + diff --git a/model/gen-ai/registry.yaml b/model/gen-ai/registry.yaml index 918b0a2f21..f66afe294e 100644 --- a/model/gen-ai/registry.yaml +++ b/model/gen-ai/registry.yaml @@ -141,6 +141,96 @@ groups: brief: 'Output tokens (completion, response, etc.)' brief: The type of token being counted. examples: ['input', 'output'] + - id: gen_ai.agent.workflow.name + stability: experimental + type: string + brief: The name of the workflow that the agent is part of. + examples: ["Data Processing Pipeline"] + - id: gen_ai.agent.workflow.tasks + stability: experimental + type: string[] + brief: List of tasks included in the workflow. + examples: ["Data Collection", "Data Analysis"] + - id: gen_ai.agent.workflow.agents + stability: experimental + type: string[] + brief: System or environment where the workflow is executed.. + examples: ["CrewAI", "LangGraph"] + - id: gen_ai.agent.workflow.system + stability: experimental + type: string + brief: The GenAI system used to run the workflow. + examples: ["OpenAI"] + - id: gen_ai.agent.id # alternatives: assistant (openai) + stability: experimental + type: string + brief: The unique identifier of the GenAI agent. + examples: ['asst_5j66UpCpwteGg4YSxUnt7lPY'] + - id: gen_ai.agent.name + stability: experimental + type: string + brief: Human-readable name of the GenAI agent provided by the application. + examples: ["Math Tutor", "Fiction Writer"] + - id: gen_ai.agent.description + stability: experimental + type: string + brief: Free-form description of the GenAI agent provided by the application. + examples: ["Helps with math problems", "Generates fiction stories"] + - id: gen_ai.agent.workflow_name + stability: experimental + type: string + brief: The name of the workflow that the agent is part of. + examples: ["Math Tutoring", "Fiction Writing"] + - id: gen_ai.agent.tools + stability: experimental + type: string[] + brief: List of tools available to the agent. + examples: ["Web Scraper", "Analyzer"] + - id: gen_ai.agent.tool.name + stability: experimental + type: string + brief: Name of the tool utilized by the agent. + examples: ["Web Scraper"] + - id: gen_ai.agent.tool.function + stability: experimental + type: string + brief: Description of the function of the tool utilized by the agent. + examples: ["Scrapes the web for data"] + - id: gen_ai.agent.tool.output_format + stability: experimental + type: string + brief: The format of the output generated by the tool. + examples: ["JSON", "CSV"] + - id: gen_ai.agent.tool.output + stability: experimental + type: string + brief: The format of the output generated by the tool. + examples: {"data": [...]} + - id: gen_ai.agent.task.name + stability: experimental + type: string + brief: The name of the task that the agent is performing. + examples: ["Data Collection"] + - id: gen_ai.agent.task.agent_name + stability: experimental + type: string + brief: The name of the agent that is performing the task. + examples: ["Web Scraper"] + - id: gen_ai.agent.task.description + stability: experimental + type: string + brief: A description of the task that the agent is performing. + examples: ["Collecting data from various sources"] + - id: gen_ai.agent.task.input + stability: experimental + type: string + brief: The input required for the task. + examples: ["Identify the next big trend for AI Agent"] + - id: gen_ai.agent.task.output + stability: experimental + type: string + brief: The output generated by the task. + examples: {"data": [...]} - id: gen_ai.operation.name stability: experimental type: @@ -157,6 +247,38 @@ groups: value: "embeddings" brief: 'Embeddings operation such as [OpenAI Create embeddings API](https://platform.openai.com/docs/api-reference/embeddings/create)' stability: experimental + - id: create_agent + value: "create_agent" + brief: 'Create GenAI agent' + stability: experimental + - id: create_task + value: "create_task" + brief: 'Create GenAI task' + stability: experimental + - id: create_tool + value: "create_tool" + brief: 'Create GenAI tool' + stability: experimental + - id: create_workflow + value: "create_workflow" + brief: 'Create GenAI Workflow' + stability: experimental + - id: run_workflow + value: "run_workflow" + brief: 'Run GenAI Workflow' + stability: experimental + - id: run_agent + value: "run_agent" + brief: 'Run GenAI Agent' + stability: experimental + - id: run_task + value: "run_task" + brief: 'Run GenAI Task' + stability: experimental + - id: run_tool + value: "run_tool" + brief: 'Run GenAI Tool' + stability: experimental brief: The name of the operation being performed. note: > If one of the predefined values applies, but specific system uses a different name it's RECOMMENDED to document it in the semantic diff --git a/model/gen-ai/spans.yaml b/model/gen-ai/spans.yaml index e6a3c743be..65934ab758 100644 --- a/model/gen-ai/spans.yaml +++ b/model/gen-ai/spans.yaml @@ -122,3 +122,135 @@ groups: - ref: server.port requirement_level: conditionally_required: If not default (443). + - id: trace.gen_ai.client.create_workflow + type: span + stability: experimental + brief: > + Describes GenAI external tool creation and is usually applicable when working + with remote external services. + note: | + The `gen_ai.operation.name` MUST be `create_tool`. + The **span name** SHOULD be `create_tool {gen_ai.agent.tool.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + extends: trace.gen_ai.client.create_task + attributes: + - ref: gen_ai.agent.tool.name + requirement_level: required + - id: trace.gen_ai.client.create_agent + type: span + stability: experimental + brief: > + Describes GenAI agent creation and is usually applicable when working + with remote agent services. + note: | + The `gen_ai.operation.name` MUST be `create_agent`. + The **span name** SHOULD be `create_agent {gen_ai.agent.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + extends: trace.gen_ai.client.common_attributes + attributes: + - ref: gen_ai.system + requirement_level: required + - ref: gen_ai.agent.id + requirement_level: + conditionally_required: if applicable. + - ref: gen_ai.agent.name + requirement_level: + conditionally_required: If provided by the application. + - ref: gen_ai.agent.description + requirement_level: + conditionally_required: If provided by the application. + - ref: gen_ai.request.model + requirement_level: + conditionally_required: If provided by the application. + note: > + The name of the GenAI model a request is being made to. If the model is supplied by a vendor, + then the value must be the exact name of the model requested. If the model is a fine-tuned + custom model, the value should have a more specific name than the base model that's been fine-tuned. + - ref: gen_ai.request.temperature + requirement_level: + conditionally_required: If provided by the application. + - ref: gen_ai.request.top_p + requirement_level: + conditionally_required: If provided by the application. + - id: trace.gen_ai.client.create_task + type: span + stability: experimental + brief: > + Describes GenAI task creation and is usually applicable when working + with remote task services. + note: | + The `gen_ai.operation.name` MUST be `create_task`. + The **span name** SHOULD be `create_task {gen_ai.agent.task.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + attributes: + - ref: gen_ai.agent.task.name + requirement_level: required + - id: trace.gen_ai.client.create_tool + type: span + stability: experimental + brief: > + Describes GenAI external tool creation and is usually applicable when working + with remote external services. + note: | + The `gen_ai.operation.name` MUST be `create_tool`. + The **span name** SHOULD be `create_tool {gen_ai.agent.tool.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + attributes: + - ref: gen_ai.agent.tool.name + requirement_level: required + - id: trace.gen_ai.client.run_tool + type: span + stability: experimental + brief: > + Describes GenAI external tool creation and is usually applicable when working + with remote external services. + note: | + The `gen_ai.operation.name` MUST be `create_tool`. + The **span name** SHOULD be `create_tool {gen_ai.agent.tool.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + extends: trace.gen_ai.client.create_tool + attributes: + - ref: gen_ai.agent.tool.name + requirement_level: required + - id: trace.gen_ai.client.run_task + type: span + stability: experimental + brief: > + Describes GenAI external tool creation and is usually applicable when working + with remote external services. + note: | + The `gen_ai.operation.name` MUST be `create_tool`. + The **span name** SHOULD be `create_tool {gen_ai.agent.tool.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + extends: trace.gen_ai.client.create_task + attributes: + - ref: gen_ai.agent.tool.name + requirement_level: required + - id: trace.gen_ai.client.create_workflow + type: span + stability: experimental + brief: > + Describes GenAI external tool creation and is usually applicable when working + with remote external services. + note: | + The `gen_ai.operation.name` MUST be `create_tool`. + The **span name** SHOULD be `create_tool {gen_ai.agent.tool.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + extends: trace.gen_ai.client.create_task + attributes: + - ref: gen_ai.agent.tool.name + requirement_level: required + - id: trace.gen_ai.client.run_workflow + type: span + stability: experimental + brief: > + Describes GenAI external tool creation and is usually applicable when working + with remote external services. + note: | + The `gen_ai.operation.name` MUST be `create_tool`. + The **span name** SHOULD be `create_tool {gen_ai.agent.tool.name}`. + Semantic conventions for individual GenAI systems and frameworks MAY specify different span name format. + extends: trace.gen_ai.client.create_task + attributes: + - ref: gen_ai.agent.tool.name + requirement_level: required