Skip to content

Commit

Permalink
More ragas data points
Browse files Browse the repository at this point in the history
  • Loading branch information
@heiko-braun
heiko-braun committed Jan 8, 2024
1 parent 922896c commit 0846a49
Show file tree
Hide file tree
Showing 5 changed files with 199 additions and 63 deletions.
24 changes: 12 additions & 12 deletions core/agent.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def configure_retriever(collection_name):

retriever = qdrant.as_retriever(
search_type="mmr",
search_kwargs={"fetch_k":15, "k": 5, "lambda_mult":0.75}
search_kwargs={"fetch_k":15, "k": 5, "lambda_mult":0.8}
)

return retriever
Expand All @@ -54,25 +54,25 @@ def create_lookup_tool(retriever, name, description):
"Useful when you need to answer questions about tools for developing Camel applications. Input should be a list of tools for developing Camel applications.",
)

spring_reference = create_lookup_tool(
configure_retriever("spring_reference"),
"search_spring_reference",
"Useful when you need to answer questions about Camel Components used with Spring Boot. Input should be a list of Camel components to integrate third-party systems.",
)
# spring_reference = create_lookup_tool(
# configure_retriever("spring_reference"),
# "search_spring_reference",
# "Useful when you need to answer questions about Camel Components used with Spring Boot. Input should be a list of Camel components to integrate third-party systems.",
# )

spring_started_tool = create_lookup_tool(
configure_retriever("spring_get_started_2"),
"search_spring_getting_started",
"Useful when you need to answer questions about leveraging Spring Boot with Camel. Input should be a a list of terms related to Spring Boot project setup.",
)
# spring_started_tool = create_lookup_tool(
# configure_retriever("spring_get_started_2"),
# "search_spring_getting_started",
# "Useful when you need to answer questions about leveraging Spring Boot with Camel. Input should be a a list of terms related to Spring Boot project setup.",
# )

quarkus_started_tool = create_lookup_tool(
configure_retriever("quarkus_getting_started_2"),
"search_quarkus_getting_started",
"Useful when you need to answer questions about leveraging Quarkus with Camel. Input should be a list of terms related to Camel Quarkus project setup.",
)

tools = [CamelCoreTool(), spring_started_tool, spring_reference, tooling_guide, QuarkusReferenceTool(), quarkus_started_tool]
tools = [CamelCoreTool(), tooling_guide, QuarkusReferenceTool(), quarkus_started_tool]

# LLM instructions
agent_llm = ChatOpenAI(temperature=0, streaming=True, model="gpt-3.5-turbo-1106")
Expand Down
179 changes: 128 additions & 51 deletions eval.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Questions: 3\n",
"Grounded Truths: 3\n"
"Questions: 6\n",
"Grounded Truths: 6\n"
]
}
],
Expand Down Expand Up @@ -93,11 +93,50 @@
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"starting tool ... global error handler RouteBuilder\n",
"Embedding ms: 0.34453701972961426\n",
"0.802 : _manual_error-handler.html.txt_1\n",
"0.786 : _next_lpr-component.html.txt_3\n",
"0.799 : _eips_message-history.html.txt_2\n",
"0.8 : _manual_exception-clause.html.txt_12\n",
"0.801 : _manual_error-handler.html.txt_2\n",
"ending tool ... \n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"Input: 1\n",
"Outputs: 1\n",
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"starting tool ... jBang\n",
"ending tool ... \n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"Input: 1\n",
"Outputs: 1\n",
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"starting tool ... Rest DSL\n",
"Embedding ms: 0.3432190418243408\n",
"0.825 : _manual_rest-dsl.html.txt_2\n",
"0.819 : _manual_rest-dsl.html.txt_1\n",
"0.811 : _manual_rest-dsl.html.txt_0\n",
"0.81 : _manual_rest-dsl.html.txt_7\n",
"0.805 : _next_rest-component.html.txt_0\n",
"ending tool ... \n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"Input: 1\n",
"Outputs: 1\n",
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"starting tool ... file idempotent\n",
"Embedding ms: 0.32719993591308594\n",
"Embedding ms: 0.6420547962188721\n",
"0.812 : _next_file-component.html.txt_26\n",
"0.805 : _next_file-component.html.txt_25\n",
"0.794 : _next_azure-files-component.html.txt_11\n",
Expand All @@ -112,7 +151,7 @@
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"starting tool ... event aggregation in Apache Camel\n",
"Embedding ms: 0.37224912643432617\n",
"Embedding ms: 0.3488612174987793\n",
"0.831 : _faq_how-do-the-direct-event-seda-and-vm-endpoints-compare.html.txt\n",
"0.829 : _others_cloudevents.html.txt\n",
"0.82 : _eips_aggregate-eip.html.txt_10\n",
Expand All @@ -126,22 +165,28 @@
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"starting tool ... file splitting\n",
"Embedding ms: 0.4099709987640381\n",
"0.783 : _next_hdfs-component.html.txt_7\n",
"0.781 : _eips_split-eip.html.txt_0\n",
"0.772 : _next_sftp-component.html.txt_7\n",
"0.777 : _next_file-component.html.txt_24\n",
"0.768 : _dataformats_univocityFixed-dataformat.html.txt_0\n",
"starting tool ... file reading\n",
"Embedding ms: 0.35344600677490234\n",
"0.777 : _next_file-component.html.txt_22\n",
"0.769 : _dataformats_univocityFixed-dataformat.html.txt_0\n",
"0.739 : _support_.txt\n",
"0.773 : _next_sftp-component.html.txt_13\n",
"0.771 : _next_ftps-component.html.txt_2\n",
"ending tool ... \n",
"starting tool ... AWS S3\n",
"Embedding ms: 0.3646571636199951\n",
"0.86 : _next_aws2-s3-component.html.txt_11\n",
"0.849 : _next_aws2-s3-component.html.txt_1\n",
"0.842 : _next_aws2-s3-component.html.txt_14\n",
"0.84 : _next_aws2-s3-component.html.txt_3\n",
"0.831 : _next_aws-summary.html.txt\n",
"ending tool ... \n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"Input: 2\n",
"Outputs: 2\n",
"Answer len: 3\n",
"Context len: 3\n"
"Answer len: 6\n",
"Context len: 6\n"
]
}
],
Expand Down Expand Up @@ -171,7 +216,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"outputs": [
{
Expand All @@ -185,7 +230,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00, 6.08s/it]\n"
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00, 6.74s/it]\n"
]
},
{
Expand All @@ -199,7 +244,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:56<00:00, 56.05s/it]\n"
"100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [03:43<00:00, 223.19s/it]\n"
]
},
{
Expand All @@ -213,7 +258,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:25<00:00, 25.14s/it]\n"
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:58<00:00, 58.98s/it]\n"
]
},
{
Expand All @@ -227,7 +272,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00, 3.58s/it]\n"
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:07<00:00, 7.72s/it]\n"
]
},
{
Expand Down Expand Up @@ -264,69 +309,101 @@
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>how you can I register a global error handler ...</td>\n",
" <td>To register a global error handler for the Rou...</td>\n",
" <td>[page_content='\"errorHandler(noErrorHandler())...</td>\n",
" <td>[\\nTo register a global error handler for the ...</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.250</td>\n",
" <td>0.940842</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>How can I test a Camel route using jBang?\\n</td>\n",
" <td>To test a Camel route using jBang, you can use...</td>\n",
" <td>[[Document(page_content='\"CHAPTER 4. USING CAM...</td>\n",
" <td>[\\nTo test a Camel route using jBang, you can ...</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.000</td>\n",
" <td>0.989555</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Can you show an example how to use the Rest DS...</td>\n",
" <td>To use the Rest DSL in Java to build an endpoi...</td>\n",
" <td>[page_content='\"Inline Rest DSL as a single ro...</td>\n",
" <td>[\\nTo create a REST endpoint that responds to ...</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.700</td>\n",
" <td>0.813892</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>This is my route\\n\\n from(\"sftp://userName:...</td>\n",
" <td>To avoid processing the same files in case of ...</td>\n",
" <td>[page_content='\"Using a file based idempotent ...</td>\n",
" <td>[\\nTo avoid processing the same files in case ...</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.666667</td>\n",
" <td>0.770217</td>\n",
" <td>0.875</td>\n",
" <td>0.763041</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <th>4</th>\n",
" <td>I am having an issue while working on the requ...</td>\n",
" <td>To listen to the events generated by one route...</td>\n",
" <td>To listen to events generated by one route and...</td>\n",
" <td>[page_content='\"Article source: https://rhaeto...</td>\n",
" <td>[\\nTo listen to the events of one route by ano...</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.400000</td>\n",
" <td>0.795870</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>I want to read a local file, create equally si...</td>\n",
" <td>To achieve the desired functionality of readin...</td>\n",
" <td>[page_content='\"String\\n\\n\\n\\nCamelFileNameCon...</td>\n",
" <td>[\\nTo achieve this, you can use Apache Camel t...</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>0.200000</td>\n",
" <td>0.792767</td>\n",
" <td>1.000</td>\n",
" <td>0.791901</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" question \\\n",
"0 This is my route\\n\\n from(\"sftp://userName:... \n",
"1 I am having an issue while working on the requ... \n",
"2 I want to read a local file, create equally si... \n",
"0 how you can I register a global error handler ... \n",
"1 How can I test a Camel route using jBang?\\n \n",
"2 Can you show an example how to use the Rest DS... \n",
"3 This is my route\\n\\n from(\"sftp://userName:... \n",
"4 I am having an issue while working on the requ... \n",
"\n",
" answer \\\n",
"0 To avoid processing the same files in case of ... \n",
"1 To listen to the events generated by one route... \n",
"2 To achieve the desired functionality of readin... \n",
"0 To register a global error handler for the Rou... \n",
"1 To test a Camel route using jBang, you can use... \n",
"2 To use the Rest DSL in Java to build an endpoi... \n",
"3 To avoid processing the same files in case of ... \n",
"4 To listen to events generated by one route and... \n",
"\n",
" contexts \\\n",
"0 [page_content='\"Using a file based idempotent ... \n",
"1 [page_content='\"Article source: https://rhaeto... \n",
"2 [page_content='\"String\\n\\n\\n\\nCamelFileNameCon... \n",
"0 [page_content='\"errorHandler(noErrorHandler())... \n",
"1 [[Document(page_content='\"CHAPTER 4. USING CAM... \n",
"2 [page_content='\"Inline Rest DSL as a single ro... \n",
"3 [page_content='\"Using a file based idempotent ... \n",
"4 [page_content='\"Article source: https://rhaeto... \n",
"\n",
" ground_truths context_precision \\\n",
"0 [\\nTo avoid processing the same files in case ... 0.0 \n",
"1 [\\nTo listen to the events of one route by ano... 1.0 \n",
"2 [\\nTo achieve this, you can use Apache Camel t... 1.0 \n",
"0 [\\nTo register a global error handler for the ... 1.0 \n",
"1 [\\nTo test a Camel route using jBang, you can ... 0.0 \n",
"2 [\\nTo create a REST endpoint that responds to ... 0.0 \n",
"3 [\\nTo avoid processing the same files in case ... 0.0 \n",
"4 [\\nTo listen to the events of one route by ano... 1.0 \n",
"\n",
" context_recall faithfulness answer_relevancy \n",
"0 1.0 0.666667 0.770217 \n",
"1 1.0 0.400000 0.795870 \n",
"2 NaN 0.200000 0.792767 "
"0 1.0 0.250 0.940842 \n",
"1 1.0 1.000 0.989555 \n",
"2 1.0 0.700 0.813892 \n",
"3 1.0 0.875 0.763041 \n",
"4 1.0 1.000 0.791901 "
]
},
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
Expand Down
15 changes: 15 additions & 0 deletions ragas_data/4.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
How can I test a Camel route using jBang?
__###__
To test a Camel route using jBang, you can use the "camel run" command provided by Camel JBang. This command allows you to run Camel routes directly from the command line. Here's an example of how you can use jBang to test a Camel route:

jbang camel@apache/camel run MyCamelRoute.java

In this example, "MyCamelRoute.java" is the file containing your Camel route. You can replace it with the actual file name of your Camel route.

Additionally, you can also run Camel routes directly from the OS clipboard using jBang. This allows you to copy the route code and quickly run it. For example, you can copy the route code to your clipboard and then run the route using the following command:

jbang run clipboard.java

Replace "clipboard.java" with the appropriate file extension based on the content of the clipboard (e.g., java, xml, or yaml).

These commands demonstrate how you can use jBang to test Camel routes directly from the command line or the OS clipboard.
18 changes: 18 additions & 0 deletions ragas_data/5.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
how you can I register a global error handler for the RouteBuilder?
__###__
To register a global error handler for the RouteBuilder in Apache Camel, you can use the errorHandler method in the configure method of the RouteBuilder. Here's an example of how to do this in Java DSL:

RouteBuilder builder = new RouteBuilder() {
public void configure() {
errorHandler(deadLetterChannel("seda:error"));

// Define your routes here
}
};

In this example, the deadLetterChannel method is used to specify the error handler for the RouteBuilder. This sets up a global error handler that applies to all routes defined within the RouteBuilder.

Additionally, you can also configure route-specific error handlers by using the errorHandler method within a specific route definition.

If you are using Spring-based configuration, you can configure the error handler as a Spring bean and scope it globally, per route, or per policy using the errorHandlerRef attribute.

Loading

0 comments on commit 0846a49

Please sign in to comment.