examples/report_generation/research_paper_report_generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0w1c_cWgyYto"
      },
      "source": [
        "# Research Paper Report Generating Agent.\n",
        "\n",
        "<a href=\"https://colab.research.google.com/github/run-llama/llamacloud-demo/blob/main/examples/report_generation/research_paper_report_generation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
        "\n",
        "Report generation is a frequent use case among our enterprise customers. We are developing a demo for report generation based on a practical use case that we commonly encounter.\n",
        "\n",
        "The pace at which AI papers are being published on arXiv is incredibly fast, and many people struggle to keep up with the latest updates. It would be valuable to create a report based on papers published by criteria such as date, author, company, or affiliation or specific papers in the outline from the user.\n",
        "\n",
        "Here’s a proposed workflow:\n",
        "1. Use the arXiv API to pull daily papers.\n",
        "2. Generate metadata such as publication date, update date, authors, research lab, etc.\n",
        "3. Index the data on LlamaCloud.\n",
        "4. Repeat steps 1-3 on a daily basis.\n",
        "5. Create an outline for the report. (Ideally from user)\n",
        "6. Develop a report-generating agent.\n",
        "7. Generate report based on the outline.\n",
        "\n",
        "**NOTE:** \n",
        "\n",
        "1. Please adjust the paper titles in the outline based on the date the notebook was run, as they may differ.\n",
        "2. For this iteration we did not use filters during retrieval or querying stage.\n",
        "\n",
        "![research_paper_report_generation](research_paper_report_generation.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qXNwRrA0BCE"
      },
      "source": [
        "### Installation\n",
        "\n",
        "We'll be utilizing various packages along with LlamaIndex:\n",
        "\n",
        "1. LlamaCloud - For creating a managed index in the cloud.\n",
        "2. LlamaParse - For effective document parsing.\n",
        "3. arxiv - For accessing the latest research papers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r6lfvAIl0Cmt",
        "outputId": "d26fd30b-c620-4be8-f314-3d7f52bb9b6e"
      },
      "outputs": [],
      "source": [
        "!pip install -U llama-index llama-index-indices-managed-llama-cloud llama-parse llama-cloud arxiv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9oCXpvCDvaDa"
      },
      "source": [
        "### SetupAPI Keys\n",
        "\n",
        "Set up the `LLAMA_CLOUD_API_KEY` and `OPENAI_API_KEY` for accessing the LlamaCloud managed index and OpenAI LLMs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "jG8v-K8Suxet"
      },
      "outputs": [],
      "source": [
        "import nest_asyncio\n",
        "\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "UfGEPRpOu5sz"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = 'sk-...' # Get your API key from https://platform.openai.com/account/api-keys\n",
        "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\" # Get your API key from https://cloud.llamaindex.ai/api-key"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WG7TgcTVvfZa"
      },
      "source": [
        "### Setup LLM\n",
        "\n",
        "For this demonstration, we'll use the `OpenAI` LLM, but you are free to experiment with any LLM of your choice for further exploration.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "Zpmho3Oyu61_"
      },
      "outputs": [],
      "source": [
        "from llama_index.llms.openai import OpenAI\n",
        "\n",
        "llm = OpenAI()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLiN7v9dvjt4"
      },
      "source": [
        "### Download `arxiv` papers based on topics.\n",
        "\n",
        "For this demonstration, we will download papers related to specific research topics of interest, focusing on `RAG and Agent.`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "LvIr4v7uu8Ac"
      },
      "outputs": [],
      "source": [
        "research_paper_topics = [\"RAG\", \"Agent\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "rLE8qF8Ou-UO"
      },
      "outputs": [],
      "source": [
        "import arxiv\n",
        "\n",
        "from pathlib import Path\n",
        "\n",
        "def download_papers(client, topics, num_results_per_topic):\n",
        "    \"\"\"Function to download papers from arxiv for given topics and number of results per topic\"\"\"\n",
        "    for topic in topics:\n",
        "\n",
        "        # sort by recent data and with max results\n",
        "        search = arxiv.Search(\n",
        "        query = topic,\n",
        "        max_results = num_results_per_topic,\n",
        "        sort_by = arxiv.SortCriterion.SubmittedDate\n",
        "        )\n",
        "\n",
        "        # get the results\n",
        "        results = client.results(search)\n",
        "\n",
        "        # download the pdf\n",
        "        for r in results:\n",
        "            r.download_pdf()\n",
        "\n",
        "def list_pdf_files(directory):\n",
        "    # List all .pdf files using pathlib\n",
        "    pdf_files = [file.name for file in Path(directory).glob('*.pdf')]\n",
        "    return pdf_files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will download three research papers for each topic."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "Bo5A8lbUu-z1"
      },
      "outputs": [],
      "source": [
        "# create a client\n",
        "client = arxiv.Client()\n",
        "\n",
        "download_papers(client, research_paper_topics, 3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AflAIQvavwng"
      },
      "source": [
        "### Parse the documents using `LlamaParse`\n",
        "\n",
        "We'll use `LlamaParse` to parse documents with `type=markdown` because LLMs excel at interpreting the text and tables found in PDFs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "61z_nk1cvBMt"
      },
      "outputs": [],
      "source": [
        "from llama_parse import LlamaParse\n",
        "\n",
        "def parse_files(pdf_files):\n",
        "    \"\"\"Function to parse the pdf files using LlamaParse in markdown format\"\"\"\n",
        "\n",
        "    parser = LlamaParse(\n",
        "        result_type=\"markdown\",  # \"markdown\" and \"text\" are available\n",
        "        num_workers=4,  # if multiple files passed, split in `num_workers` API calls\n",
        "        verbose=True,\n",
        "    )\n",
        "\n",
        "    documents = []\n",
        "\n",
        "    for index, pdf_file in enumerate(pdf_files):\n",
        "        print(f\"Processing file {index + 1}/{len(pdf_files)}: {pdf_file}\")\n",
        "        document = parser.load_data(pdf_file)\n",
        "        documents.append(document)\n",
        "\n",
        "    return documents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Parse the downloaded documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KUWgLD6OvCnz",
        "outputId": "6a817bfa-335e-4421-c065-65bdb01f1d4a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Processing file 1/6: 2410.13825v1.AgentOccam__A_Simple_Yet_Strong_Baseline_for_LLM_Based_Web_Agents.pdf\n",
            "Started parsing the file under job_id d53aa174-0758-4320-888d-29c3b332d639\n",
            "Processing file 2/6: 2410.13553v1.Integrating_Temporal_Representations_for_Dynamic_Memory_Retrieval_and_Management_in_Large_Language_Models.pdf\n",
            "Started parsing the file under job_id 81dd0a7a-3ad7-419c-b1d2-11d11b94dceb\n",
            "Processing file 3/6: 2410.13671v1.HEALTH_PARIKSHA__Assessing_RAG_Models_for_Health_Chatbots_in_Real_World_Multilingual_Settings.pdf\n",
            "Started parsing the file under job_id 0dee1b41-a6c7-4f97-80c7-fa23d740b546\n",
            "Processing file 4/6: 2410.13824v1.Harnessing_Webpage_UIs_for_Text_Rich_Visual_Understanding.pdf\n",
            "Started parsing the file under job_id e89139eb-26b2-4f07-89ec-d1c4ca44461f\n",
            "Processing file 5/6: 2410.13860v1.VLM_Grounder__A_VLM_Agent_for_Zero_Shot_3D_Visual_Grounding.pdf\n",
            "Started parsing the file under job_id c4815177-79a5-4476-b38d-ad7c15ba94e7\n",
            "Processing file 6/6: 2410.13716v1.MIRAGE_Bench__Automatic_Multilingual_Benchmark_Arena_for_Retrieval_Augmented_Generation_Systems.pdf\n",
            "Started parsing the file under job_id a73ec7fc-2e4e-49a2-b088-74a1614c518d\n"
          ]
        }
      ],
      "source": [
        "directory = './'\n",
        "pdf_files = list_pdf_files(directory)\n",
        "\n",
        "documents = parse_files(pdf_files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vl85DzTewNPx"
      },
      "source": [
        "### Utils\n",
        "\n",
        "Here, we define some utilities to help us extract metadata from each document, create a LlamaCloud pipeline/index, and upload the documents to the pipeline/index.\n",
        "\n",
        "1. `Metadata` - Pydantic model to extract metadata of author names, companies and general AI tags.\n",
        "2. `get_papers_metadata` - Extracts the metadata information from the research paper.\n",
        "3. `create_llamacloud_pipeline` - Create `LlamaCloud` pipeline.\n",
        "4. `upload_documents` - Upload the research papers to `LlamaCloud` index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "TWwLAHRvvFTO"
      },
      "outputs": [],
      "source": [
        "from llama_cloud.types import CloudDocumentCreate\n",
        "from pydantic import BaseModel, Field\n",
        "from typing import List\n",
        "from llama_cloud.client import LlamaCloud\n",
        "from llama_index.core.prompts import PromptTemplate\n",
        "from llama_index.core.async_utils import run_jobs\n",
        "\n",
        "class Metadata(BaseModel):\n",
        "    \"\"\"Output containing the authors names, authors companies, and general AI tags.\"\"\"\n",
        "\n",
        "    author_names: List[str] = Field(..., description=\"List of author names of the paper. Give empty list if not available\")\n",
        "\n",
        "    author_companies: List[str] = Field(..., description=\"List of author companies of the paper. Give empty list if not available\")\n",
        "\n",
        "    ai_tags: List[str] = Field(..., description=\"List of general AI tags related to the paper. Give empty list if not available\")\n",
        "\n",
        "def create_llamacloud_pipeline(pipeline_name, embedding_config, transform_config, data_sink_id=None):\n",
        "    \"\"\"Function to create a pipeline in llamacloud\"\"\"\n",
        "\n",
        "    client = LlamaCloud(token=os.environ[\"LLAMA_CLOUD_API_KEY\"])\n",
        "\n",
        "    pipeline = {\n",
        "        'name': pipeline_name,\n",
        "        'transform_config': transform_config,\n",
        "        'embedding_config': embedding_config,\n",
        "        'data_sink_id': data_sink_id\n",
        "    }\n",
        "\n",
        "    pipeline = client.pipelines.upsert_pipeline(request=pipeline)\n",
        "\n",
        "    return client, pipeline\n",
        "\n",
        "async def get_papers_metadata(text):\n",
        "    \"\"\"Function to get the metadata from the given paper\"\"\"\n",
        "    prompt_template = PromptTemplate(\"\"\"Generate authors names, authors companies, and general top 3 AI tags for the given research paper.\n",
        "\n",
        "    Research Paper:\n",
        "\n",
        "    {text}\"\"\")\n",
        "\n",
        "    metadata = await llm.astructured_predict(\n",
        "        Metadata,\n",
        "        prompt_template,\n",
        "        text=text,\n",
        "    )\n",
        "\n",
        "    return metadata\n",
        "\n",
        "async def get_document_upload(document, llm):\n",
        "    text_for_metadata_extraction = document[0].text + document[1].text + document[2].text\n",
        "    full_text = \"\\n\\n\".join([doc.text for doc in document])\n",
        "    metadata = await get_papers_metadata(text_for_metadata_extraction)\n",
        "    return CloudDocumentCreate(\n",
        "        text=full_text,\n",
        "        metadata={\n",
        "            'author_names': metadata.author_names,\n",
        "            'author_companies': metadata.author_companies,\n",
        "            'ai_tags': metadata.ai_tags\n",
        "        }\n",
        "     )\n",
        "                 \n",
        "async def upload_documents(client, documents):\n",
        "    \"\"\"Function to upload the documents to the cloud\"\"\"\n",
        "\n",
        "    # Upload the documents to the cloud\n",
        "    extract_jobs = []\n",
        "    for document in documents:\n",
        "        extract_jobs.append(get_document_upload(document, llm))\n",
        "    \n",
        "    document_upload_objs = await run_jobs(extract_jobs, workers=4)\n",
        "\n",
        "    _ = client.pipelines.create_batch_pipeline_documents(pipeline.id, request=document_upload_objs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1XHFEq0lxODG"
      },
      "source": [
        "### Create `LlamaCloud` pipeline.\n",
        "\n",
        "We will first create a `LlamaCloud` pipeline (empty index) before uploading documents. We need `embedding_config` and `transform_config` for the same.\n",
        "\n",
        "`embedding_config` - This config provides details about the embedding model and the corresponding API key used for creating embeddings during the indexing stage. Here we use OpenAI embeddings.\n",
        "\n",
        "`transform_config` - This config outlines the `chunk_size` and `chunk_overlap` parameters used during the indexing stage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "id": "wzBupKauvG_F",
        "outputId": "9ed1c4ef-6107-4391-808b-cd2cb55cd322"
      },
      "outputs": [],
      "source": [
        "# Embedding config\n",
        "embedding_config = {\n",
        "    'type': 'OPENAI_EMBEDDING',\n",
        "    'component': {\n",
        "        'api_key': os.environ[\"OPENAI_API_KEY\"], # editable\n",
        "        'model_name': 'text-embedding-ada-002' # editable\n",
        "    }\n",
        "}\n",
        "\n",
        "# Transformation auto config\n",
        "transform_config = {\n",
        "    'mode': 'auto',\n",
        "    'config': {\n",
        "        'chunk_size': 1024,\n",
        "        'chunk_overlap': 20\n",
        "    }\n",
        "}\n",
        "\n",
        "client, pipeline = create_llamacloud_pipeline('report_generation', embedding_config, transform_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bx1-xdTAxRNF"
      },
      "source": [
        "### Upload documents to `LlamaCloud` index.\n",
        "\n",
        "Now that we have set up a pipeline (an empty index), we will upload the downloaded documents using the specified `embedding_config` and `transform_config` configurations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tMux_8vXvIic",
        "outputId": "a9267f50-a297-47bd-981d-27c3fb3ca554"
      },
      "outputs": [],
      "source": [
        "await upload_documents(client, documents)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SJSRsTC9xVNu"
      },
      "source": [
        "### Create Index and QueryEngine\n",
        "\n",
        "Let's connect to the created LlamaCloud index and build a QueryEngine to use it for report generation.\n",
        "\n",
        "We will utilize hybrid search combined with re-ranking (cohere-reranker) for this purpose."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "9MtqMfOHvKQG"
      },
      "outputs": [],
      "source": [
        "from llama_index.indices.managed.llama_cloud import LlamaCloudIndex\n",
        "\n",
        "# connect to existing index\n",
        "index = LlamaCloudIndex(\n",
        "          name=\"report_generation\",\n",
        "          project_name=\"Default\",\n",
        "          api_key=os.environ['LLAMA_CLOUD_API_KEY'])\n",
        "\n",
        "query_engine = index.as_query_engine(\n",
        "  dense_similarity_top_k=10,\n",
        "  sparse_similarity_top_k=10,\n",
        "  alpha=0.5,\n",
        "  enable_reranking=True,\n",
        "  rerank_top_n = 5,\n",
        "  retrieval_mode=\"chunks\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPTTwZjfxYyh"
      },
      "source": [
        "### Utils to create queries for generating report based on outline.\n",
        "\n",
        "#### Sample Outline of the Report\n",
        "\n",
        "#### START OF OUTLINE OF THE REPORT\n",
        "\n",
        "```\n",
        "# Research Paper Report on RAG - Retrieval Augmented Generation and Agentic World.\n",
        "\n",
        "## 1. Introduction\n",
        "\n",
        "## 2. Retrieval Augmented Generation (RAG) and Agents\n",
        "2.1. Fundamentals of RAG and Agents.\n",
        "2.2. Current State and Applications\n",
        "\n",
        "## 3. Latest Papers:\n",
        "3.1. Paper-1 title (to be filled).\n",
        "3.2. Paper-2 title (to be filled).\n",
        "3.3. Paper-3 title (to be filled).\n",
        "\n",
        "## 4. Conclusion:\n",
        "```\n",
        "\n",
        "#### END OF OUTLINE OF THE REPORT\n",
        "\n",
        "Here is a sample outline for the report we intend to generate. We need to populate sections like   `Introduction, Retrieval Augmented Generation (RAG) and Agents`, and its sub-sections `Fundamentals of RAG and Agents`, `Current State and Applications`, and `Latest Papers`, as well as the final `Conclusion`. This can be done either by using an LLM or the LlamaCloud Index.\n",
        "\n",
        "To complete these sections, we'll need to query the index/LLM for relevant information. We will craft queries based on the sub-sections and sections within the context of the report's title. Here are some utilities to assist in this task.\n",
        "\n",
        "1. `extract_title`: Function to extract the title from the first line of the outline.\n",
        "2. `generate_query_with_llm`: Function to generate a query for a report using LLM.\n",
        "3. `classify_query`: Function to classify the query as either 'LLM' or 'INDEX' based on the query content.\n",
        "4. `parse_outline_and_generate_queries`: Function to parse the outline and generate queries for each section and subsection.\n",
        "\n",
        "**NOTE**: The utilities should be adjusted based on the specific outline of the report we are considering. This ensures that they align with the sections and sub-sections we need to populate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "L4ux_12JvN7l"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "\n",
        "def extract_title(outline):\n",
        "    \"\"\"Function to extract the title from the first line of the outline\"\"\"\n",
        "\n",
        "    first_line = outline.strip().split('\\n')[0]\n",
        "    return first_line.strip('# ').strip()\n",
        "\n",
        "def generate_query_with_llm(title, section, subsection):\n",
        "    \"\"\"Function to generate a query for a report using LLM\"\"\"\n",
        "\n",
        "    prompt = f\"Generate a research query for a report on {title}. \"\n",
        "    prompt += f\"The query should be for the subsection '{subsection}' under the main section '{section}'. \"\n",
        "    prompt += \"The query should guide the research to gather relevant information for this part of the report. The query should be clear, short and concise. \"\n",
        "\n",
        "    response = llm.complete(prompt)\n",
        "\n",
        "    return str(response).strip()\n",
        "\n",
        "def classify_query(query):\n",
        "    \"\"\"Function to classify the query as either 'LLM' or 'INDEX' based on the query content\"\"\"\n",
        "\n",
        "    prompt = f\"\"\"Classify the following query as either \"LLM\" if it can be answered directly by a large language model with general knowledge, or \"INDEX\" if it likely requires querying an external index or database for specific or up-to-date information.\n",
        "\n",
        "    Query: \"{query}\"\n",
        "\n",
        "    Consider the following:\n",
        "    1. If the query asks for general knowledge, concepts, or explanations, classify as \"LLM\".\n",
        "    2. If the query asks for specific facts, recent events, or detailed information that might not be in the LLM's training data, classify as \"INDEX\".\n",
        "    3. If unsure, err on the side of \"INDEX\".\n",
        "\n",
        "    Classification:\"\"\"\n",
        "\n",
        "    classification = str(llm.complete(prompt)).strip().upper()\n",
        "\n",
        "    if classification not in [\"LLM\", \"INDEX\"]:\n",
        "        classification = \"INDEX\"  # Default to INDEX if the response is unclear\n",
        "\n",
        "    return classification\n",
        "\n",
        "def parse_outline_and_generate_queries(outline):\n",
        "    \"\"\"Function to parse the outline and generate queries for each section and subsection\"\"\"\n",
        "    \n",
        "    lines = outline.strip().split('\\n')\n",
        "    title = extract_title(outline)\n",
        "    current_section = \"\"\n",
        "    queries = {}\n",
        "\n",
        "    for line in lines[1:]:  # Skip the title line\n",
        "        if line.startswith('## '):\n",
        "            current_section = line.strip('# ').strip()\n",
        "            queries[current_section] = {}\n",
        "        elif re.match(r'^\\d+\\.\\d+\\.', line):\n",
        "            subsection = line.strip()\n",
        "            query = generate_query_with_llm(title, current_section, subsection)\n",
        "            classification = classify_query(query)\n",
        "            queries[current_section][subsection] = {\"query\": query, \"classification\": classification}\n",
        "\n",
        "    # Handle sections without subsections\n",
        "    for section in queries:\n",
        "        if not queries[section]:\n",
        "            query = generate_query_with_llm(title, section, \"General overview\")\n",
        "            queries[section][\"General\"] = {\"query\": query, \"classification\": \"LLM\"}\n",
        "\n",
        "    return queries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQ48Oti9xfSH"
      },
      "source": [
        "### `ReportGenerationAgent`\n",
        "\n",
        "Here we create an agent to generate the final report based on the outline.\n",
        "\n",
        "Following are the steps following in generating the report.\n",
        "\n",
        "1. Generates queries to fill the report from the outline.\n",
        "2. Generates answers for the queries using LlamaCloud index.\n",
        "3. Fill back the answers to relevant parts in the report.\n",
        "4. Format the final report."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "ZSlT5arMvROX"
      },
      "outputs": [],
      "source": [
        "from typing import Any, List\n",
        "from llama_index.core.llms.function_calling import FunctionCallingLLM\n",
        "from llama_index.core.workflow import Workflow, StartEvent, StopEvent, Context, step\n",
        "from llama_index.core.workflow import Event\n",
        "\n",
        "class ReportGenerationEvent(Event):\n",
        "    pass\n",
        "\n",
        "\n",
        "class ReportGenerationAgent(Workflow):\n",
        "    \"\"\"Report generation agent.\"\"\"\n",
        "\n",
        "    def __init__(\n",
        "        self,\n",
        "        query_engine: Any,\n",
        "        llm: FunctionCallingLLM | None = None,\n",
        "        **kwargs: Any,\n",
        "    ) -> None:\n",
        "        super().__init__(**kwargs)\n",
        "        self.query_engine = query_engine\n",
        "        self.llm = llm or OpenAI(model='gpt-4o-mini')\n",
        "\n",
        "    def format_report(self, section_contents, outline):\n",
        "        \"\"\"Format the report based on the section contents.\"\"\"\n",
        "        report = \"\"\n",
        "\n",
        "        for section, subsections in section_contents.items():\n",
        "            section_match = re.match(r'^(\\d+\\.)\\s*(.*)$', section)\n",
        "            if section_match:\n",
        "                section_num, section_title = section_match.groups()\n",
        "                \n",
        "                if \"introduction\" in section.lower():\n",
        "                    introduction_num, introduction_title = section_num, section_title\n",
        "                elif \"conclusion\" in section.lower():\n",
        "                    conclusion_num, conclusion_title = section_num, section_title\n",
        "                else:\n",
        "                    combined_content = \"\\n\".join(subsections.values())\n",
        "                    summary_query = f\"Provide a short summary for section '{section}':\\n\\n{combined_content}\"\n",
        "                    section_summary = str(llm.complete(summary_query))\n",
        "                    report += f\"# {section_num} {section_title}\\n\\n{section_summary}\\n\\n\"\n",
        "\n",
        "                    report = self.get_subsections_content(subsections, report)\n",
        "\n",
        "        # Add introduction\n",
        "\n",
        "        introduction_query = f\"Create an introduction for the report:\\n\\n{report}\"\n",
        "        introduction = str(self.llm.complete(introduction_query))\n",
        "        report = f\"# {introduction_num} {introduction_title}\\n\\n{introduction}\\n\\n\" + report\n",
        "\n",
        "        # Add conclusion\n",
        "\n",
        "        conclusion_query = f\"Create a conclusion for the report:\\n\\n{report}\"\n",
        "        conclusion = str(self.llm.complete(conclusion_query))\n",
        "        report += f\"# {conclusion_num} {conclusion_title}\\n\\n{conclusion}\"\n",
        "\n",
        "        # Add title\n",
        "        title = extract_title(outline)\n",
        "        report = f\"# {title}\\n\\n{report}\"\n",
        "        return report\n",
        "\n",
        "    def get_subsections_content(self, subsections, report):\n",
        "        \"\"\"Generate content for each subsection in the outline.\"\"\"\n",
        "        # Sort subsections by their keys before adding them to the report\n",
        "        for subsection in sorted(subsections.keys(), key=lambda x: re.search(r'(\\d+\\.\\d+)', x).group(1) if re.search(r'(\\d+\\.\\d+)', x) else x):\n",
        "            content = subsections[subsection]\n",
        "            subsection_match = re.search(r'(\\d+\\.\\d+)\\.\\s*(.+)', subsection)\n",
        "            if subsection_match:\n",
        "                subsection_num, subsection_title = subsection_match.groups()\n",
        "                report += f\"## {subsection_num} {subsection_title}\\n\\n{content}\\n\\n\"\n",
        "            else:\n",
        "                report += f\"## {subsection}\\n\\n{content}\\n\\n\"\n",
        "        return report\n",
        "\n",
        "    def generate_section_content(self, queries, reverse=False):\n",
        "        \"\"\"Generate content for each section and subsection in the outline.\"\"\"\n",
        "        section_contents = {}\n",
        "        for section, subsections in queries.items():\n",
        "            section_contents[section] = {}\n",
        "            subsection_keys = reversed(sorted(subsections.keys())) if reverse else sorted(subsections.keys())\n",
        "            for subsection in subsection_keys:\n",
        "                data = subsections[subsection]\n",
        "                query = data['query']\n",
        "                classification = data['classification']\n",
        "                if classification == \"LLM\":\n",
        "                    answer = str(llm.complete(query + \" Give a short answer.\"))\n",
        "                else:\n",
        "                    answer = str(query_engine.query(query))\n",
        "                section_contents[section][subsection] = answer\n",
        "        return section_contents\n",
        "\n",
        "    @step(pass_context=True)\n",
        "    async def queries_generation_event(self, ctx: Context, ev: StartEvent) -> ReportGenerationEvent:\n",
        "        \"\"\"Generate queries for the report.\"\"\"\n",
        "        ctx.data[\"outline\"] = ev.outline\n",
        "        queries = parse_outline_and_generate_queries(ctx.data[\"outline\"])\n",
        "\n",
        "        return ReportGenerationEvent(queries=queries)\n",
        "\n",
        "    @step(pass_context=True)\n",
        "    async def generate_report(\n",
        "        self, ctx: Context, ev: ReportGenerationEvent\n",
        "    ) -> StopEvent:\n",
        "        \"\"\"Generate report.\"\"\"\n",
        "\n",
        "        queries = ev.queries\n",
        "\n",
        "        # Generate contents for sections in reverse order\n",
        "        section_contents = self.generate_section_content(queries, reverse=True)\n",
        "        # Format and compile the final report\n",
        "        report = self.format_report(section_contents, ctx.data[\"outline\"])\n",
        "       \n",
        "        return StopEvent(result={\"response\": report})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NhOsBSyGyRo6"
      },
      "source": [
        "### Outline of the report.\n",
        "\n",
        "Here's the outline of the report.\n",
        "\n",
        "Please update the paper titles in the 'Latest Papers' section according to the user's interests."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "-ep8LrWCvUNz"
      },
      "outputs": [],
      "source": [
        "outline = \"\"\"\n",
        "# Research Paper Report on RAG - Retrieval Augmented Generation and Agentic World.\n",
        "\n",
        "## 1. Introduction\n",
        "\n",
        "## 2. Retrieval Augmented Generation (RAG) and Agents\n",
        "2.1. Fundamentals of RAG and Agents.\n",
        "2.2. Current State and Applications\n",
        "\n",
        "## 3. Latest Papers:\n",
        "3.1. HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings\n",
        "3.2. MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems\n",
        "3.3. VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding\n",
        "\n",
        "## 4. Conclusion:\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "asup1BBSyVoo"
      },
      "source": [
        "### Generate report\n",
        "\n",
        "Now that everything is set up, we will create an agent to generate the report."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "KmRkoaIFvUrt"
      },
      "outputs": [],
      "source": [
        "agent = ReportGenerationAgent(\n",
        "    query_engine=query_engine,\n",
        "    llm=llm,\n",
        "    verbose=True,\n",
        "    timeout=1200.0,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IVa0FDMsvW7l",
        "outputId": "2b4b249f-2fd0-46e3-fff7-5add54fc43cd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Running step queries_generation_event\n",
            "Step queries_generation_event produced event ReportGenerationEvent\n",
            "Running step generate_report\n",
            "Step generate_report produced event StopEvent\n"
          ]
        }
      ],
      "source": [
        "report = await agent.run(outline=outline)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dChMzQ3JvYV3",
        "outputId": "e390b3ad-d074-4950-eb5e-8345a77bc42b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "# Research Paper Report on RAG - Retrieval Augmented Generation and Agentic World.\n",
            "\n",
            "# 1. Introduction\n",
            "\n",
            "In this report, we delve into the advancements in Retrieval-Augmented Generation (RAG) and Agents technologies, focusing on their impact on memory recall and management in AI systems. These technologies aim to enhance response generation in dialogue agents by combining retrieval and generation-based methods, improving memory recall accuracy, and context-awareness in interactions. We also explore the latest papers in the field, including zero-shot 3D visual grounding, multilingual benchmarking for RAG systems, and the assessment of RAG models for health chatbots in real-world multilingual settings. These studies shed light on the significant performance variations, challenges, and advancements in RAG and Agents technologies, showcasing their potential in revolutionizing dialogue agents' cognitive abilities and interaction capabilities.\n",
            "\n",
            "# 2. Retrieval Augmented Generation (RAG) and Agents\n",
            "\n",
            "Overall, RAG and Agents technologies focus on improving memory recall and management in AI systems, enabling more realistic and context-aware interactions in dialogue agents. These advancements represent significant progress in the field of artificial intelligence and natural language processing, offering sophisticated solutions for enhancing dialogue agents' understanding of user interactions and their ability to engage in meaningful conversations.\n",
            "\n",
            "## 2.1 Fundamentals of RAG and Agents.\n",
            "\n",
            "The key fundamentals of Retrieval-Augmented Generation (RAG) involve combining retrieval and generation-based methods to enhance response generation in dialogue agents. RAG systems aim to improve memory recall and management by integrating temporal representations into memory vectors, allowing for more nuanced handling of temporal and contextual dynamics. These systems dynamically update memory connections using techniques like dynamic time warping and employ a propagation control mechanism to ensure efficient recall by limiting stimuli spread. In an agentic world, RAG systems interact by autonomously recalling relevant memories for response generation, managing memory significance in a temporal context similar to human memory recall, and enhancing the cognitive abilities of dialogue agents within context-aware dialogue AI systems.\n",
            "\n",
            "## 2.2 Current State and Applications\n",
            "\n",
            "RAG, or Retrieval-Augmented Generation, is a method that combines retrieval and generation-based techniques to enhance response generation in dialogue agents. It aims to improve memory recall and management in AI systems by integrating temporal representations into memory vectors. This approach allows for a more nuanced handling of temporal and contextual dynamics, leading to improved memory retrieval accuracy and context-awareness in interactions. On the other hand, Agentic World technology focuses on developing generative agents that can simulate credible human behavior by storing, organizing, and retrieving experiences to enable realistic behaviors in interactive environments. These technologies represent advancements in the field of artificial intelligence and natural language processing, offering sophisticated solutions for enhancing dialogue agents' understanding of user interactions and their ability to engage in meaningful conversations.\n",
            "\n",
            "# 3. Latest Papers:\n",
            "\n",
            "The latest papers discussed in this section focus on advancements in zero-shot 3D visual grounding, multilingual benchmarking for retrieval-augmented generation systems, and the assessment of RAG models for health chatbots in real-world multilingual settings. The VLM-Grounder agent demonstrates superior performance in zero-shot 3D visual grounding tasks without relying on 3D geometry or object priors. MIRAGE-Bench provides a comprehensive evaluation platform for multilingual RAG systems, enabling model comparison and training of learning to rank models. The study on RAG models for health chatbots reveals performance variations, lower factual correctness for non-English queries, and challenges with code-mixed and culturally relevant queries.\n",
            "\n",
            "## 3.1 HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings\n",
            "\n",
            "The key findings in assessing RAG models for health chatbots in real-world multilingual settings, particularly in the Health-PARIKSHA project, include significant performance variations among models, lower factual correctness for responses to non-English queries compared to English queries, and the observation that instruction-tuned Indic models do not consistently perform well on Indic language queries. Additionally, the study highlighted challenges such as code-mixed and culturally relevant queries in the dataset that posed difficulties for the evaluated models.\n",
            "\n",
            "## 3.2 MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems\n",
            "\n",
            "MIRAGE-Bench serves as an automatic multilingual benchmark arena for retrieval-augmented generation systems in the context of RAG. It incorporates heuristic-based evaluation with an arena-based leaderboard using a trainable learning to rank model as a surrogate judge. The key features and functionalities include the ability to evaluate RAG extensively across 18 languages on Wikipedia, utilizing a standardized benchmark constructed from the MIRACL retrieval dataset. MIRAGE-Bench assesses multilingual generation by benchmarking diverse multilingual-focused LLMs, providing a synthetic arena-based leaderboard for model comparison. Additionally, it enables the training of a learning to rank model for estimating the performance of newer models without the need for an expensive LLM judge, offering better interpretability and retrainability with different heuristic features.\n",
            "\n",
            "## 3.3 VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding\n",
            "\n",
            "The VLM-Grounder agent in the context of zero-shot 3D visual grounding dynamically stitches image sequences, utilizes a grounding and feedback scheme to locate the target object, and employs a multi-view ensemble projection to accurately estimate 3D bounding boxes. It operates solely based on 2D images, surpassing previous zero-shot methods in performance on benchmarks like ScanRefer and Nr3D without relying on 3D geometry or object priors. The agent excels in associating language with visual information, providing a novel framework for zero-shot 3D visual grounding tasks.\n",
            "\n",
            "# 4. Conclusion:\n",
            "\n",
            "# 4. Conclusion\n",
            "\n",
            "In conclusion, the advancements in Retrieval-Augmented Generation (RAG) and Agents technologies represent significant progress in enhancing memory recall and management in AI systems, particularly in dialogue agents. These technologies combine retrieval and generation-based methods to improve response generation, memory recall accuracy, and context-awareness in interactions. The latest papers discussed in this report highlight the diverse applications of RAG and Agents technologies, including zero-shot 3D visual grounding, multilingual benchmarking for RAG systems, and the assessment of RAG models for health chatbots in real-world multilingual settings. These studies showcase the potential of RAG and Agents technologies in revolutionizing dialogue agents' cognitive abilities and interaction capabilities, paving the way for more sophisticated and context-aware AI systems in the future.\n"
          ]
        }
      ],
      "source": [
        "print(report['response'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Save the final report"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [],
      "source": [
        "with open(\"report.md\", \"w\") as f:\n",
        "    f.write(report['response'])"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "llamacloud",
      "language": "python",
      "name": "llamacloud"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}