OpenSearch - Vector Search - RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "082aa845",
   "metadata": {},
   "source": [
    "# Build Multimodal RAG with Amazon OpenSearch Service"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55c7b731",
   "metadata": {},
   "source": [
    "In this notebook, you will build and run different types search method using a sample retail dataset. You will start with a text only search, followed by a multi-modal semantic search with both text and image, and finally you will used conversational search, all from Amazon OpenSearch Service."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "029f69f6",
   "metadata": {},
   "source": [
    "_Note: The retail dataset used contains 2,465 retail product samples that belong to different categories such as accessories, home decor, apparel, housewares, books, and instruments. Each product contains metadata including the ID, current stock, name, category, style, description, price, image URL, and gender affinity of the product. You will be only use the product image and product description fields in this solution._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "853ae065",
   "metadata": {},
   "source": [
    "Here are the key steps in this notebook:\n",
    "\n",
    "1. Setup OpenSearch connection and create connection to ML models.\n",
    "2. Ingest data into OpenSearch domain.\n",
    "3. Experiment with different search options like lexical (text only), multimodal (semantic), and conversational search.\n",
    "4. (Bonus / Optional) Deploy a web application to experiment with a shopping assistant chat bot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df42f90b-5e78-41d4-8bea-63fa3718298d",
   "metadata": {},
   "source": [
    "![title](diagrams/notebook-diagram-main.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8be245f",
   "metadata": {},
   "source": [
    "## 1. Lab Pre-requisites\n",
    "\n",
    "\n",
    "For this notebook we require a few libraries. We'll use the Python clients for Amazon OpenSearch Service and Amazon Bedrock, and OpenSearch ML Client library for generating multimodal embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2df92a75",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 1.1. Import libraries & initialize resources\n",
    "The code blocks below will import all the relevant libraries and modules used in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd35fc57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install opensearch-py -q\n",
    "!pip install opensearch_py_ml -q\n",
    "!pip install deprecated -q\n",
    "!pip install requests_aws4auth -q\n",
    "print(\"Installs completed.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9d38375",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import Python libraries\n",
    "import boto3\n",
    "import json\n",
    "from opensearchpy import OpenSearch, RequestsHttpConnection\n",
    "import os\n",
    "import urllib.request\n",
    "import tarfile\n",
    "from requests_aws4auth import AWS4Auth\n",
    "from ruamel.yaml import YAML\n",
    "from PIL import Image\n",
    "import base64\n",
    "import re\n",
    "\n",
    "# Initiale variables for later use\n",
    "embedding_connector_id = \"\"\n",
    "embedding_model_id = \"\"\n",
    "llm_connector_id = \"\"\n",
    "llm_model_id = \"\"\n",
    "\n",
    "print(\"Imports and initialization completed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9240112b",
   "metadata": {},
   "source": [
    "### 1.2. Get CloudFormation stack output variables\n",
    "\n",
    "We have preconfigured a few resources by creating a CloudFormation stack in the account. Names and ARN of these resources will be used within this lab. Load the output variables here to be used in later parts of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2d4f51e",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a Boto3 session\n",
    "session = boto3.Session()\n",
    "\n",
    "# Get the account id\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "\n",
    "# Get the current region\n",
    "region = session.region_name\n",
    "\n",
    "cfn = boto3.client('cloudformation')\n",
    "\n",
    "# Method to obtain output variables from Cloudformation stack. \n",
    "def get_cfn_outputs(stackname):\n",
    "    outputs = {}\n",
    "    for output in cfn.describe_stacks(StackName=stackname)['Stacks'][0]['Outputs']:\n",
    "        outputs[output['OutputKey']] = output['OutputValue']\n",
    "    return outputs\n",
    "\n",
    "## Setup variables to use for the rest of the demo\n",
    "cloudformation_stack_name = \"multimodal-rag-opensearch\"\n",
    "\n",
    "outputs = get_cfn_outputs(cloudformation_stack_name)\n",
    "aos_host = outputs['OpenSearchDomainEndpoint']\n",
    "# s3_bucket = outputs['s3BucketTraining']\n",
    "bedrock_inf_iam_role = outputs['BedrockBatchInferenceRole']\n",
    "bedrock_inf_iam_role_arn = outputs['BedrockBatchInferenceRoleArn']\n",
    "# sagemaker_notebook_url = outputs['SageMakerNotebookURL']\n",
    "notebook_iam_role_arn = outputs['NotebookRoleArn']\n",
    "\n",
    "# We will just print all the variables so you can easily copy if needed.\n",
    "outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9955879-7c47-44b8-a792-af54879b4199",
   "metadata": {},
   "source": [
    "### 1.3. Retrieve internal OpenSearch credentials (for this step only)\n",
    "\n",
    "Create a connection to OpenSearch using username and password."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95518e3a-6bfa-43e0-914b-dabc62a21fa9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Connect to OpenSearch using the internal username and password obtained from AWS Secrets Manager\n",
    "kms = boto3.client('secretsmanager')\n",
    "aos_credentials = json.loads(kms.get_secret_value(SecretId=outputs['OpenSearchSecret'])['SecretString'])\n",
    "auth = (\n",
    "\n",
    "aos_credentials['username'], aos_credentials['password'])\n",
    "\n",
    "# Create OpenSearch client\n",
    "aos_client = OpenSearch(\n",
    "    hosts=[f'https://{aos_host}'],\n",
    "    http_auth=auth,\n",
    "    use_ssl=True,\n",
    "    verify_certs=True,\n",
    "    connection_class=RequestsHttpConnection\n",
    ")\n",
    "print(f\"Connected to OpenSearch endpoint :{aos_client}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaae8614-cf6b-4fcb-8f11-6a5b6571803e",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 1.4 Map this Notebook's IAM role to OpenSearch backend role\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e948634",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the role mapping to grant permissions to  AOS username and notebook_iam_role_arn\n",
    "role_name = \"all_access\"\n",
    "\n",
    "role_mapping = {\n",
    "    \"backend_roles\": [notebook_iam_role_arn],\n",
    "    \"users\" : [ aos_credentials['username'] ]\n",
    "}\n",
    "\n",
    "# Create the role mapping\n",
    "response = aos_client.security.create_role_mapping(role=role_name, body=role_mapping)\n",
    "print(\"Role mapping created:\", response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ec1eae6",
   "metadata": {},
   "source": [
    "### 1.5 Download the retail dataset to our Notebook instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aca8ce36",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "os.makedirs('tmp/images', exist_ok = True)\n",
    "metadata_file = urllib.request.urlretrieve('https://ws-assets-prod-iad-r-pdx-f3b3f9f1a7d6a3d0.s3.us-west-2.amazonaws.com/cfda120e-98ef-49df-b875-6f0f33e99615/sample_data/products-data.yml', 'tmp/images/products.yaml')\n",
    "img_filename,headers= urllib.request.urlretrieve('https://ws-assets-prod-iad-r-pdx-f3b3f9f1a7d6a3d0.s3.us-west-2.amazonaws.com/cfda120e-98ef-49df-b875-6f0f33e99615/sample_data/images.tar.gz', 'tmp/images/images.tar.gz')              \n",
    "print(img_filename)\n",
    "file = tarfile.open('tmp/images/images.tar.gz')\n",
    "file.extractall('tmp/images/')\n",
    "file.close()\n",
    "#remove images.tar.gz\n",
    "os.remove('tmp/images/images.tar.gz')\n",
    "print(\"Data download and extraction completed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f468131",
   "metadata": {},
   "source": [
    "## 2. Create, register, and deploy the OpenSearch Service ML connector to the Amazon Titan Multimodal Embeddings G1 model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f216c6ba-6904-4e34-89a1-74f0a748df0b",
   "metadata": {},
   "source": [
    "![title](diagrams/notebook-diagram-ml-connector.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80f24f4a",
   "metadata": {},
   "source": [
    "#### NOTE: AUTHENTICATION CELL\n",
    "\n",
    "At any point in this lab, if you get a failure message - The security token included in the request is expired. You can resolve it by running this cell again. The cell refreshes the security credentials that is required for the rest of the lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a053595",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Connect to OpenSearch using the IAM Role of this Jupyter notebook\n",
    "# Create AWS4Auth instance\n",
    "credentials = boto3.Session().get_credentials()\n",
    "awsauth = AWS4Auth(\n",
    "    credentials.access_key,\n",
    "    credentials.secret_key,\n",
    "    region,\n",
    "    'es',\n",
    "    session_token=credentials.token\n",
    ")\n",
    "\n",
    "# Create OpenSearch client\n",
    "aos_client = OpenSearch(\n",
    "    hosts=[f'https://{aos_host}'],\n",
    "    http_auth=awsauth,\n",
    "    use_ssl=True,\n",
    "    verify_certs=True,\n",
    "    connection_class=RequestsHttpConnection,\n",
    "    timeout=60\n",
    ")\n",
    "print(\"Connection details: \")\n",
    "aos_client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5791932c-44e7-4c23-bee2-cc915d6de2e6",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.1 Create the connector to Amazon Bedrock Multimodal embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a113cb5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Create an OpenSearch remote model connector with Amazon Bedrock Titan MM Embedding model.\n",
    "\n",
    "if not embedding_connector_id:\n",
    "    payload = {\n",
    "        \"name\": \"Amazon Bedrock Connector: embedding\",\n",
    "        \"description\": \"The connector to bedrock Titan multimodal embedding model\",\n",
    "        \"version\": 1,\n",
    "        \"protocol\": \"aws_sigv4\",\n",
    "        \"credential\": {\n",
    "          \"roleArn\": f\"arn:aws:iam::{account_id}:role/{bedrock_inf_iam_role}\"\n",
    "           },\n",
    "        \"parameters\": {\n",
    "        \"region\": region,\n",
    "        \"service_name\": \"bedrock\",\n",
    "        \"model\": \"amazon.titan-embed-image-v1\"\n",
    "            },\n",
    "        \"actions\": [\n",
    "            {\n",
    "          \"action_type\": \"predict\",\n",
    "          \"method\": \"POST\",\n",
    "          \"url\": \"https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke\",\n",
    "          \"headers\": {\n",
    "            \"content-type\": \"application/json\",\n",
    "            \"x-amz-content-sha256\": \"required\"\n",
    "              },\n",
    "            \"request_body\": \"{ \\\"inputText\\\": \\\"${parameters.inputText:-null}\\\", \\\"inputImage\\\": \\\"${parameters.inputImage:-null}\\\" }\",\n",
    "          \"pre_process_function\": \"\\n    StringBuilder parametersBuilder = new StringBuilder(\\\"{\\\");\\n    if (params.text_docs.length > 0 && params.text_docs[0] != null) {\\n      parametersBuilder.append(\\\"\\\\\\\"inputText\\\\\\\":\\\");\\n      parametersBuilder.append(\\\"\\\\\\\"\\\");\\n      parametersBuilder.append(params.text_docs[0]);\\n      parametersBuilder.append(\\\"\\\\\\\"\\\");\\n      \\n      if (params.text_docs.length > 1 && params.text_docs[1] != null) {\\n        parametersBuilder.append(\\\",\\\");\\n      }\\n    }\\n    \\n    \\n    if (params.text_docs.length > 1 && params.text_docs[1] != null) {\\n      parametersBuilder.append(\\\"\\\\\\\"inputImage\\\\\\\":\\\");\\n      parametersBuilder.append(\\\"\\\\\\\"\\\");\\n      parametersBuilder.append(params.text_docs[1]);\\n      parametersBuilder.append(\\\"\\\\\\\"\\\");\\n    }\\n    parametersBuilder.append(\\\"}\\\");\\n    \\n    return  \\\"{\\\" +\\\"\\\\\\\"parameters\\\\\\\":\\\" + parametersBuilder + \\\"}\\\";\",\n",
    "          \"post_process_function\": \"\\n      def name = \\\"sentence_embedding\\\";\\n      def dataType = \\\"FLOAT32\\\";\\n      if (params.embedding == null || params.embedding.length == 0) {\\n          return null;\\n      }\\n      def shape = [params.embedding.length];\\n      def json = \\\"{\\\" +\\n                 \\\"\\\\\\\"name\\\\\\\":\\\\\\\"\\\" + name + \\\"\\\\\\\",\\\" +\\n                 \\\"\\\\\\\"data_type\\\\\\\":\\\\\\\"\\\" + dataType + \\\"\\\\\\\",\\\" +\\n                 \\\"\\\\\\\"shape\\\\\\\":\\\" + shape + \\\",\\\" +\\n                 \\\"\\\\\\\"data\\\\\\\":\\\" + params.embedding +\\n                 \\\"}\\\";\\n      return json;\\n    \"\n",
    "                }\n",
    "          ] \n",
    "        }\n",
    "\n",
    "    response = aos_client.transport.perform_request(\n",
    "        'POST',\n",
    "        '/_plugins/_ml/connectors/_create',\n",
    "        body=json.dumps(payload),\n",
    "        headers={\"Content-Type\": \"application/json\"}\n",
    "    )\n",
    "    \n",
    "    embedding_connector_id = response['connector_id']\n",
    "else:\n",
    "    print(f\"Connector already exists - {embedding_connector_id}\")\n",
    "    \n",
    "print(\"Embedding connector ID: \" + embedding_connector_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06eed3ef",
   "metadata": {},
   "source": [
    "### 2.2 Register the model\n",
    "Once the connector is created, register and deploy the model using the following 2 cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b6a074b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Register the multimodal embedding model\n",
    "if not embedding_model_id:\n",
    "    # Prepare the payload\n",
    "    payload = {\n",
    "        \"name\": \"Bedrock Titan mm embeddings model\",\n",
    "        \"function_name\": \"remote\",\n",
    "        \"description\": \"Bedrock Titan mm embeddings model\",\n",
    "        \"connector_id\": embedding_connector_id\n",
    "    }\n",
    "    # Make the request\n",
    "    response = aos_client.transport.perform_request(\n",
    "        'POST',\n",
    "        '/_plugins/_ml/models/_register',\n",
    "        body=json.dumps(payload),\n",
    "        headers={\"Content-Type\": \"application/json\"}\n",
    "    )\n",
    "    embedding_model_id = response['model_id']\n",
    "else:\n",
    "    print(\"skipping model registration - model already exists\")\n",
    "print(\"Model registered under model_id: \"+embedding_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "705a6220-8724-45bc-aaca-c8c9e7befeb1",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.3 Deploy the multimodal embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "845c62c4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Deploy the embedding model\n",
    "response = aos_client.transport.perform_request(\n",
    "    'POST',\n",
    "    '/_plugins/_ml/models/'+embedding_model_id+'/_deploy',\n",
    "    headers={\"Content-Type\": \"application/json\"}\n",
    ")\n",
    "print(\"Deployment status of the model, \"+embedding_model_id+\" : \"+response['status'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "852500c8",
   "metadata": {},
   "source": [
    "### 2.4 Test the integration between the OpenSearch domain and Amazon Bedrock multimodal embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6153286",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "img = \"tmp/images/footwear/2d2d8ec8-4806-42a7-b8ba-ceb15c1c7e84.jpg\"\n",
    "with open(img, \"rb\") as image_file:\n",
    "    input_image_binary = base64.b64encode(image_file.read()).decode(\"utf8\")\n",
    "    \n",
    "payload = {\n",
    "\"parameters\": {\n",
    "\"inputText\": \"Sleek, stylish black sneakers made for urban exploration. With fashionable looks and comfortable design, these sneakers keep your feet looking great while you walk the city streets in style\",\n",
    "\"inputImage\":input_image_binary\n",
    "}\n",
    "}\n",
    "\n",
    "response = aos_client.transport.perform_request(\n",
    "    'POST',\n",
    "    '/_plugins/_ml/models/'+embedding_model_id+'/_predict',\n",
    "    body=json.dumps(payload),\n",
    "    headers={\"Content-Type\": \"application/json\"}\n",
    ")\n",
    "\n",
    "try:\n",
    "    embed = response['inference_results'][0]['output'][0]['data'][0:10]\n",
    "    shape = response['inference_results'][0]['output'][0]['shape'][0]\n",
    "    print(\"Embedding test completed.\")\n",
    "    print(\"First 10 dimensions:\")\n",
    "    print(str(embed))\n",
    "    print(\"\\n\")\n",
    "    print(\"Total: \" + str(shape) + \" dimensions\")\n",
    "except KeyError as e:\n",
    "    print(f\"KeyError: {e}\")\n",
    "    print(\"The response does not contain the expected data structure.\")\n",
    "except Exception as e:\n",
    "    print(f\"Error: {e}\")\n",
    "    print(\"An unexpected error occurred.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "234e0cf1-5540-4477-a823-55d512ae6224",
   "metadata": {},
   "source": [
    "## 🎉👏🎈Congrats! You have succesfully connected to OpenSearch and created an integration to multimodal embedding model!! 🎉👏🎈"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05cd7e48",
   "metadata": {},
   "source": [
    "## 3. Create the OpenSearch ingestion pipeline and index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71342102-f371-49d7-b164-008dd15efaa8",
   "metadata": {},
   "source": [
    "![title](diagrams/notebook-diagram-index-data.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40d48551",
   "metadata": {},
   "source": [
    "\n",
    "Create an ingestion pipeline that will call Amazon Bedrock Titan Multimodal embedding model and convert the text and image into multimodal vector embeddings. Ingest pipeline is a feature in OpenSearch that allows you to define certain actions to be automatically be performed at the time of data ingestion. You could do simple processing such as adding a static field, modify an existing field, or call a remote model to get inference and store inference output together with the indexed record/document. In our case inference output is a vector embedding.\n",
    "\n",
    "This ingestion pipeline is going to call our remote model and convert the `product_description` field and the `image_binary` fields to vector embedding and store it in the field called `vector_embedding`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34a8f3b1-a79e-457f-889c-d815f5b5b7fd",
   "metadata": {},
   "source": [
    "### 3.1 Create the ingestion pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5951fc54",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_id = \"bedrock-multimodal-ingest-pipeline\"\n",
    "payload = {\n",
    "    \"description\": \"A text/image embedding pipeline\",\n",
    "    \"processors\": [\n",
    "        {\n",
    "            \"text_image_embedding\": {\n",
    "                \"model_id\": embedding_model_id,\n",
    "                \"embedding\": \"vector_embedding\",\n",
    "                \"field_map\": {\n",
    "                    \"text\": \"product_description\",\n",
    "                    \"image\": \"image_binary\"\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "response = aos_client.ingest.put_pipeline(id=pipeline_id, body=payload)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b607f46-e6ed-4a78-bc2d-daea2add3c66",
   "metadata": {},
   "source": [
    "### 3.2 Create the k-NN index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f85afd1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Check if the index exists. Delete and recreate if it does. \n",
    "if aos_client.indices.exists(index='bedrock-multimodal-rag'):\n",
    "    print(\"The index exists. Deleting...\")\n",
    "    response = aos_client.indices.delete(index='bedrock-multimodal-rag')\n",
    "    \n",
    "payload = {\n",
    "  \"settings\": {\n",
    "    \"index.knn\": True,\n",
    "    \"default_pipeline\": \"bedrock-multimodal-ingest-pipeline\"\n",
    "  },\n",
    "  \"mappings\": {\n",
    "      \n",
    "    \"_source\": {\n",
    "     \n",
    "    },\n",
    "    \"properties\": {\n",
    "      \"vector_embedding\": {\n",
    "        \"type\": \"knn_vector\",\n",
    "        \"dimension\": shape,\n",
    "        \"method\": {\n",
    "          \"name\": \"hnsw\",\n",
    "          \"engine\": \"faiss\",\n",
    "          \"parameters\": {}\n",
    "        }\n",
    "      },\n",
    "      \"product_description\": {\n",
    "        \"type\": \"text\"\n",
    "      },\n",
    "        \"image_url\": {\n",
    "        \"type\": \"text\"\n",
    "      },\n",
    "      \"image_binary\": {\n",
    "        \"type\": \"binary\"\n",
    "      },\n",
    "      \"price\": {\n",
    "        \"type\": \"float\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "print(\"Creating index...\")\n",
    "response = aos_client.indices.create(index='bedrock-multimodal-rag',body=payload)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f42df6cf",
   "metadata": {},
   "source": [
    "### 3.3 Index the dataset to the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ceea00f0",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "def resize_image(photo, width, height):\n",
    "    Image.MAX_IMAGE_PIXELS = 100000000\n",
    "    \n",
    "    with Image.open(photo) as image:\n",
    "        image.verify()\n",
    "    with Image.open(photo) as image:    \n",
    "        \n",
    "        if image.format in [\"JPEG\", \"PNG\"]:\n",
    "            file_type = image.format.lower()\n",
    "            path = image.filename.rsplit(\".\", 1)[0]\n",
    "\n",
    "            image.thumbnail((width, height))\n",
    "            image.save(f\"{path}-resized.{file_type}\")\n",
    "    return file_type, path\n",
    "\n",
    "# Load the products from the dataset\n",
    "yaml = YAML()\n",
    "items_ = yaml.load(open('tmp/images/products.yaml'))\n",
    "\n",
    "batch = 0\n",
    "count = 0\n",
    "body_ = ''\n",
    "batch_size = 100\n",
    "last_batch = int(len(items_)/batch_size)\n",
    "action = json.dumps({ 'index': { '_index': 'bedrock-multimodal-rag' } })\n",
    "\n",
    "for item in items_:\n",
    "    count+=1\n",
    "    fileshort = \"tmp/images/\"+item[\"category\"]+\"/\"+item[\"image\"]\n",
    "    payload = {}\n",
    "    payload['image_url'] = fileshort\n",
    "    payload['product_description'] = item['description']\n",
    "    payload['price'] = item['price']\n",
    "    \n",
    "    #resize the image and generate image binary\n",
    "    file_type, path = resize_image(fileshort, 2048, 2048)\n",
    "\n",
    "    with open(fileshort.split(\".\")[0]+\"-resized.\"+file_type, \"rb\") as image_file:\n",
    "        input_image = base64.b64encode(image_file.read()).decode(\"utf8\")\n",
    "    \n",
    "    os.remove(fileshort.split(\".\")[0]+\"-resized.\"+file_type)\n",
    "    payload['image_binary'] = input_image\n",
    "    \n",
    "    body_ = body_ + action + \"\\n\" + json.dumps(payload) + \"\\n\"\n",
    "    \n",
    "    if(count == batch_size): # When count reaches batch size, send to bulk for indexing.\n",
    "        response = aos_client.bulk(\n",
    "        index = \"bedrock-multimodal-rag\",\n",
    "        body = body_\n",
    "        )\n",
    "        batch += 1\n",
    "        count = 0\n",
    "        print(\"batch \"+str(batch) + \" ingestion done!\")\n",
    "        if(batch != last_batch):\n",
    "            body_ = \"\"\n",
    "        \n",
    "            \n",
    "#ingest the remaining rows\n",
    "response = aos_client.bulk(\n",
    "        index = \"bedrock-multimodal-rag\",\n",
    "        body = body_\n",
    "        )\n",
    "        \n",
    "print(\"All \"+str(last_batch)+\" batches ingested into index\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8858ee12-86dd-4633-ba9e-b8ec3e680216",
   "metadata": {},
   "source": [
    "### 3.4 Check indexing by running an OpenSearch query: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b278e344",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "res = aos_client.search(index=\"bedrock-multimodal-rag\", body={\"query\": {\"match_all\": {}}})\n",
    "print(\"Records found: %d.\" % res['hits']['total']['value'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e664a12d-e9fc-438b-972f-26e7d4814732",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 🎉👏🎈Congrats! You have succesfully ingested sample data into to OpenSearch using an ingest pipeline!!! 🎉👏🎈"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0b08140",
   "metadata": {},
   "source": [
    "## 4. Lexical and vector search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b110fe2f-5d77-4a84-b247-e4b17360cf63",
   "metadata": {},
   "source": [
    "![title](diagrams/notebook-diagram-vector-query.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95425e04-b1a9-49b2-ae73-695f8a9ed24c",
   "metadata": {},
   "source": [
    "### 4.1 Lexical search\n",
    "Try using different keywords and phrases to see different results.\n",
    "Replace the `query` string with your search and then run the code block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e556860",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lexical search: Using query text only\n",
    "query = \"shoes\"\n",
    "index_name = \"bedrock-multimodal-rag\"\n",
    "search_body = {\n",
    "    \"_source\": {\n",
    "        \"exclude\": [\"vector_embedding\"]\n",
    "    },\n",
    "    \"query\": {\n",
    "        \"match\": {\n",
    "            \"product_description\": {\n",
    "                \"query\": query\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"size\": 5\n",
    "}\n",
    "\n",
    "# Perform the search\n",
    "response = aos_client.search(\n",
    "    body=search_body,\n",
    "    index=index_name\n",
    ")\n",
    "\n",
    "#Output the results\n",
    "count = 1\n",
    "for hit in response['hits']['hits']:\n",
    "    print(str(count) + \". \" + hit[\"_source\"][\"product_description\"])\n",
    "    print(\"Price: \"+ str(hit[\"_source\"][\"price\"]))\n",
    "    image = Image.open(hit[\"_source\"][\"image_url\"])\n",
    "    new_size = (300, 200)\n",
    "    resized_img = image.resize(new_size)\n",
    "    resized_img.show()\n",
    "    count+=1\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6a1cb55",
   "metadata": {},
   "source": [
    "### 4.2 Vector search with both image and text as inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ceb9828",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Multimodal Search using text and image as inputs\n",
    "query_text = \"travel\"\n",
    "query_image = \"./archive/simple_bag.jpg\"\n",
    "\n",
    "img = Image.open(query_image) \n",
    "print(\"Input text query: \"+query_text)\n",
    "print(\"Input query Image:\")\n",
    "img.show()\n",
    "\n",
    "# Define the query and search body\n",
    "with open(query_image, \"rb\") as image_file:\n",
    "    query_image_binary = base64.b64encode(image_file.read()).decode(\"utf8\")\n",
    "search_body = {\n",
    "    \"_source\": {\n",
    "        \"exclude\": [\n",
    "            \"vector_embedding\"\n",
    "        ]\n",
    "        },\n",
    "        \"query\": {    \n",
    "            \"neural\": {\n",
    "                \"vector_embedding\": {           \n",
    "                    \"query_image\":query_image_binary,\n",
    "                    \"query_text\":query_text,     \n",
    "                    \"model_id\": embedding_model_id,\n",
    "                    \"k\": 5\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        \"size\":5\n",
    "      }\n",
    "\n",
    "# Perform the search\n",
    "response = aos_client.search(\n",
    "    body=search_body,\n",
    "    index=index_name\n",
    ")\n",
    "\n",
    "#Output the results\n",
    "print('Search results:')\n",
    "count = 1\n",
    "for hit in response['hits']['hits']:\n",
    "    print(str(count) + \". \" + hit[\"_source\"][\"product_description\"])\n",
    "    print(\"Price: \"+ str(hit[\"_source\"][\"price\"]))\n",
    "    image = Image.open(hit[\"_source\"][\"image_url\"])\n",
    "    new_size = (300, 200)\n",
    "    resized_img = image.resize(new_size)\n",
    "    resized_img.show()\n",
    "    count+=1\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77feb61e",
   "metadata": {},
   "source": [
    "## 5. Multimodal conversational search\n",
    "\n",
    "Conversational search lets you ask questions in natural language, receive a text response based on a provided context, and ask additional  questions with context.\n",
    "\n",
    "In this section, you will be using OpenSearch Service as the knowledge database to run multimodal retrieval and augment the LLM prompt with the relevant context. You will be using Claude v2 as the foundational model to generate responses and provide fashion advice to end users based on the available retail items stored in OpenSearch Service.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe3b251-e1ee-49b7-83ef-5e258c85b0fb",
   "metadata": {},
   "source": [
    "![title](diagrams/notebook-diagram-conversational-search.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68d1a87f",
   "metadata": {},
   "source": [
    "### 5.1 Create an OpenSearch Bedrock Claude LLM connector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9daf7603",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not llm_connector_id:\n",
    "    connector_payload = {\n",
    "        \"name\": \"Amazon Bedrock Connector: Claude 2\",\n",
    "        \"description\": \"The connector to bedrock Claude V2\",\n",
    "        \"version\": 1,\n",
    "        \"protocol\": \"aws_sigv4\",\n",
    "        \"credential\": {\n",
    "            \"roleArn\": f\"arn:aws:iam::{account_id}:role/{bedrock_inf_iam_role}\"\n",
    "        },\n",
    "        \"parameters\": {\n",
    "            \"region\": region,\n",
    "            \"service_name\": \"bedrock\",\n",
    "            \"auth\": \"Sig_V4\",\n",
    "            \"model\": \"anthropic.claude-v2\"\n",
    "        },\n",
    "        \"actions\": [\n",
    "            {\n",
    "                \"action_type\": \"predict\",\n",
    "                \"method\": \"POST\",\n",
    "                \"headers\": {\n",
    "                    \"content-type\": \"application/json\"\n",
    "                },\n",
    "                \"url\": \"https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke\",\n",
    "                \"request_body\": \"{\\\"prompt\\\":\\\"\\\\n\\\\nHuman: ${parameters.inputs}\\\\n\\\\nAssistant:\\\",\\\"max_tokens_to_sample\\\":300,\\\"temperature\\\":0.5,\\\"top_k\\\":250,\\\"top_p\\\":1,\\\"stop_sequences\\\":[\\\"\\\\\\\\n\\\\\\\\nHuman:\\\"]}\"\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "    \n",
    "    # Create the connector\n",
    "    response = aos_client.transport.perform_request(\n",
    "        'POST',\n",
    "        '/_plugins/_ml/connectors/_create',\n",
    "        body=connector_payload\n",
    "    )\n",
    "    # Print the response\n",
    "    llm_connector_id = response['connector_id']\n",
    "else:\n",
    "    print(f\"Connector already exists\")\n",
    "    \n",
    "print(\"LLM connector ID: \" + llm_connector_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef6123b4",
   "metadata": {},
   "source": [
    "### 5.2 Register and deploy the Claude LLM connector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d92645d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Register and deploy the llm model\n",
    "if not llm_model_id:\n",
    "    payload = { \n",
    "        \"name\": \"Amazon Bedrock Connector: Claude v2\",\n",
    "        \"function_name\": \"remote\",\n",
    "        \"description\": \"The connector to bedrock Claude v2\",\n",
    "        \"connector_id\": llm_connector_id\n",
    "    }\n",
    "    # Make the request\n",
    "    response = aos_client.transport.perform_request(\n",
    "        'POST',\n",
    "        '/_plugins/_ml/models/_register',\n",
    "        body=json.dumps(payload),\n",
    "        headers={\"Content-Type\": \"application/json\"}\n",
    "    )\n",
    "    llm_model_id = response['model_id']\n",
    "# Display the response    \n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb824b38",
   "metadata": {},
   "source": [
    "### 5.3 Test the LLM connector by running an inference \n",
    "We will run the test inference without searching the index by passing an input directly to the LLM connector we just deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fb698c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prepare the query string\n",
    "payload = {\n",
    "    \"parameters\": {\n",
    "        \"inputs\": \"What are the most important features of a travel bag? Be concise.\"\n",
    "    }\n",
    "}\n",
    "# Make the request\n",
    "response = aos_client.transport.perform_request(\n",
    "    'POST',\n",
    "    '/_plugins/_ml/models/'+llm_model_id+'/_predict',\n",
    "    body=payload,\n",
    "    headers={\"Content-Type\": \"application/json\"}\n",
    ")\n",
    "# Check the response \n",
    "if response['inference_results'][0]['status_code'] == 200:\n",
    "    llm_gen = response['inference_results'][0]['output'][0]['dataAsMap']['completion']\n",
    "    print(str(\"Claude generated response without context: \\n\\n\"+llm_gen))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1bc1b7-880f-4f99-9744-d206e9bd26ec",
   "metadata": {},
   "source": [
    "### 5.4 Enable the memory and rag pipeline features of OpenSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723877d0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set plugin settings using the cluster.put_settings method of the OpenSearch client\n",
    "response = aos_client.cluster.put_settings(\n",
    "    body={\n",
    "        \"persistent\": {\n",
    "            \"plugins.ml_commons.memory_feature_enabled\": \"true\",\n",
    "            \"plugins.ml_commons.rag_pipeline_feature_enabled\": \"true\"\n",
    "        }\n",
    "    }\n",
    ")\n",
    "# Print response. Look for 'acknowledged': True\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c6075c",
   "metadata": {},
   "source": [
    "### 5.5 Create a search pipeline (shopping assistant)\n",
    "In order to have a conversational search, the LLM needs to remember the context of the entire conversation to have follow-up questions. This is composed of two components:\n",
    "- Conversational memory to provide the history context\n",
    "- Retrieval-augmented generation (RAG) pipeline to provide the search context\n",
    "\n",
    "The retrieval_augmented_generation processor, part of the RAG pipeline, is a search results processor that intercepts query results, retrieves previous messages from the conversation, and sends a prompt to a large language model (LLM), saving the response in conversational memory and returning both the original OpenSearch query results and the LLM response.\n",
    "\n",
    "Let's create the search pipeline with the retrieval_augmented_generation processor to use at search time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8e39298",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prepare the response processor attributes\n",
    "payload = {\n",
    "    \"response_processors\": [\n",
    "    {\n",
    "      \"retrieval_augmented_generation\": {\n",
    "        \"tag\": \"bedrock_rag-pipeline_demo\",\n",
    "        \"description\": \"Search pipeline using Bedrock Claude v2 Connector for RAG\",\n",
    "        \"model_id\": llm_model_id,\n",
    "        \"context_field_list\": [\"product_description\"],\n",
    "        \"system_prompt\": \"You are a helpful shopping advisor that uses their vast knowledge of fashion tips to make great recommendations people will enjoy.\",\n",
    "        \"user_instructions\": \"As a shopping advisor, be friendly and approachable. Greet the customer warmly. Evaluate each item provided in the context and provide a concise recommendation about each item to matches best the customer question using the order and number of search result related to each item. If there are items in the provided context that do not match the user question, explain that this may be due to insufficient items in the inventory. Finally, thank the client and let them know you're available if they have any other questions.\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "# Make the request\n",
    "response = aos_client.transport.perform_request(\n",
    "    'PUT',\n",
    "    \"/_search/pipeline/multimodal_rag_pipeline\",\n",
    "    body=payload,\n",
    "    headers={\"Content-Type\": \"application/json\"}\n",
    ")\n",
    "# Print response. Look for 'acknowledged': True\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5debf27c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5.6 Create a conversational memory object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b041545",
   "metadata": {},
   "source": [
    "#### NOTE: Re-run the below cell to create a new memory ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2df16508",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prepare the query string\n",
    "payload = {\n",
    "    \n",
    "    \"name\": \"Conversation about products\"\n",
    "}\n",
    "# Make the request\n",
    "response = aos_client.transport.perform_request(\n",
    "    'POST',\n",
    "    \"/_plugins/_ml/memory/\",\n",
    "    body=payload,\n",
    "    headers={\"Content-Type\": \"application/json\"}\n",
    ")\n",
    "# Persist memory_id\n",
    "memory_id = response['memory_id']\n",
    "# Print the 'memory_id'\n",
    "print(\"The new memory id is: \" +memory_id) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "701d062d-2d10-42c4-940e-b99d3ec8f6ec",
   "metadata": {},
   "source": [
    "### 5.7 Start a conversational search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "246e3737",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# RAG using multimoadal search to provide prompt context\n",
    "# Text and image as inputs\n",
    "query_text = \"for a long trip.\"\n",
    "query_image = \"./archive/simple_bag.jpg\"\n",
    "\n",
    "img = Image.open(query_image) \n",
    "print(\"Input text query: \"+query_text)\n",
    "print(\"Input query Image:\")\n",
    "img.show()\n",
    "\n",
    "# Define the query and search body\n",
    "with open(query_image, \"rb\") as image_file:\n",
    "    query_image_binary = base64.b64encode(image_file.read()).decode(\"utf8\")\n",
    "\n",
    "response = aos_client.search(\n",
    "    index='bedrock-multimodal-rag',\n",
    "    body={\n",
    "        \"_source\": {\n",
    "            \"exclude\": [\"vector_embedding\", \"image_binary\"]\n",
    "        },\n",
    "        \"query\": {\n",
    "            \"neural\": {\n",
    "                \"vector_embedding\": {\n",
    "                    \"query_image\": query_image_binary,\n",
    "                    \"query_text\": query_text,\n",
    "                    \"model_id\": embedding_model_id,\n",
    "                    \"k\": 5\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        \"size\": 5,\n",
    "        \"ext\": {\n",
    "            \"generative_qa_parameters\": {\n",
    "                \"llm_model\": \"bedrock/claude\",\n",
    "                \"llm_question\": query_text,\n",
    "                \"memory_id\": memory_id,\n",
    "                \"context_size\": 5,\n",
    "                \"message_size\": 5,\n",
    "                \"timeout\": 60\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    params={\n",
    "        \"search_pipeline\": \"multimodal_rag_pipeline\"\n",
    "    },\n",
    "    request_timeout=30\n",
    ")\n",
    "# Extract the generated 'shopping assistant' recommendations\n",
    "# Split the string into lines\n",
    "# lines = response['ext']['retrieval_augmented_generation']['answer'].split('\\n')\n",
    "recommendations = response['ext']['retrieval_augmented_generation']['answer']\n",
    "# for line in lines:\n",
    "#     if re.match(r'[^\\s\\0]+', line):\n",
    "#         recommendations.append(line.strip())\n",
    "    \n",
    "# Output the search results and shopping assistnat recommendations together\n",
    "print('Search results and Shopping assistant recommendations:')\n",
    "count = 1\n",
    "for hit in response['hits']['hits']:\n",
    "    print(\"Search result \"+str(count) + \": \")\n",
    "    print(\"Price: \"+ str(hit[\"_source\"][\"price\"]))\n",
    "    print(hit[\"_source\"][\"product_description\"])\n",
    "    # print(\"Shopping assistant: \")\n",
    "    # print(recommendations[count-1])\n",
    "    image = Image.open(hit[\"_source\"][\"image_url\"])\n",
    "    new_size = (300, 200)\n",
    "    resized_img = image.resize(new_size)\n",
    "    resized_img.show()\n",
    "    count+=1\n",
    "    print('')\n",
    "print(\"Shopping assistant: \")\n",
    "print(recommendations)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4c975b9",
   "metadata": {},
   "source": [
    "### 5.8 Ask a follow up question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28236e75",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query_text = \"thanks. What is likely to be best for air travel?\"\n",
    "\n",
    "print(\"Input text query: \" + query_text)\n",
    "\n",
    "# Perform the search (no new image this time)\n",
    "response = aos_client.search(\n",
    "    index='bedrock-multimodal-rag',\n",
    "    body={\n",
    "        \"_source\": {\n",
    "            \"exclude\": [\"vector_embedding\", \"image_binary\"]\n",
    "        },\n",
    "        \"query\": {\n",
    "            \"neural\": {\n",
    "                \"vector_embedding\": {\n",
    "                    #\"query_image\": query_image_binary, (leaving out the image for now)\n",
    "                    \"query_text\": query_text,\n",
    "                    \"model_id\": embedding_model_id,\n",
    "                    \"k\": 5\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        \"size\": 5,\n",
    "        \"ext\": {\n",
    "            \"generative_qa_parameters\": {\n",
    "                \"llm_model\": \"bedrock/claude\",\n",
    "                \"llm_question\": query_text,\n",
    "                \"memory_id\": memory_id,\n",
    "                \"context_size\": 5,\n",
    "                \"message_size\": 5,\n",
    "                \"timeout\": 60\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    params={\n",
    "        \"search_pipeline\": \"multimodal_rag_pipeline\"\n",
    "    },\n",
    "    request_timeout=30\n",
    ")\n",
    "# Extract the recommendations from the response\n",
    "# Split the string into lines\n",
    "# lines = response['ext']['retrieval_augmented_generation']['answer'].split('\\n')\n",
    "recommendations = response['ext']['retrieval_augmented_generation']['answer']\n",
    "# for line in lines:\n",
    "#     if re.match(r'[^\\s\\0]+', line):\n",
    "#         recommendations.append(line.strip())\n",
    "# Output the search results and shopping assistnat recommendations together\n",
    "print('Search results and Shopping assistant recommendations:')\n",
    "count = 1\n",
    "for hit in response['hits']['hits']:\n",
    "    print(\"Search result \"+str(count) + \": \")\n",
    "    print(hit[\"_source\"][\"product_description\"])\n",
    "    # print(\"Shopping assistant: \")\n",
    "    # print(recommendations[count-1])\n",
    "    image = Image.open(hit[\"_source\"][\"image_url\"])\n",
    "    new_size = (300, 200)\n",
    "    resized_img = image.resize(new_size)\n",
    "    resized_img.show()\n",
    "    count+=1\n",
    "    print('')\n",
    "print(\"Shopping assistant: \")\n",
    "print(recommendations)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "914facab",
   "metadata": {},
   "source": [
    "### 5.9 Check the conversation history\n",
    "\n",
    "To verify that the messages were added to the memory, provide the memory_ID to the Get Messages API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "226b9306",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Make the request\n",
    "response = aos_client.transport.perform_request(\n",
    "    'GET',\n",
    "    \"/_plugins/_ml/memory/\"+memory_id +'/messages',\n",
    "    headers={\"Content-Type\": \"application/json\"}\n",
    ")\n",
    "# Print the response. You should see a dictionary containing a list of messages.\n",
    "#print(response) \n",
    "for message in response['messages']:\n",
    "    print(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "655ecf2c-ad4b-45f6-a1da-4be935ef33cc",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 🎉👏🎈Congrats! You have succesfully experimented with lexical, multimodal, and conversational search in Amazon OpenSearch service!! 🎉👏🎈"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b08dd75",
   "metadata": {},
   "source": [
    "# Bonus: Deploy your web application\n",
    "\n",
    "To deploy this code as an application we will use [Streamlit](https://streamlit.io/). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "669597e1",
   "metadata": {},
   "source": [
    "**Step 1:** Export the connector and model IDs you created earlier in this notebook. We will store them to a file so that they can be referenced outside of the notebook and persisted between kernel restarts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82f6240e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "connector_ids = {}\n",
    "connector_ids['llm_connector_id'] = llm_connector_id\n",
    "connector_ids['llm_model_id'] = llm_model_id\n",
    "connector_ids['embedding_connector_id'] = embedding_connector_id\n",
    "connector_ids['embedding_model_id'] = embedding_model_id\n",
    "connector_ids['aos_host'] = aos_host\n",
    "\n",
    "with open('connector_ids.json', 'w') as file:\n",
    "    json.dump(connector_ids, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dc06a1e",
   "metadata": {},
   "source": [
    "**Step 2:** Since you will launch the app.py script using the Notebook instance's shell, you need to install the package libraries in the shell environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e78c846",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install streamlit -q\n",
    "print(\"Install completed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc39565-ae59-4041-99dd-997ef2f80015",
   "metadata": {},
   "source": [
    "**Step 3:** Execute this code to get the URL to access the app.\n",
    "\n",
    "This is the address the Jupyter server running this on this Notebook instance. **NOTE: the URL won't work until you execute the `streamlit run` cell below.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20eb1e8f-28d9-4e04-afb4-fd5421546f1e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "temp_url = boto3.client('sagemaker').create_presigned_notebook_instance_url(\n",
    "    NotebookInstanceName='semantic-search-nb',\n",
    "    SessionExpirationDurationInSeconds=1800\n",
    ")\n",
    "app_url = temp_url['AuthorizedUrl'].split('?')[0] + \"/proxy/absolute/8501\"\n",
    "print(\"App URL to use after executing the next code block:\")\n",
    "app_url"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac226163",
   "metadata": {},
   "source": [
    "**Step 3:** Run the streamlit application from the shell. Then use the App URL to access it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37e96561",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!streamlit run app.py --server.baseUrlPath=\"/proxy/absolute/8501\" --theme.base=dark >> tmp/temp.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f6bd5e2-e64e-4f32-b533-66aad5f54ab5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 🎉👏🎈Congrats! You have succesfully deployed a shopping assistant chatbot web application and completed the Builder's session!! 🎉👏🎈"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "421b907a-1c67-4da4-8c4f-ae89dab94f25",
   "metadata": {},
   "source": [
    "![title](diagrams/notebook-diagram-main.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "toc-autonumbering": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}