13_image_to_ad_copy.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Use Vision API to detect products and generate advertising copy\n",
                "\n",
                "This notebook demonstrates how to use GPT-4 Vision API and the Chat API to automatically generate advertising copy from product images. This method can be useful for marketing and advertising teams, as well as for e-commerce platforms.\n",
                "\n",
                "The full code is available on GitHub."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Building the models\n",
                "\n",
                "### Product\n",
                "\n",
                "For the Product model, we define a class that represents a product extracted from an image and store the name, key features, and description. The product attributes are dynamically determined based on the content of the image.\n",
                "\n",
                "Note that it is easy to add Validators and other Pydantic features to the model to ensure that the data is valid and consistent."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [],
            "source": [
                "from pydantic import BaseModel, Field\n",
                "from typing import List, Optional\n",
                "\n",
                "class Product(BaseModel):\n",
                "    \"\"\"\n",
                "    Represents a product extracted from an image using AI.\n",
                "\n",
                "    The product attributes are dynamically determined based on the content\n",
                "    of the image and the AI's interpretation. This class serves as a structured\n",
                "    representation of the identified product characteristics.\n",
                "    \"\"\"\n",
                "\n",
                "    name: str = Field(\n",
                "        description=\"A generic name for the product.\",\n",
                "        example=\"Headphones\",\n",
                "    )\n",
                "    key_features: Optional[List[str]] = Field(\n",
                "        description=\"A list of key features of the product that stand out.\",\n",
                "        default=None,\n",
                "    )\n",
                "\n",
                "    description: Optional[str] = Field(\n",
                "        description=\"A description of the product.\",\n",
                "        default=None,\n",
                "    )\n",
                "\n",
                "    # Can be customized and automatically generated\n",
                "    def generate_prompt(self):\n",
                "        prompt = f\"Product: {self.name}\\n\"\n",
                "        if self.description:\n",
                "            prompt += f\"Description: {self.description}\\n\"\n",
                "        if self.key_features:\n",
                "            prompt += f\"Key Features: {', '.join(self.key_features)}\\n\"\n",
                "        return prompt"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Identified Product\n",
                "\n",
                "We also define a class that represents a list of products identified in the images. We also add an error flag and message to indicate if there was an error in the processing of the image."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [],
            "source": [
                "class IdentifiedProduct(BaseModel):\n",
                "    \"\"\"\n",
                "    Represents a list of products identified in the images.\n",
                "    \"\"\"\n",
                "\n",
                "    products: Optional[List[Product]] = Field(\n",
                "        description=\"A list of products identified by the AI.\",\n",
                "        example=[\n",
                "            Product(\n",
                "                name=\"Headphones\",\n",
                "                description=\"Wireless headphones with noise cancellation.\",\n",
                "                key_features=[\"Wireless\", \"Noise Cancellation\"],\n",
                "            )\n",
                "        ],\n",
                "        default=None,\n",
                "    )\n",
                "\n",
                "    error: bool = Field(default=False)\n",
                "    message: Optional[str] = Field(default=None)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Advertising Copy\n",
                "\n",
                "Finally, the AdCopy models stores the output in a structured format with a headline and the text."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": [
                "class AdCopy(BaseModel):\n",
                "    \"\"\"\n",
                "    Represents a generated ad copy.\n",
                "    \"\"\"\n",
                "\n",
                "    headline: str = Field(\n",
                "        description=\"A short, catchy, and memorable headline for the given product. The headline should invoke curiosity and interest in the product.\",\n",
                "    )\n",
                "    ad_copy: str = Field(\n",
                "        description=\"A long-form advertisement copy for the given product. This will be used in campaigns to promote the product with a persuasive message and a call-to-action with the objective of driving sales.\",\n",
                "    )\n",
                "    name: str = Field(description=\"The name of the product being advertised.\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Calling the API\n",
                "\n",
                "### Product Detection\n",
                "\n",
                "The read_images function uses OpenAI's vision model to process a list of image URLs and identify products in each of them. We utilize the instructor library to patch the OpenAI client for this purpose."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": [
                "def read_images(image_urls: List[str]) -> IdentifiedProduct:\n",
                "    \"\"\"\n",
                "    Given a list of image URLs, identify the products in the images.\n",
                "    \"\"\"\n",
                "\n",
                "    logger.info(f\"Identifying products in images... {len(image_urls)} images\")\n",
                "\n",
                "    return client_image.chat.completions.create(\n",
                "        model=\"gpt-4o\",\n",
                "        response_model=IdentifiedProduct,\n",
                "        max_tokens=1024,  # can be changed\n",
                "        temperature=0,\n",
                "        messages=[\n",
                "            {\n",
                "                \"role\": \"user\",\n",
                "                \"content\": [\n",
                "                    {\n",
                "                        \"type\": \"text\",\n",
                "                        \"text\": \"Identify products using the given images and generate key features for each product.\",\n",
                "                    },\n",
                "                    *[\n",
                "                        {\"type\": \"image_url\", \"image_url\": {\"url\": url}}\n",
                "                        for url in image_urls\n",
                "                    ],\n",
                "                ],\n",
                "            }\n",
                "        ],\n",
                "    )"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "This gives us a list of products identified in all the images.\n",
                "\n",
                "### Generate advertising copy\n",
                "\n",
                "Then, we can use the generate_ad_copy function to generate advertising copy for each of the products identified in the images.\n",
                "\n",
                "Two clients are defined for the two different models. This is because the gpt-4-vision-preview model is not compatible with the gpt-4-1106-preview model in terms of their response format."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [],
            "source": [
                "def generate_ad_copy(product: Product) -> AdCopy:\n",
                "    \"\"\"\n",
                "    Given a product, generate an ad copy for the product.\n",
                "    \"\"\"\n",
                "\n",
                "    logger.info(f\"Generating ad copy for product: {product.name}\")\n",
                "\n",
                "    return client_copy.chat.completions.create(\n",
                "        model=\"gpt-4o\",\n",
                "        response_model=AdCopy,\n",
                "        temperature=0.3,\n",
                "        messages=[\n",
                "            {\n",
                "                \"role\": \"system\",\n",
                "                \"content\": \"You are an expert marketing assistant for all products. Your task is to generate an advertisement copy for a product using the name, description, and key features.\",\n",
                "            },\n",
                "            {\"role\": \"user\", \"content\": product.generate_prompt()},\n",
                "        ],\n",
                "    )"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Putting it all together\n",
                "\n",
                "Finally, we can put it all together in a single function that takes a list of image URLs and generates advertising copy for the products identified in the images. Please refer to the [full code](https://github.com/jxnl/instructor/blob/main/examples/vision/image_to_ad_copy.py) for the complete implementation."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "## Input file\n",
                "\n",
                "The input file is currently a list of image URLs, but this trivial to change to any required format."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [],
            "source": [
                "image_urls = [\n",
                "    \"https://contents.mediadecathlon.com/p1279823/9a1c59ad97a4084a346c014740ae4d3ff860ea70b485ee65f34017ff5e9ae5f7/recreational-ice-skates-fit-50-black.jpg?format=auto\",\n",
                "    \"https://contents.mediadecathlon.com/p1279822/a730505231dbd6747c14ee93e8f89e824d3fa2a5b885ec26de8d7feb5626638a/recreational-ice-skates-fit-50-black.jpg?format=auto\",\n",
                "    \"https://contents.mediadecathlon.com/p2329893/1ed75517602a5e00245b89ab6a1c6be6d8968a5a227c932b10599f857f3ed4cd/mens-hiking-leather-boots-sh-100-x-warm.jpg?format=auto\",\n",
                "    \"https://contents.mediadecathlon.com/p2047870/8712c55568dd9928c83b19c6a4067bf161811a469433dc89244f0ff96a50e3e9/men-s-winter-hiking-boots-sh-100-x-warm-grey.jpg?format=auto\"\n",
                "]"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### TODO: Implementation from [full code](https://github.com/jxnl/instructor/blob/main/examples/vision/image_to_ad_copy.py)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": []
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.12.4"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 4
}