tutorials/multimodal-image-processing/llama-3.2-image-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6555c580",
   "metadata": {},
   "source": [
    "# Using Llama 3.2 Vision for Image Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47670e56",
   "metadata": {},
   "source": [
    "In our [previous tutorial on Llama 3.2 Vision](https://github.com/groq/groq-api-cookbook/blob/main/tutorials/llama-3.2-vision-function-calling.ipynb), we used function calling capabilities to identify the dog breed from an image and use it to call the [API Ninjas Dogs API](https://api-ninjas.com/api/dogs) to extract additional information about the breed. In this tutorial we'll take it a step further to identify more features about the images and create a robust image classification tool for those same dog pictures."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e441f5a1",
   "metadata": {},
   "source": [
    "To do so we'll use Llama 3.2 Vision in conjunction with [Groq API's JSON mode](https://console.groq.com/docs/text-chat#json-mode-object-object) to extract specific information from the provided image, such as the breed of dog or its characteristics. By doing so, we'll show how to create a pipeline for image classification and data extraction, converting unstructured images into structured data that can be easily searched, filtered, and analyzed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "942c7579",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7d7113fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from groq import Groq\n",
    "import base64\n",
    "from IPython.display import Image \n",
    "import os\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d4de208",
   "metadata": {},
   "source": [
    "To complete this tutorial, you'll need a Groq API key. If you don't already have one, you can easily create a free account on GroqCloud by clicking [here](https://console.groq.com) and generating your API key. Once you have your API key, you'll need to set it as an environment variable called GROQ_API_KEY for the tutorial to work. We'll be working with the llama-3.2-11b-vision-dev model throughout this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "319112fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Groq()\n",
    "model = 'llama-3.2-11b-vision-preview'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "872cc6b6",
   "metadata": {},
   "source": [
    "### Basic LLaVA usage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f865ab",
   "metadata": {},
   "source": [
    "For this tutorial, we'll be using the same set of dog images from [Britannica](https://www.britannica.com/animal/dog) that we utilized in the other tutorials. You can access these images in the Groq API cookbook repository, specifically in the [images folder](https://github.com/groq/groq-api-cookbook/tree/main/tutorials/llava-image-processing/images). We'll be encoding a local image to base64 to integrate it with Llama 3.2 Vision. This is one of two methods to use an image with the vision model - the other approach involves providing the actual URL of the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5f96b29e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "UklGRloVAABXRUJQVlA4IE4VAADwdQCdASosAcMAPm00lUgkIyIlo3RbWLANiWduMTTJrEA2Tql6gui6WzmBAaEOYTf50u/W70/P/Mts5yHuys1pmgH5Rn+/5h/rzgjkJKhH7xJ7/4FZ/5fh9R2F6tJ+5R5mJECIsHvuKYHUl9EFp7C0hr7Vf06IJQRQmUnPI5zEg9c808HpUxP9kc4ejcWcqH1QIkWq0M/J9OrBdlhxds9Mn2JgZPOSkXUaT1Tqk3uK+c/j4F0vDBPmCAwxuhRBKvb90bkLR6MOmBJ4/QSHq0TIeTkDvFz+jfOOJabYkuq8qsO2Kcfxuo9D9hig6kODQHvdJc/dnLEyW6cauL/1sOYF3HOeTxNKj68Jqrufsn5aJafME0HBGcd+70Pn6hQxQubABudg4mbfsrHWyXsxEUIViyOdbVjV7UORgBIeSxc/5AAVN+vg54diIkbXBaZpK2fy/sqtrK9oKHCl/KnFmXnyDc8rBzxvrvvSr2yzsd36zxxXvX6RO55mFvqrXWjR2KIgJCXl5SG/2mgMT7PVoXweg5LhqB0KfjTEq1Wyers963F7a1j7ir5DK1nCK/uA+tt6aE3QYIAf1tXPpylJqhSvKCX+Az5fm3gBK6xQJmdkNTyZBDoXsVn/g2T10RLPUs3FtkUZN2qiRepepzp4QlXykq7KzTugK2HVZIuidO9iHs2kSU6AjkonYoOnAH0TXbWKOSQO3DXGN7nx4c8pczRxckmq3xV2Mk6XWSTiNSHbn5SZwzm4da48ctM4M5QEjS3YnGHvu5UAueDBUreyKtRJCrPPsGO6I4gOl5ORgoqqhpRLMaMm3UiGNTdIpmE4/szWHW4u7JfSdAIXoc4cJaeeR3Omb3J5/Db4sLyBBmPxFfbFTOtT9JZ4iK4vOAyJiZHibI+kVDtn0bJ0HdbfK0dq1UwuZgyVstX93BM6AFdUmknaHaulc/dTr0C0SzIdjdGDGh8TI0SsVS5MfQKusoYr5T/vJ28B9XH8DIIzoTeH8LYWkMq6MA/KPQAm9woeEi4QN7gCUY14PwKWFZuV43u5ogH+jk4R1DhzSBKqihhJ1PMe64B5vC7vV8bJensSPfwCZu91stN1W5KKxl+qNR3lf18D7+w8vl4YgmpNS1/7YTvqErJLeO9CZwqp9l6YFiKr45FU2QZi48/sN2yUYu3mh8L/2cT2sud8mxJPVwgKDLxaQVQVxnNOy528yIaUfe6gPgb7D61i7ny1ZOUdLco1/WV3Y9/n3ojZbVUhh/cLTYvvbHs74vYAAP78qUjclm+pBW8g7q+a//1ku3H8Wgnw1ryYZ89XK6SPars/x7Lgz45oBjs50U6BrrulRiff7VZ1B2y6NdTfmk2YvqKHNo7mtn1K3K8xP7CjlpY/PxLgrq02YS1HWDl9PQwqDWtfagU1i3IDbUQXC/DnZdBJsTsgu9HOhjELF8M32llhXyksBNQrVJ/CZRP4PdwuikY4hSq7KYIwQyTne/wlydiaHoqgf9npYkHaqstPPw1ZZ1CGW+ORmyQh0rftefqzBjVV6Ldh9m1W/50GIMnQRabw4FLvycZx/dIgdpAVAXQhy2lYznM5YS0+RJhdMJALnMSfc3EXSGYzwYHD7huIsJ4xmszEf14K0ZtG63hcvg9Do9hK4rlaUZyOw9mYHUehFo3lxw0MDajbidFW/JP3jr1SThmE/WPvcC9ywGN2DbSykFamkuSt4wbBgKWYGEv7HMToExY+wFLUbFN1KeNtgSoxeX+HPQSvYLbCt/e7JoduZrhqbwvSQEmK5tjgLgqG3tluGMwTzl0pS9VzgL1/TC1tRQDS6A3tBd1A5v3aq2NZ8rmEW0jSBRAlXl1fIOoB71ojuAJuDhrHRrd4XTQ8rJY8emx7RNWnZbbwF/LrbD0y1A08cMiXS03hvlmvAVmZ7DB8R3EjyOqepeNIu+g2ohtvk1X469n+7zNDYnDRveJDTBvUb5qjFRT33YvQ0JHT1JViv2fCJcuKwAwqfzXp83uWaWio3SyCm+AC7rbGc5+/YSEj6qGxDCaBXHqfRSfwaEQM81IPAjJcd+KjfCXhRDjux/7Zbo/vA9WRMWKn6fKq4N+szuIQrMn2flCtcIHqkUXlcs6YUcsWwBvai9D+Yq6jmd0uB2WwSL/1VKwB7nZvbLMzrnCAPx/NpQR9rHpmPDq/WdX8bcqo7y7SciZqVn+MA9lDbmPApg/+3ZjtI3oB/zNpRaHyLAMpjYSYogVuZlTFUAEdysq8iH9DchM7OH/kLd/lmdeedBfHEc07WtMwLHaMvMBU1wmAjs3I0mB6y8Fpi+39nrDNg7reqUc51932YeWFllpNp66MTj42Tpv6L3yWFy+8DtUKNdSzOtncKzwWbJt9HI41DHk4Rf0F7LbGX4xmyWi7NANB5juNZoe7SSMEslO5TZO8jXT5aksBHJ2gDnj6KTJcKmqf1OCKUkB9i/8eV61O3ydN37bEzAh1ycNlYFfTh0yHBrmTxW4f7qGdXV6GC0srYVRkgYmQzfCrYOE15DJ3g3FPCEm/OUczGN7UZZaSoDPFP23MRdTUxr49bMB/WcEdzqvOR8E1basT5ALoOkzv0duBg0A1Oe18Kq9LjsCgbwlJf3d8gggD6gIo3e+sYkLoFaPTZtkn992SQRXTpiV5iqnWKN/Z3KxAv+aZip+vBnAqjs+UBm92oRZHnVQ9xZUDJVzStwjaHsBL22I1YnA6g3XZiVeS40H8v+HsqSzwYmNZdB+yKvr0T2D1Ij72OBcxoiCy9UpQphHbAr0H1vfZp47fbDVO0T4D8gRsKC1Ox5O+IHPHG44aRzEHzRWdVlLwYCTnrUmg0Ge0WMSAh9rLkPNEhaFB3LNMfdWA6zeV3KsGKpN16HVlDA7EDG4iwjLvB6LMbqUSjmHTtqdgOogkQzsxxMPnr2P8cFqOaShZ0IJTU6AIIJP/Yi5Kt3ZfHS47zULGje/oFAbrKQ91w7FtUJoCybjK81CI5hC/PSEGMsCWVgZ2PCFLRK/RPiFLL89Ljz/PDYH31sf6nKtLOj5M6wVW4AsAxgQEPy6zek+iS0vAFx8C2tX9X/ivu+nw3td4exEB92lN/BVDEaT6IzDXmoMvoNroxdnQcM6OPUnWBbQRrpz+fRtuSwvyEuuqaxIfM3UYpB9HgOSmy+RcKuQcqbV5Pr7eE4FJnmxTUJ7ofxilyB38DXLwK2QLuw1NUWjJEn8ubMHDOex8gvR8D1OzLvdiiS1qbdMgd9jo5duha2ujcg9CRpigI/wPY3+BEUQcyiIacRthyxvqqh0gtu0Ilw3B5Xg/mevpEUl2tMoqk0oK2KDSHzjd7PvUM8a+MNd9zTCbWvvsPqFko57SPAv8xNS/e4OW0lMQOzhaersgM0PN5Bw8cQRAD1hbADavxGEdy9atdNUIG0yMx5KsUxuqqVPeZhcx6zMXcwXUoarNMWcBv2iw7p3a1guoHk5JoNfupdiXxTFN07CxpD5IZIPKUVn0XZdcjAdwG+MIdFUwtaHMMdhscQ0KJyRdHyfi9yq5Uh++vs/IrE9JHedKgqUpaF13XCXhz5sqcK/B31ZJ8XjijC+PtFQ48P/6jz3Pnxf8alFZMigPmXoYDXPnmmP5zwRGgaJyC2yHGm0DrNjDGTJCjlYOu9PCAc3czDP7ZuKxk1OuQks+XtrXYepQOiGvXJ9BE11tmEgTzGItxbdxVHxfIhQo/ziEtYWdljTmvg5QceODSX4Fru7inTjLj0fytb/AOCak3akwichxnZRa8I+g1tzOt0Vql/w3MNsMbm7ie8jH/MRzlqi7ncLtn0Gok2ONeNED0VtVW3KGaWGhP6nYAxQhVXGKaSUWFIc65ZWN0BYmE/MugKiE94urNd/ui53UxEGXgqER2FEt2/41VtvvfDs5GmJS+yj2U/be7Ntf29clC2eU/S4OUHuAjhko4DW3Cu7N6p9NEw5WJddx2TPRIVksUyhCpUKm9DryQPu/28qATv/a4uac7gioxsqfDyngESaGbGQ7wSjxAkUaVJGhJjEAjpWoTjZc3r5auRUMDKeaLvijy8xLkWORaK0OFU26qqEC78gcL8MIIk2+ULeNrephBqGc52Ar9LcVMJbNORFOtS93L0mm/iFlols8HuICitbmhfEOdjsY7dVWSaPVZHrv6iPm8ldT8HDEGlfsJDNbwKiN/K1vwrwfjb//zImASLRtTsUdEjgdVwZSG7nEqOR2mJ9g3X0CvGK4bgkwSnRf5f2jX2muFFia3M7DI6cZUjxw8TJxNLTEBk4KTusuyhFNrA+g5MuOM293I9uQbZ2Ixcroa5QLgtcmkyBZ3r2hucrRw98qcSGsXi31dZlKQMlajqXrS1OtvRDYDbehjc0K74CLD6m1US0nNZ2nnLPIlvTKLKTRxnK8IUZO1lsigGc09hKRs6rzU1GvooWVtgDs63pO8Ikdnnibl+aSC7Q77qG5gPvKzytQZRTNrVDfke99XonkPu0/HS1u2s6miE99MnM7zqsaScH4gVuY3Llj0mzJwAf1iDElURlf5bLXwvy4FA7IMsi6gFwGLYJlP57x/TPAa/FqjD8lNmSH2r+Anm98Zg2cqOH8rD+gqRHXd6P1z+eVvJjJa+dln88ffyzxUtq2q3l8k6hd8RBWcU21EUdr2YxKyYgKeEVyIWAA5fcMfO1I6ZwSdUjXrJBuYOLJJ7++GmpRjTTxuACcDOvtCUXIfftRxoTU8E4OeMkkRh9Zv84u8KkBWzthWW+tqwhDzUIs8snEzU7jqQiE3U0K2GunK0mNE/nbF2aRh3HKR9Qx+g8Pv8zz/Seh1+PTuWECLJEV3Z2r2Z/0C8pLAco4f69RZ31zpkR9NQH+I0JXoVLdeccs7FhLPhucbdjIiL9w19vAWtcpphTNRnb+9BZ67Q0hWt/HC7Dlhy/xsWk+jwOgG10ag6nIpb6Lf2u0vvmtCWqBKcdf3Pr6U5zbe6Jfm1BpdzTa2uxGYp445e8JE7QENvrbzCF/P7nGxb3ouUrGlzvkVbHO3yLtcAjxcr29e4Pih1fqtKswVrVnTSYwZPxi07J5xX6XJy7XuK35E5qSQGCTaWC3PDphujdIITIJ0flsdRcQO9X4+Jrc60/D6/BfCzhBVE7Dc4hL7OzGaUIjeGae1o0HCnH8BMLk256G8fulQ2gv1v9llVt6FTD+C7Z7IWCzO87FE3r95oXzL1lpSq4MkC3W1jI1qrDyT0/4AdM3RDKNVjZPWcgkOGBUaRX+pIV9o/+S9IolSnWcRaBTt4FCYlK3lzXHQc3uRa/syO6S2Htk/lorWH810IEYg/Xr1ZQSof0iZqvkM9AyOqcrhVNrJd7mf0juG68GSd3RpcrS1XMY9Y/b7+DKnAI4k//HXw+jN0MQKQSIBqXJZ28h1hZ6hZ7LTFpvk2VS/ur1pnE9wNJVKJYEyFMcb4ITrDc5BKo063i+qBEw90G3QKGFAX6jF/z9GKSK2SSLaY/yrGfbPEzTRgu/NnTKpVqJvUmi9kxdGn0VDOzqg+mRovkbstNF589CM8F7hAaJBggJPhOVnBObaXXjlLUGVE+TPAju7NfSIm/Kr0yy/RmdDTIhEj5gxdUyroxSkdgp3uUVdS0/bjODVw/NgVzPIfbCBuFewScsivOhtkR/u8UinkvcEwLzi2h0Fmu79K3yYjJfkoMWj9oMIs+ntTCphiB+qPyssYi9H8Khpoi9Y20tM6GIrMyC1C5WeQW3vmqui0gaK8N+Br14LGhKMKHFvWXWxgSvkAWp/ZuCgKtOJBqcAitBpGjgep2rIkQRR7+aoPRdx58XhcytJWw4Wvzftx+XnaRuWI52HNFnbCvr58qcLk+4+ncgQym6gDw38axnK6Jxnl9YxG/XLfhgOcFHm5O/NHNd2Zb/1jtfxN43pSANJ1tNm+0x+RTZGQkBg9eXfjyp5kFddLpT5Abm/Wp6firlpUVBcSMXbIRrdvdG5aFKpk5+vTyct8ekVd68R5BHloudj/ho1QxobaFlRKgAV7vgqMoqH3U821tR8jRMRCEDFVfimpH8mMDQegOkLVHN3iLlPfSt4hK4VVpYVD2Gxg1vxVgv8i2A60SH1VQJ+N06zeGL1O0hEvDTigfK+Qx881ibvfRiTbgIBb4b2XMI6SBszxyTrOWX8mmczYPOPHkpUBwv4q6cLIfqFW7Awiade3k+cV7hFYS/fY0JAHlj23BHHMjODEExxJWxAyp8a88naqbStW0uddcJLpheijjb76LrjO0KXHT6HXNyKI2e8d5/ZmqeeH+oHy6whRTpcSHW+j8n2mMdGi/MuIXKzoR1ALLMOB+1U99or/8gdADWT4/gXnQpyAO09pdbW6jk+xqjjFxRurFQZiWuhV9scYKfHcElsRm1AxjiuqApCsuOsuZp9Jna85NHwOhmc7cq1NhIwTFp6Pf3WCq05m+QgJfEMbUN97lGK3Xyz3Erg7jSdatT72tuQaWYZ197kw96aqCrNPjyuP3iIgorFggPQsPP31lL9KVsire+QgBg3lPNC6rzsCOxUoJSsJ/jK6KZuO19k9tZ86K7SetAhm5eCsQFXQYnOmWp35YDAOchIUJYz9cv3riasapLWVj0ycCydOIcPP/fmYVMWA1DJM4vQQL9f9ya4WdBtJmfnXBoZU2Qs9hf5NqfjF40/Mfo6LwVrL0pTbUsA2q6qDuck/ovGzyGKS9Pv24DcTSWpxHBgoCD0rhLHKsZ/VwHfVxt1N8b/qjku/YFBYjPAYmUBvfhX1bqf3puvmiA4HaEwxcRHq0NBZw/O6UN3Rodn7GUqs8szaj/g54IVDcUwcqtLZ3uCDHVrLByeR7gKz77crNkNkH0jee1RtIQEBZ2i+85j9dicvgr2hiHJ39pY9i/hax+DQ4WV8tezhxCp9Sr2xfMQ65lzXvxM48k4ktRFNurWLyJ68+8hiTzPItdBo4vCQL+RdCCGLIzvLuTNKY9m2f0V07K2yNbLSEj1ImBoBnw+5+DXkrjMiqj0Y0bpi4OnxWvYovgBiMjHJejZJHGH+JU3LaNelH4AB1Ka0JAgg4t4bqNRMlKvn/R+nXeG1ErWkAivlwP1MLjFH38He26rKkbsuNcLTommyJowloaQTJmV+X55evosXvLuh/16xeu+o4M9cBiC4ya+BqtmHE4od7D8nbGlTMYmnaMK7n0A0kHg/hrhpaCgxYczMloYGKF6ubVdt1a2+SA+H+G1vPBHTDzIS9+IMn8pzShlQOUTyxbSnE9k2Olj4gcOotPYjH5rPG8tKEOkTu5gyRsqGrHAAA=",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load and display the image\n",
    "image_path = 'images/dachshund.png'\n",
    "Image(image_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16406b6b",
   "metadata": {},
   "source": [
    "To use locally saved images with Llama 3.2 Vision, we'll need to encode them to base64 first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7b202bb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_image(image_path):\n",
    "  with open(image_path, \"rb\") as image_file:\n",
    "    return base64.b64encode(image_file.read()).decode('utf-8')\n",
    "\n",
    "base64_image = encode_image(image_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96637a40",
   "metadata": {},
   "source": [
    "Analyzing an image with Llama 3.2 Vision using Groq API functions the same as with LLaVA. We'll make a request to the `https://api.groq.com/openai/v1/chat/completions` API endpoint, passing the image and prompt in a JSON structure that includes the image path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "381a9065",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The image shows a side view of a black and tan dachshund. The dog's dark brown coat is emphasized more on the outer body, with light brown splotches running on its throat, chest, and paws. Its tail is plaited/braided and held over the dog's back, and its ears, which are floppy, are tucked over its head.\n",
      "\n",
      "The dog is standing on a paved surface, indicating that the photo may have been taken in a purpose-built dog training area, dog park, or outside an animal clinic. The background is out of focus and appears to be a white wall made of tiles with a line of darker brown trees. \n",
      "Overall, the image is well-composed, making the dog the clear focus of the shot.\n"
     ]
    }
   ],
   "source": [
    "user_prompt = 'Tell me about this image'\n",
    "chat_completion = client.chat.completions.create(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": [\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": {\n",
    "                        \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n",
    "                    },\n",
    "                },\n",
    "                {\"type\": \"text\", \"text\": user_prompt},\n",
    "            ],\n",
    "        }\n",
    "    ],\n",
    "    model=model\n",
    ")\n",
    "\n",
    "print(chat_completion.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac7dba7-1c46-427a-b822-bf17e5418f94",
   "metadata": {},
   "source": [
    "However, Llama 3.2 Vision offers more functionality beyond just image recognition. It offers most of the functionality of any other chat model, including tool use and JSON mode (the only thing it can't do is process a system prompt). By leveraging JSON mode, we can utilize Llama 3.2 Vision to identify the breed of the dog in the image and return the result in a structured, easily digestible format. Note that to use JSON mode in the Groq API you must include \"JSON\" in your system or user prompt, and specify the desired output (see this [cookbook tutorial](https://github.com/groq/groq-api-cookbook/blob/main/tutorials/json-mode-social-determinants-of-health/SDOH-Json-mode.ipynb) for more information):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "89563052",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dog_breed': 'Dachshund'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_prompt='''\n",
    "You are an image classification API specializing in dog photos that responds in JSON. What dog breed is shown in this image? \n",
    "Return in the following JSON format:\n",
    "\n",
    "{\n",
    "  \"dog_breed\": \"string (e.g., 'Golden Retriever', 'German Shepherd', 'Mixed')\"\n",
    "}\n",
    "'''\n",
    "\n",
    "def image_classification(base64_image, user_prompt):\n",
    "    chat_completion = client.chat.completions.create(\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\n",
    "                        \"type\": \"image_url\",\n",
    "                        \"image_url\": {\n",
    "                            \"url\": f\"data:image/jpeg;base64,{base64_image}\",\n",
    "                        },\n",
    "                    },\n",
    "                    {\"type\": \"text\", \"text\": user_prompt},\n",
    "                ],\n",
    "            }\n",
    "        ],\n",
    "        model=model,\n",
    "        response_format = {\"type\": \"json_object\"}\n",
    "    )\n",
    "\n",
    "    return json.loads(chat_completion.choices[0].message.content)\n",
    "\n",
    "image_classification(base64_image, user_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "485b6787",
   "metadata": {},
   "source": [
    "### Image Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2143a88-c7bd-44c7-854e-49d58482bf48",
   "metadata": {},
   "source": [
    "With JSON mode enabled, we can leverage Llama 3.2 Vision for robust image classification at scale, making it easy to analyze and categorize large datasets. By iterating through each image in the folder, we can utilize Llama 3.2 Vision to classify them based on various features, such as breed, color, number of dogs, dog activity, and more. This functionality could be particularly valuable for a pet adoption agency, which could use it to streamline the adoption process by uploading an image of a new adoptable pet and instantly making it searchable and filterable by certain features on their website."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae6b951-6a8a-4fc1-8f93-767c61febc4e",
   "metadata": {},
   "source": [
    "First, let's expand our JSON mode prompt to include an array of additional features, and test it on our Dachshund image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "060e2c14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dog_breed': 'Dachshund',\n",
       " 'dog_color': 'Black and Tan',\n",
       " 'number_of_dogs': 1,\n",
       " 'puppies_present': False,\n",
       " 'dog_size': 'Small',\n",
       " 'is_pet': True,\n",
       " 'dog_activity': 'Standing',\n",
       " 'background_environment': 'Outdoors',\n",
       " 'dog_accessories': 'None'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_prompt = '''\n",
    "Your job is to extract structured data from an image of dogs and output the structured data in JSON.\n",
    "The JSON schema should strictly be the following:\n",
    "  {\n",
    "    \"dog_breed\": \"string (e.g., 'Golden Retriever', 'German Shepherd', 'Mixed')\",\n",
    "    \"dog_color\": \"string (categorical: 'Black', 'White', 'Brown', 'Golden', 'Multiple')\",\n",
    "    \"number_of_dogs\": \"integer (number of dogs in the image)\",\n",
    "    \"puppies_present\": \"boolean (TRUE if there are puppies in the image)\",\n",
    "    \"dog_size\": \"string (categorical: 'Toy', 'Small', 'Medium', 'Large')\",\n",
    "    \"is_pet\": \"boolean (True if the dog is a pet, False if it might be a wild animal like a wolf or coyote)\",\n",
    "    \"dog_activity\": \"string (categorical: 'Sitting', 'Standing', 'Running', 'Playing', 'Sleeping')\",\n",
    "    \"background_environment\": \"string (categorical: 'Indoors', 'Outdoors', 'Park', 'Beach', 'Forest')\",\n",
    "    \"dog_accessories\": \"string (accessories or items the dog is wearing or carrying, e.g., 'Collar', 'Leash', 'Bandana', 'Toy', 'Stick', 'Ball', 'None')\"\n",
    "  } \n",
    "  \n",
    "Use the information from the following dog photo to construct the proper JSON output.\n",
    "'''\n",
    "\n",
    "image_json = image_classification(base64_image, user_prompt)\n",
    "image_json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5c10175-47f1-431d-a79f-a2ff3da1ff88",
   "metadata": {},
   "source": [
    "Looks good! Now, we can apply it to each dog image in our file folder and append the JSON outputs to a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d7a26062-c0c8-41bd-9679-8e97c182bd69",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_folder = 'images/'\n",
    "image_files = [f for f in os.listdir(image_folder) if f.endswith('.png')]\n",
    "image_json_list = []\n",
    "\n",
    "for image_file in sorted(image_files):   \n",
    "    image_path = image_folder + image_file\n",
    "\n",
    "    # Encode Image\n",
    "    base64_image = encode_image(image_path)\n",
    "\n",
    "    # Classify image in JSON mode\n",
    "    image_json = image_classification(base64_image, user_prompt)\n",
    "\n",
    "    # Add image file name as a feature\n",
    "    image_json['image_file'] = image_file\n",
    "\n",
    "    # Append to image JSON list\n",
    "    image_json_list.append(image_json)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb4332ee-5a80-4de0-a589-2b89591c2b20",
   "metadata": {},
   "source": [
    "Finally, we'll convert the results to a pandas dataframe. Now we have a structured table of features from all 18 dog images that can be processed, analyzed, or stored in a database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c6046950-0487-4208-ae22-780a9c6cc23b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dog_breed</th>\n",
       "      <th>dog_color</th>\n",
       "      <th>number_of_dogs</th>\n",
       "      <th>puppies_present</th>\n",
       "      <th>dog_size</th>\n",
       "      <th>is_pet</th>\n",
       "      <th>dog_activity</th>\n",
       "      <th>background_environment</th>\n",
       "      <th>dog_accessories</th>\n",
       "      <th>image_file</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Australian Shepherd</td>\n",
       "      <td>Multi</td>\n",
       "      <td>6</td>\n",
       "      <td>True</td>\n",
       "      <td>Small</td>\n",
       "      <td>True</td>\n",
       "      <td>Sitting</td>\n",
       "      <td>Outdoors</td>\n",
       "      <td>None</td>\n",
       "      <td>australian_shepherd_puppies.png</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Basset Hound</td>\n",
       "      <td>Brown and White</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>Medium</td>\n",
       "      <td>True</td>\n",
       "      <td>Sitting</td>\n",
       "      <td>Outdoors</td>\n",
       "      <td>None</td>\n",
       "      <td>basset_hounds.png</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>English Bulldog</td>\n",
       "      <td>White and Brown</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>Medium</td>\n",
       "      <td>True</td>\n",
       "      <td>Standing</td>\n",
       "      <td>Outdoors</td>\n",
       "      <td>None</td>\n",
       "      <td>bulldog.png</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chihuahua/Mixed</td>\n",
       "      <td>Brown</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td>Small</td>\n",
       "      <td>True</td>\n",
       "      <td>Standing</td>\n",
       "      <td>Outdoors</td>\n",
       "      <td>None</td>\n",
       "      <td>chihuahuas.png</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Coyote</td>\n",
       "      <td>Brown/Golden</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>Large</td>\n",
       "      <td>False</td>\n",
       "      <td>Running</td>\n",
       "      <td>Outdoors</td>\n",
       "      <td>None</td>\n",
       "      <td>coyote.png</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             dog_breed        dog_color  number_of_dogs  puppies_present   \n",
       "0  Australian Shepherd            Multi               6             True  \\\n",
       "1         Basset Hound  Brown and White               2             True   \n",
       "2      English Bulldog  White and Brown               1            False   \n",
       "3      Chihuahua/Mixed            Brown               3            False   \n",
       "4               Coyote     Brown/Golden               1            False   \n",
       "\n",
       "  dog_size  is_pet dog_activity background_environment dog_accessories   \n",
       "0    Small    True      Sitting               Outdoors            None  \\\n",
       "1   Medium    True      Sitting               Outdoors            None   \n",
       "2   Medium    True     Standing               Outdoors            None   \n",
       "3    Small    True     Standing               Outdoors            None   \n",
       "4    Large   False      Running               Outdoors            None   \n",
       "\n",
       "                        image_file  \n",
       "0  australian_shepherd_puppies.png  \n",
       "1                basset_hounds.png  \n",
       "2                      bulldog.png  \n",
       "3                   chihuahuas.png  \n",
       "4                       coyote.png  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dog_image_df = pd.DataFrame(image_json_list)\n",
    "dog_image_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "149e4063-8472-47e4-909b-58197d71bdfc",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae4d9e2a-9c03-4f90-a572-3e7affa45ab4",
   "metadata": {},
   "source": [
    "One of the most valuable things LLMs can do is convert unstructured data to structured data, and as we've seen in this tutorial, you can use Llama 3.2 Vision to do just that with image data. This ability to extract meaningful features from images opens up a wide range of possibilities for applications such as image classification, object detection, and more, and we're excited to see the innovative solutions that developers will create using Llama 3.2 Vision on Groq API!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}