function_calling_python_quickstart.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2edc81e382cf"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "906e07f6e562"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeadDkMiISin"
      },
      "source": [
        "# Gemini API: Function calling with Python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEXQ3OwKIa-O"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://ai.google.dev/tutorials/function_calling_python_quickstart\"><img src=\"https://ai.google.dev/static/site-assets/images/docs/notebook-site-button.png\" height=\"32\" width=\"32\" />View on ai.google.dev</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/tutorials/function_calling_python_quickstart.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/google/generative-ai-docs/blob/main/site/en/tutorials/function_calling_python_quickstart.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "df1767a3d1cc"
      },
      "source": [
        "You can provide Gemini models with descriptions of functions. The model may ask you to call a function and send back the result to help the model handle your query."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FFPBKLapSCkM"
      },
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFNV1e3ASJha"
      },
      "source": [
        "### Install the Python SDK\n",
        "\n",
        "The Python SDK for the Gemini API is contained in the [`google-generativeai`](https://pypi.org/project/google-generativeai/) package. Install the dependency using pip:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9OEoeosRTv-5"
      },
      "outputs": [],
      "source": [
        "!pip install -U -q google-generativeai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KCFF5VSTbcAR"
      },
      "source": [
        "### Import packages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRC2HngneEeQ"
      },
      "source": [
        "Import the necessary packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TS9l5igubpHO"
      },
      "outputs": [],
      "source": [
        "import pathlib\n",
        "import textwrap\n",
        "import time\n",
        "\n",
        "import google.generativeai as genai\n",
        "\n",
        "\n",
        "from IPython import display\n",
        "from IPython.display import Markdown\n",
        "\n",
        "def to_markdown(text):\n",
        "  text = text.replace('•', '  *')\n",
        "  return Markdown(textwrap.indent(text, '> ', predicate=lambda _: True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHYFrFPjSGNq"
      },
      "source": [
        "### Set up your API key\n",
        "\n",
        "Before you can use the Gemini API, you must first obtain an API key. If you don't already have one, create a key with one click in Google AI Studio.\n",
        "\n",
        "<a class=\"button button-primary\" href=\"https://makersuite.google.com/app/apikey\" target=\"_blank\" rel=\"noopener noreferrer\">Get an API key</a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tHhsUxDTdw0W"
      },
      "source": [
        "In Colab, add the key to the secrets manager under the \"🔑\" in the left panel. Give it the name `API_KEY`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VmSlTHXxb5pV"
      },
      "source": [
        "Once you have the API key, pass it to the SDK. You can do this in two ways:\n",
        "\n",
        "* Put the key in the `GOOGLE_API_KEY` environment variable (the SDK will automatically pick it up from there).\n",
        "* Pass the key to `genai.configure(api_key=...)`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ab9ASynfcIZn"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    # Used to securely store your API key\n",
        "    from google.colab import userdata\n",
        "\n",
        "    # Or use `os.getenv('API_KEY')` to fetch an environment variable.\n",
        "    GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n",
        "except ImportError:\n",
        "    import os\n",
        "    GOOGLE_API_KEY = os.environ['GOOGLE_API_KEY']\n",
        "\n",
        "genai.configure(api_key=GOOGLE_API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3f383614ec30"
      },
      "source": [
        "## Function Basics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b82c1aecb657"
      },
      "source": [
        "You can pass a list of functions to the `tools` argument when creating a `genai.GenerativeModel`.\n",
        "\n",
        "> Important: The SDK converts the function's argument's type annotations to a format the API understands. The API only supports a limited selection of argument types, and this automatic conversion only supports a subset of that: `int | float | bool | str | list | dict`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "42b27b02d2f5",
        "outputId": "e3b8a644-013f-447e-cc7d-da08f641c197"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "genai.GenerativeModel(\n",
              "    model_name='models/gemini-1.0-pro',\n",
              "    generation_config={},\n",
              "    safety_settings={},\n",
              "    tools=<google.generativeai.types.content_types.FunctionLibrary object at 0x10e73fe90>,\n",
              ")"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def multiply(a:float, b:float):\n",
        "    \"\"\"returns a * b.\"\"\"\n",
        "    return a*b\n",
        "\n",
        "model = genai.GenerativeModel(model_name='gemini-1.0-pro',\n",
        "                              tools=[multiply])\n",
        "\n",
        "model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d5fd91032a1e"
      },
      "source": [
        "The recomended way to use function calling is through the chat interface. The main reason is that `FunctionCalls` fit nicely into chat's multi-turn structure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d3b91c855257"
      },
      "outputs": [],
      "source": [
        "chat = model.start_chat(enable_automatic_function_calling=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1481a6159399"
      },
      "source": [
        "With automatic function calling enabled `chat.send_message` automatically calls your function if the model asks it to.\n",
        "\n",
        "It appears to simply return a text response, containing the correct answer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "81d8def3d865",
        "outputId": "989c992c-4f7e-438f-b10b-6602bd7ba202"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'The total number of mittens is 2508.'"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response = chat.send_message('I have 57 cats, each owns 44 mittens, how many mittens is that in total?')\n",
        "response.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "951c0f83f72e",
        "outputId": "faf96a83-5959-4154-e006-4b096ae6fdb3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "2508"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "57*44"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7731e35f2383"
      },
      "source": [
        "If you look in the `ChatSession.history` you can see the sequence of events:\n",
        "\n",
        "1. You sent the question.\n",
        "2. The model replied with a `glm.FunctionCall`.\n",
        "3. The `genai.ChatSession` executed the function locally and sent the model back a `glm.FunctionResponse`.\n",
        "4. The model used the function output in its answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9f7eff1e8e60",
        "outputId": "a9cb45c9-7b1f-49bc-cfce-6c96fd7730ae"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "user -> {'text': 'I have 57 cats, each owns 44 mittens, how many mittens is that in total?'}\n",
            "--------------------------------------------------------------------------------\n",
            "model -> {'function_call': {'name': 'multiply', 'args': {'a': 57.0, 'b': 44.0}}}\n",
            "--------------------------------------------------------------------------------\n",
            "user -> {'function_response': {'name': 'multiply', 'response': {'result': 2508.0}}}\n",
            "--------------------------------------------------------------------------------\n",
            "model -> {'text': 'The total number of mittens is 2508.'}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "for content in chat.history:\n",
        "    part = content.parts[0]\n",
        "    print(content.role, \"->\", type(part).to_dict(part))\n",
        "    print('-'*80)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2471fd72f05e"
      },
      "source": [
        "In general the state diagram is:\n",
        "\n",
        "<img src=\"https://ai.google.dev/images/tutorials/function_call_state_diagram.png\" alt=\"The model can always reply with text, or a FunctionCall. Iff the model sends a FunctionCall the user must reply with a FunctionResponse\" width=50%>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f42d69800cff"
      },
      "source": [
        "The model can respond with multiple function calls before returning a text response, and function calls come before the text response."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9610f3465a69"
      },
      "source": [
        "While this was all handled automatically, if you need more control, you can:\n",
        "\n",
        "- Leave the default `enable_automatic_function_calling=False` and process the `glm.FunctionCall` responses yourself.\n",
        "- Or use `GenerativeModel.generate_content`, where you also need to manage the chat history."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JFz04WEgOwWp"
      },
      "source": [
        "## [Optional] Low level access"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Js4Y4mO20txL"
      },
      "source": [
        "The automatic extraction of the schema from python functions doesn't work in all cases. For example: it doesn't handle cases where you describe the fields of a nested dictionary-object, but the API does support this. The API is able to describe any of the follwing types:\n",
        "\n",
        "```\n",
        "AllowedType = (int | float | bool | str | list['AllowedType'] | dict[str, AllowedType]\n",
        "```\n",
        "\n",
        "The `google.ai.generativelanguage` client library provides access to the low level types giving you full control."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S53E0EE8TBUF"
      },
      "outputs": [],
      "source": [
        "import google.ai.generativelanguage as glm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4f73eef235e"
      },
      "source": [
        "First peek inside the model's `_tools` attribute, you can see how it describes the function(s) you passed it to the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e36166b2c1b6",
        "outputId": "a090698e-83e6-47a0-882e-ed57fbbb98e5"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[function_declarations {\n",
              "   name: \"multiply\"\n",
              "   description: \"returns a * b.\"\n",
              "   parameters {\n",
              "     type_: OBJECT\n",
              "     properties {\n",
              "       key: \"b\"\n",
              "       value {\n",
              "         type_: NUMBER\n",
              "       }\n",
              "     }\n",
              "     properties {\n",
              "       key: \"a\"\n",
              "       value {\n",
              "         type_: NUMBER\n",
              "       }\n",
              "     }\n",
              "     required: \"a\"\n",
              "     required: \"b\"\n",
              "   }\n",
              " }]"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def multiply(a:float, b:float):\n",
        "    \"\"\"returns a * b.\"\"\"\n",
        "    return a*b\n",
        "\n",
        "model = genai.GenerativeModel(model_name='gemini-1.0-pro',\n",
        "                             tools=[multiply])\n",
        "\n",
        "model._tools.to_proto()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFD4U7ym04F5"
      },
      "source": [
        "This returns the list of `glm.Tool` objects that would be sent to the API. If the printed format is not familiar, it's because these are Google protobuf classes. Each `glm.Tool` (1 in this case) contains a list of `glm.FunctionDeclarations`, which describe a function and its arguments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eY6RmFQ76FVu"
      },
      "source": [
        "Here is a declaration for the same multiply function written using the `glm` classes.\n",
        "\n",
        "Note that these classes just describe the function for the API, they don't include an implementation of it. So using this doesn't work with automatic function calling, but functions don't always need an implementation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qCwHM4WbC4wb"
      },
      "outputs": [],
      "source": [
        "calculator = glm.Tool(\n",
        "    function_declarations=[\n",
        "      glm.FunctionDeclaration(\n",
        "        name='multiply',\n",
        "        description=\"Returns the product of two numbers.\",\n",
        "        parameters=glm.Schema(\n",
        "            type=glm.Type.OBJECT,\n",
        "            properties={\n",
        "                'a':glm.Schema(type=glm.Type.NUMBER),\n",
        "                'b':glm.Schema(type=glm.Type.NUMBER)\n",
        "            },\n",
        "            required=['a','b']\n",
        "        )\n",
        "      )\n",
        "    ])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "19ad564235a6"
      },
      "source": [
        "Equivalently, you can describe this as a JSON-compatible object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5f2804046c94"
      },
      "outputs": [],
      "source": [
        "calculator = {'function_declarations': [\n",
        "      {'name': 'multiply',\n",
        "       'description': 'Returns the product of two numbers.',\n",
        "       'parameters': {'type_': 'OBJECT',\n",
        "       'properties': {\n",
        "         'a': {'type_': 'NUMBER'},\n",
        "         'b': {'type_': 'NUMBER'}},\n",
        "       'required': ['a', 'b']}}]}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4cefe2c3c808",
        "outputId": "30a26e44-31b8-4009-bcf2-bef90c7f0f8b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "function_declarations {\n",
              "  name: \"multiply\"\n",
              "  description: \"Returns the product of two numbers.\"\n",
              "  parameters {\n",
              "    type_: OBJECT\n",
              "    properties {\n",
              "      key: \"b\"\n",
              "      value {\n",
              "        type_: NUMBER\n",
              "      }\n",
              "    }\n",
              "    properties {\n",
              "      key: \"a\"\n",
              "      value {\n",
              "        type_: NUMBER\n",
              "      }\n",
              "    }\n",
              "    required: \"a\"\n",
              "    required: \"b\"\n",
              "  }\n",
              "}"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "glm.Tool(calculator)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jS6ruiTp6VBf"
      },
      "source": [
        "Either way, you pass a representation of a `glm.Tool` or list of tools to"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xwhWG22cIIDU"
      },
      "outputs": [],
      "source": [
        "model = genai.GenerativeModel('gemini-pro', tools=calculator)\n",
        "chat = model.start_chat()\n",
        "\n",
        "response = chat.send_message(\n",
        "    f\"What's 234551 X 325552 ?\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "517ca06297bb"
      },
      "source": [
        "Like before the model returns a `glm.FunctionCall` invoking the calculator's `multiply` function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xhey4QA0DTJf",
        "outputId": "aac45bf3-5187-4216-9aa9-684937390652"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[index: 0\n",
              "content {\n",
              "  parts {\n",
              "    function_call {\n",
              "      name: \"multiply\"\n",
              "      args {\n",
              "        fields {\n",
              "          key: \"b\"\n",
              "          value {\n",
              "            number_value: 325552\n",
              "          }\n",
              "        }\n",
              "        fields {\n",
              "          key: \"a\"\n",
              "          value {\n",
              "            number_value: 234551\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "    }\n",
              "  }\n",
              "  role: \"model\"\n",
              "}\n",
              "finish_reason: STOP\n",
              "]"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response.candidates"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07eecbaedd5e"
      },
      "source": [
        "Execute the function yourself:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "88758eebfd5c",
        "outputId": "8bd8cdcb-58d1-4fba-9152-be124d9faf74"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "76358547152.0"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "fc = response.candidates[0].content.parts[0].function_call\n",
        "assert fc.name == 'multiply'\n",
        "\n",
        "result = fc.args['a'] * fc.args['b']\n",
        "result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6ef0e9651cf"
      },
      "source": [
        "Send the result to the model, to continue the conversation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f3c67066411e"
      },
      "outputs": [],
      "source": [
        "response = chat.send_message(\n",
        "    glm.Content(\n",
        "    parts=[glm.Part(\n",
        "        function_response = glm.FunctionResponse(\n",
        "          name='multiply',\n",
        "          response={'result': result}))]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b7c032834f41"
      },
      "source": [
        "## Summary\n",
        "\n",
        "Basic function calling is supported in the SDK. Remember that it is easier to manage using chat-mode, because of the natural back and forth structure. You're in charge of actually calling the functions and sending results back to the model so it can produce a text-response."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "function_calling_python_quickstart.ipynb",
      "toc_visible": true,
      "provenance": []
    },
    "google": {
      "image_path": "/site-assets/images/share.png",
      "keywords": [
        "examples",
        "googleai",
        "samplecode",
        "python",
        "embed",
        "function"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}