notebooks/012_post_inference_hooks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a4c773be-4b40-4344-95e2-2fcbcb8d3119",
   "metadata": {},
   "source": [
    "# Post inference hooks for inference model data collection\n",
    "In this notebook we will have a look at how to set up post inference hooks for your inference models. The Geti SDK provides several basic triggers and actions that can be used to construct pipelines for, for instance, data collection, alerting, or other actions that need to take place based on inference results. \n",
    "\n",
    "These pipelines are referred to as `post inference hooks` and can be added to any `Deployment` for any project. In this notebook we will show how to configure them, and use them with existing deployments.\n",
    "\n",
    "To start off, we will create a post inference hook that implements the following behaviour:\n",
    "\n",
    "*For every inferred frame or image, check if the prediction contains any objects labelled `dog`. If it contains at least 1 dog, we want to collect it and send the image to the Geti server. The image will be stored in a new dataset called `Inferred images`, within the original project.*"
   ]
  },
  {
   "cell_type": "code",
   "id": "a844097d-e4f8-4c99-ae85-a01c77f91395",
   "metadata": {},
   "source": [
    "# As usual we will connect to the platform first, using the server details from the .env file\n",
    "\n",
    "from geti_sdk import Geti\n",
    "from geti_sdk.utils import get_server_details_from_env\n",
    "\n",
    "geti_server_configuration = get_server_details_from_env()\n",
    "\n",
    "geti = Geti(server_config=geti_server_configuration)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "a8bf7d8d-d841-4f15-98f3-015c6306c708",
   "metadata": {},
   "source": [
    "## Selecting a project\n",
    "\n",
    "we'll use the `COCO animal detection demo` project that we created in [notebook 002](002_create_project_from_dataset.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "id": "1584f03e-4e35-4d24-9a43-e0bbdf0fb78a",
   "metadata": {},
   "source": [
    "from geti_sdk.demos import ensure_trained_example_project\n",
    "\n",
    "PROJECT_NAME = \"COCO animal detection demo\"\n",
    "project = ensure_trained_example_project(geti=geti, project_name=PROJECT_NAME)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "c55f0489-af9a-4659-977f-391e1ccd1733",
   "metadata": {},
   "source": [
    "## Create deployment for the project"
   ]
  },
  {
   "cell_type": "code",
   "id": "2e2ae82f-bc66-4863-b423-82347737f810",
   "metadata": {},
   "source": [
    "deployment = geti.deploy_project(project=project)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "0b4ffa71-fb63-4214-a308-898e9662934e",
   "metadata": {},
   "source": [
    "## Checking deployment output\n",
    "Let's quickly load the inference models and check the inference output on a sample image"
   ]
  },
  {
   "cell_type": "code",
   "id": "6d2788bc-fd8d-4363-85b0-cfcf00825ef7",
   "metadata": {},
   "source": [
    "deployment.load_inference_models()"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "d26688a7-9e2d-4443-acb8-cd90798da662",
   "metadata": {},
   "source": [
    "import cv2\n",
    "\n",
    "from geti_sdk import Visualizer\n",
    "from geti_sdk.demos import EXAMPLE_IMAGE_PATH\n",
    "\n",
    "numpy_image = cv2.imread(EXAMPLE_IMAGE_PATH)\n",
    "numpy_rgb = cv2.cvtColor(numpy_image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "prediction = deployment.infer(numpy_rgb)\n",
    "\n",
    "visualizer = Visualizer()\n",
    "result = visualizer.draw(numpy_rgb, prediction)\n",
    "visualizer.show_in_notebook(result)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "983c3c09-06c4-4642-b4b2-394a3dcecdbf",
   "metadata": {},
   "source": [
    "## Configuring a post inference hook to send image data to Geti\n",
    "\n",
    "With the deployment all set up and ready, let's go ahead and add a post inference hook! We will configure it to behave as follows:\n",
    "\n",
    "For each inferred image or frame:\n",
    "\n",
    "- If and only if the prediction contains at least one object labelled `dog`:\n",
    "- Send the image to the Geti project, to a dedicated dataset named 'Inferred images'\n",
    "\n",
    "Basically, this behaviour can be separated into two parts: A **Trigger** and an **Action**. The first part, in which we check if the prediction contains at least one dog, is the Trigger. If the trigger activates, the Action will be carried out: Sending the data to the Intel Geti server. \n",
    "\n",
    "The reasoning here is that if the prediction contains a dog, we want to collect the image in our animal detection project so that we can include it in the next training round. To achieve this, we will use the `LabelTrigger`: It will activate if the prediction contains any objects labelled `dog`. \n",
    "\n",
    "Of course, many other triggers can be defined: For example, the `ObjectCountTrigger` can be used to activate only when a prediction contains a certain number of objects, the `EmptyLabelTrigger` will activate when the prediction does not contain any objects and the `ConfidenceTrigger` will activate when the probability for any of the predictions is below a certain threshold. \n",
    "\n",
    "The cell below shows how to define the hook outlined above. "
   ]
  },
  {
   "cell_type": "code",
   "id": "302568fe-0d1a-4333-863c-a99f2a94c9b9",
   "metadata": {},
   "source": [
    "from geti_sdk.post_inference_hooks import (\n",
    "    GetiDataCollection,\n",
    "    LabelTrigger,\n",
    "    PostInferenceHook,\n",
    ")\n",
    "\n",
    "trigger = LabelTrigger(\n",
    "    label_names=[\"dog\"]\n",
    ")  # the Trigger will activate whenever a prediction contains any object labelled `dog`\n",
    "\n",
    "action = GetiDataCollection(  # the Action will send data to a new `Inferred images` dataset in the Geti project\n",
    "    session=geti.session,\n",
    "    workspace_id=geti.workspace_id,\n",
    "    project=project,\n",
    "    dataset=\"Inferred images\",\n",
    "    log_level=\"info\",\n",
    ")\n",
    "\n",
    "hook = PostInferenceHook(  # The Hook attaches the action to the trigger\n",
    "    trigger=trigger, action=action\n",
    ")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "576322e5-1440-412b-899e-6d8bfb8aa6e2",
   "metadata": {},
   "source": [
    "Now, we just need to add the hook to the deployment"
   ]
  },
  {
   "cell_type": "code",
   "id": "edaee43a-a8bc-49ee-a6e8-592a9efe940a",
   "metadata": {},
   "source": [
    "deployment.add_post_inference_hook(hook)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "3a7354f6-57e3-48a1-bffa-123e07081fec",
   "metadata": {},
   "source": [
    "Once added, whenever we run inference on an image or video frame, the hook will execute automatically."
   ]
  },
  {
   "cell_type": "code",
   "id": "c6f9b972-3411-4345-8951-5f2c9dbcf66c",
   "metadata": {},
   "source": [
    "prediction = deployment.infer(numpy_rgb)\n",
    "print(f\"Prediction contains objects with labels: {prediction.get_label_names()}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "47bcb165-d4e8-4210-b9b2-83b662ded568",
   "metadata": {},
   "source": [
    "From the cell above, you should get a printout with the list of labels in the prediction. If the label `dog` is among them, you should also see a log line stating that the image was uploaded to the Geti project"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0740ef-de27-443d-8521-9b7aa70716fe",
   "metadata": {},
   "source": [
    "## Adding multiple hooks\n",
    "\n",
    "We can add as many hooks as we like, each with different triggers and actions. Suppose we are primarily interested in images with dogs in them, for some reason. At the same time, we know that we are feeding our model images containing dogs, so any prediction that does not contain any dog-objects is suspicious. Those images might need to be added to the training set, in order to improve the model. So we want to sort the inferred images into a 'dogs' and a 'no dogs' category.\n",
    "\n",
    "In the next cell, we'll create two hooks to achieve both these goals and add them to the deployment.\n",
    "The hooks we'll create are the following:\n",
    "\n",
    "**The 'dogs' hook**:\n",
    "- Checks if the predictions contain 1 or more `dog`s.  \n",
    "- If so, then:\n",
    "- Save the image, the prediction and the score that triggered the action to a folder `dogs` on disk. In this case, the score is the number of predicted dogs\n",
    "\n",
    "**The 'no dogs' hook**\n",
    "- Checks if the predictions do not contain any dogs\n",
    "- If so, send the image to the Geti server, to a separate dataset called `Inferred images - no dogs`"
   ]
  },
  {
   "cell_type": "code",
   "id": "7be449fe-26a0-466b-92f7-78d4477ac255",
   "metadata": {},
   "source": [
    "from geti_sdk.post_inference_hooks import FileSystemDataCollection, ObjectCountTrigger\n",
    "\n",
    "NUMBER_OF_THREADS_PER_HOOK = 10\n",
    "\n",
    "# First, remove any hooks that were added previously\n",
    "deployment.clear_inference_hooks()\n",
    "\n",
    "# Create the 'dogs' trigger, action and hook\n",
    "dogs_trigger = ObjectCountTrigger(\n",
    "    threshold=0, label_names=[\"dog\"], mode=\"greater\"\n",
    ")  # Trigger will activate whenever a prediction contains one or more objects labelled 'dog'\n",
    "\n",
    "dogs_action = FileSystemDataCollection(\n",
    "    target_folder=\"hook_data/dogs\",\n",
    "    file_name_prefix=\"image\",\n",
    "    save_predictions=True,\n",
    "    save_scores=True,\n",
    "    save_overlays=True,\n",
    "    log_level=\"debug\",\n",
    ")  # Action will store the image, prediction data, trigger score and the images with prediction overlays to the `dogs` folder on disk\n",
    "\n",
    "dogs_hook = PostInferenceHook(\n",
    "    trigger=dogs_trigger, action=dogs_action, max_threads=NUMBER_OF_THREADS_PER_HOOK\n",
    ")\n",
    "\n",
    "# Create the 'no_dogs' trigger, action and hook\n",
    "no_dogs_trigger = ObjectCountTrigger(\n",
    "    threshold=1, label_names=[\"dog\"], mode=\"lower\"\n",
    ")  # Trigger will activate whenever a prediction does not contain any objects labelled 'dog'\n",
    "\n",
    "no_dogs_action = GetiDataCollection(  # the Action will send data to a new `Inferred images - no dogs` dataset in the Geti project\n",
    "    session=geti.session,\n",
    "    workspace_id=geti.workspace_id,\n",
    "    project=project,\n",
    "    dataset=\"Inferred images - no dogs\",\n",
    ")\n",
    "\n",
    "no_dogs_hook = PostInferenceHook(\n",
    "    trigger=no_dogs_trigger,\n",
    "    action=no_dogs_action,\n",
    "    max_threads=NUMBER_OF_THREADS_PER_HOOK,\n",
    ")\n",
    "\n",
    "# Add both hooks to the deployment\n",
    "deployment.add_post_inference_hook(dogs_hook)\n",
    "deployment.add_post_inference_hook(no_dogs_hook)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "2f5c7bb9-c95e-4ec7-9047-bc753c330e71",
   "metadata": {},
   "source": [
    "Now that the hooks are created and added to the deployment, we can run the inference again.\n",
    "\n",
    "We will run it on 50 images from the COCO dataset. The images are selected such that each of them contains at least one dog. \n",
    "\n",
    "In the cell below, we first get a list of filepaths to images with `dog`s in them"
   ]
  },
  {
   "cell_type": "code",
   "id": "4065cb88-f208-45f4-bc8e-ee778e0a6b34",
   "metadata": {},
   "source": [
    "import os\n",
    "\n",
    "from geti_sdk.annotation_readers import DatumAnnotationReader\n",
    "from geti_sdk.demos import get_coco_dataset\n",
    "\n",
    "n_images = 50\n",
    "\n",
    "path = get_coco_dataset()\n",
    "ar = DatumAnnotationReader(path, annotation_format=\"coco\")\n",
    "ar.filter_dataset(labels=[\"dog\"])\n",
    "dog_image_filenames = ar.get_all_image_names()\n",
    "dog_image_filepaths = [\n",
    "    os.path.join(path, \"images\", \"val2017\", fn + \".jpg\") for fn in dog_image_filenames\n",
    "][0:n_images]\n",
    "print(f\"Selected the first {n_images} images containing dogs from COCO dataset\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "f07d88db-5c13-428d-b56a-0e0ad2d7e86e",
   "metadata": {},
   "source": [
    "Now, we can run inference on the images and measure the time required in the cell below"
   ]
  },
  {
   "cell_type": "code",
   "id": "c4fa2040-09e8-40db-9278-558c03781e3d",
   "metadata": {},
   "source": [
    "import time\n",
    "\n",
    "from tqdm import tqdm\n",
    "\n",
    "t_start = time.time()\n",
    "for image_path in tqdm(dog_image_filepaths):\n",
    "    image = cv2.imread(image_path)\n",
    "    numpy_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "    deployment.infer(numpy_rgb)\n",
    "t_elapsed = time.time() - t_start\n",
    "print(\n",
    "    f\"Inference on {n_images} images with 2 post-inference hooks completed in {t_elapsed:.2f} seconds.\"\n",
    ")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "99a6136a-06c3-4233-8986-9ead9f69c43f",
   "metadata": {},
   "source": [
    "You should now see a new folder `hook_data` in your working directory. Inside this folder, you'll find a folder titled `lots_of_dogs` and one named `no_dogs`. The `lots_of_dogs` folder contains four subfolders: `images`, `predictions`, `scores` and `overlays`. The contents of these folders are the following:\n",
    "- `images` contains the image files which triggered the hook\n",
    "- `predictions` contains the prediction data in .json format\n",
    "- `scores` contains txt files with the score for each image that caused the hook to trigger\n",
    "- `overlays` contains the images with the predictions visualized on top of them. This can be useful for checking the output visually.\n",
    "\n",
    "The file names are consistent across the subfolders, i.e. the prediction for a certain image can be found in the .json file with the same name, in the `predictions` folder.\n",
    "\n",
    "The `no_dogs` folder only contains `images` and `overlays`, because we configured the action with `save_predictions=False` and `save_scores=False`.\n",
    "\n",
    "If you take a look in those folders now, you'll find that they are populated with images, predictions, score files and overlay images."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b239dbae-8016-4385-b9b6-cb7e6e87b5ed",
   "metadata": {},
   "source": [
    "### What about overhead?\n",
    "\n",
    "Because post inference hooks are executed in seperate threads, adding them to your deployment will add minimal overhead to the inference process. Let's clear the hooks and measure the inference time again, to get an estimate of the impact."
   ]
  },
  {
   "cell_type": "code",
   "id": "e8e5992a-984e-4ccc-84f8-441b150d467d",
   "metadata": {},
   "source": [
    "# Remove any post-inference hooks\n",
    "deployment.clear_inference_hooks()\n",
    "\n",
    "# Now run the inference loop without any hooks\n",
    "t_start = time.time()\n",
    "for image_path in tqdm(dog_image_filepaths):\n",
    "    image = cv2.imread(image_path)\n",
    "    numpy_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "    deployment.infer(numpy_rgb)\n",
    "t_elapsed = time.time() - t_start\n",
    "print(\n",
    "    f\"Inference on {n_images} images without post-inference hooks completed in {t_elapsed:.2f} seconds.\"\n",
    ")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "f06c9787-80d2-40c4-a88e-beb60f10629d",
   "metadata": {},
   "source": [
    "Most likely you will notice that the inference time without any hooks is less than with the 2 hooks applied. Nevertheless, the additional time required is much smaller than if you would carry out the actions defined in the post-inference hooks after each inferred image in a synchronous manner."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "077775ac-fbea-4191-8346-df5154961faa",
   "metadata": {},
   "source": [
    "## Saving a deployment with post inference hooks\n",
    "If you save a deployment with post inference hooks, the hook configuration will be saved with it. The cell below shows how to do this."
   ]
  },
  {
   "cell_type": "code",
   "id": "4054c0d3-c6e3-40cf-baf1-a698d2b040bb",
   "metadata": {},
   "source": [
    "target_folder = os.path.join(\"deployments\", PROJECT_NAME)\n",
    "deployment.save(target_folder);"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "c07cb08c-77d1-4b37-a3b5-58fc3f1ef84b",
   "metadata": {},
   "source": [
    "Once saved, the deployment can be recreated and the post inference hooks will be added automatically. Upon executing the cell below, you should see the two post inference hooks being added to the deployment."
   ]
  },
  {
   "cell_type": "code",
   "id": "1f63c0e4-2805-4c5e-b7f6-9ef27d3699a8",
   "metadata": {},
   "source": [
    "from geti_sdk.deployment import Deployment\n",
    "\n",
    "offline_deployment = Deployment.from_folder(target_folder)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "ee0bfcfd-6065-47ea-baf6-c85f12578ddf",
   "metadata": {},
   "source": [
    "## Limiting hook execution rate\n",
    "\n",
    "Suppose that we are running inference on a video stream. In that case, we might get many sequential frames which activate a hook trigger, because frames that appear shortly after one another may look very similar. To avoid filling up our data collection folder with such near-duplicate frames, we can choose to limit the rate at which an action is allowed to run. This can be configured in the `PostInferenceHook`constructor, using the `limit_action_rate` and `max_frames_per_second` parameters.\n",
    "\n",
    "To give an example of this, in the last demo of this notebook we'll run inference 50 times *on the same image*, to simulate a video stream. We'll create a hook with the `AlwaysTrigger`, which activates after every inferred image or frame, and have it send the data to Geti using the `GetiDataCollection` action. However, to avoid filling up our dataset with 50 duplicate images, we'll limit the action execution rate to 1 frame per second.\n",
    "\n",
    "The cell below shows how to create this hook."
   ]
  },
  {
   "cell_type": "code",
   "id": "381fa7c3-72ce-47c0-8039-239629e46ee1",
   "metadata": {},
   "source": [
    "from geti_sdk.post_inference_hooks import AlwaysTrigger\n",
    "\n",
    "trigger = AlwaysTrigger()\n",
    "action = GetiDataCollection(\n",
    "    session=geti.session,\n",
    "    workspace_id=geti.workspace_id,\n",
    "    project=project,\n",
    "    dataset=\"Inferred video frames\",\n",
    "    log_level=\"debug\",\n",
    ")\n",
    "geti_hook = PostInferenceHook(\n",
    "    trigger=trigger,\n",
    "    action=action,\n",
    "    max_threads=5,\n",
    "    limit_action_rate=True,\n",
    "    max_frames_per_second=1,\n",
    ")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "18b239ff-e8a1-48b0-a256-ff6190f4a9af",
   "metadata": {},
   "source": [
    "Let's first clear the existing hooks, and then add our new hook to the deployment"
   ]
  },
  {
   "cell_type": "code",
   "id": "329cfc88-5fc2-4c3c-8e66-b6e85d14c608",
   "metadata": {},
   "source": [
    "offline_deployment.clear_inference_hooks()\n",
    "offline_deployment.add_post_inference_hook(geti_hook)\n",
    "offline_deployment.load_inference_models()"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "c14e6570-f67c-4b72-8d8d-e5b1031ecd3d",
   "metadata": {},
   "source": [
    "Now, let's run inference 50 times again, each time on the same image"
   ]
  },
  {
   "cell_type": "code",
   "id": "0c86d82f-c990-486f-981e-2b11347f255f",
   "metadata": {},
   "source": [
    "image = cv2.imread(EXAMPLE_IMAGE_PATH)\n",
    "numpy_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "t_start = time.time()\n",
    "for ind in tqdm(range(50)):\n",
    "    offline_deployment.infer(numpy_rgb)\n",
    "\n",
    "t_elapsed = time.time() - t_start\n",
    "print(\n",
    "    f\"50 inference iterations with rate-limited Geti I/O hook completed in {t_elapsed:.2f} seconds.\"\n",
    ")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "d523e182-ec0a-4f5e-bf0c-09e2ad08f378",
   "metadata": {},
   "source": [
    "Your Geti project should now contain a new dataset called `Inferred video frames`, which should contain as many images as the number of seconds the benchmark took to run (plus one, because the action fires immediately on the first frame). So if it took 8 seconds to infer 50 times, the hook should have uploaded 9 images to Geti.\n",
    "\n",
    "Note that the trigger that we use is the `AlwaysTrigger`, which activates on every inferred image or video frame, regardless of the prediction outcome. The rate limiting happens in the `Action` phase of the hook, it ensures that the action does not run more frequently than allowed by the rate limit, even if the trigger fires much more often."
   ]
  },
  {
   "cell_type": "code",
   "id": "852cf6f3-9765-4d61-ab5c-31e79bc7e73d",
   "metadata": {},
   "source": [],
   "outputs": [],
   "execution_count": null
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}