recipes/inference_labels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kili Tutorial: Importing inference labels\n",
    "\n",
    "In this tutorial, we will walk through the process of using Kili to evaluate the performance of a machine learning model in production. The goal of this tutorial is to illustrate how to push such labels, and how to visualize the quality of those predicted labels.\n",
    "\n",
    "Additionally:\n",
    "\n",
    "For an overview of Kili, visit https://kili-technology.com. You can also check out the Kili documentation https://cloud.kili-technology.com/docs.\n",
    "\n",
    "The tutorial is divided into two parts:\n",
    "\n",
    "1. Giving a bit of context\n",
    "2. How to make use of inference labels in practice\n",
    "\n",
    "This next cell connects the notebook to the Kili API. You need to update the credentials `api_key` before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/typeguard/__init__.py:917: UserWarning: no type annotations present -- not typechecking kili.queries.lock.QueriesLock.count_locks\n",
      "  warn('no type annotations present -- not typechecking {}'.format(function_name(func)))\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "#!pip install kili\n",
    "from kili.client import Kili\n",
    "\n",
    "api_key = os.getenv('KILI_USER_API_KEY')\n",
    "api_endpoint = os.getenv('KILI_API_ENDPOINT')\n",
    "# If you use Kili SaaS, use the url 'https://cloud.kili-technology.com/api/label/v2/graphql'\n",
    "\n",
    "kili = Kili(api_key=api_key, api_endpoint=api_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Agreement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say you have a trained machine learning model $m$, which can, given a data $x$, output a prediction (ie, an inference label) $l^i = m(x)$.\n",
    "\n",
    "What you will probably want to do is monitor the quality of such predictions, as the model evolves. Kili allows you to better monitor and iterate on your model, thanks to the concept of agreement. An agreement is a quantitative measure of similarity between two different labels. In Kili, there are three main features derived from agreement : \n",
    "\n",
    "- [Consensus](https://cloud.kili-technology.com/docs/quality/consensus/#docsNav), which is the agreement between two labelers.\n",
    "- [Honeypot](https://cloud.kili-technology.com/docs/quality/honeypot/) which is the agreement between a \"super human annotator\" and a labeler.\n",
    "- **Inference**, which is the agreement between a machine learning inference label and a human.\n",
    "\n",
    "Those number can be monitored from the interface as the [quality KPIs](https://cloud.kili-technology.com/docs/quality/quality-kpis/). You can find how the agreement is computed [here](https://cloud.kili-technology.com/docs/quality/consensus/#consensus-calculation)\n",
    "\n",
    "In this tutorial, we will put an emphasis on **Inference**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Use cases "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We identify two main use cases for the use of **inference** :\n",
    "\n",
    "1. **You have a model in production**. When it receives assets, it automatically feeds a Kili project with both the asset and the predicted label. **You also have human workforce, whose job is to monitor the quality of the model**. They regularly manually label some data seen by the model.\n",
    "    - When a human submits a label, the inference score for that label is automatically computed using the predicted label.\n",
    "    - Low inference scores can indicate either a model performing badly on some kind of data, or a disagreement between humans and the model. This can help you to :\n",
    "    \n",
    "        - `Detect data drift`\n",
    "        - `Identify data on which the model needs improvement`\n",
    "       \n",
    "       \n",
    "2. **You used Kili to label data**, and you have **the first iteration of your model**. You can use **a part of the dataset as testing data**, and quickly get **test scores**. You could of course use your own metrics (rather than our own definition of agreement), but using Kili allows you to quickly filter and indentify the assets where your model is most different from the ground truth.\n",
    "    - When you push an inference label on an asset, the inference score is automatically computed for all most recent labels of the different people who labeled this asset.\n",
    "    - You can filter on low inference score, to understand why your model is failing, and how to fix it (getting more data, splitting or merging categories, etc...)\n",
    "\n",
    "\n",
    "Using Kili for monitoring or developing your model allows you to quickly iterate on the data used to train your model, allowing to get a better model faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - In practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Use case 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by creating a project and defining a model which, given an asset input x, returns a category (random in our example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "random.seed(10)\n",
    "\n",
    "json_interface ={\n",
    "    \"jobs\": {\n",
    "        \"CLASSIFICATION_JOB\": {\n",
    "            \"mlTask\": \"CLASSIFICATION\",\n",
    "            \"content\": {\n",
    "                \"categories\": {\n",
    "                    \"RED\": {\"name\": \"Red\"},\n",
    "                    \"BLACK\": {\"name\": \"Black\"},\n",
    "                    \"WHITE\": {\"name\": \"White\"},\n",
    "                    \"GREY\": {\"name\": \"Grey\"}\n",
    "                },\n",
    "                \"input\": \"radio\"\n",
    "            },\n",
    "            \"required\": 0,\n",
    "            \"isChild\": False,\n",
    "            \"instruction\": \"Color\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "project_id = kili.create_project(\n",
    "    title='Project demo inference',\n",
    "    input_type='IMAGE',\n",
    "    json_interface=json_interface\n",
    ")['id']\n",
    "\n",
    "categories = ['RED', 'BLACK', 'WHITE', 'GREY']\n",
    "\n",
    "\n",
    "def machine_learning_model(x):\n",
    "    return random.choice(categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can simulate that our model is in production. Each time it receives an asset, it pushes it as well as the label it predicted to the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "stream_of_assets = [\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/black_car.jpg\",\n",
    "        'external_id': 'black_car.jpg'\n",
    "    },\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/grey_car.jpg\",\n",
    "        'external_id': 'grey_car.jpg'\n",
    "    },\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/white_car.jpg\",\n",
    "        'external_id': 'white_car.jpg'\n",
    "    },\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/red_car.jpg\",\n",
    "        'external_id': 'red_car.jpg'\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00, 18.31it/s]\n",
      "100%|██████████| 2/2 [00:00<00:00, 30.09it/s]\n",
      "100%|██████████| 3/3 [00:00<00:00, 71.20it/s]\n",
      "100%|██████████| 4/4 [00:00<00:00, 69.09it/s]\n"
     ]
    }
   ],
   "source": [
    "for i, asset in enumerate(stream_of_assets):\n",
    "    kili.append_many_to_dataset(\n",
    "        project_id=project_id,\n",
    "        content_array=[asset['url']],\n",
    "        external_id_array=[asset['external_id']]\n",
    "    )\n",
    "    asset_id = kili.assets(project_id=project_id, fields=['id'])[i]['id']\n",
    "    predicted_category = machine_learning_model(asset)\n",
    "    inference_label = {\n",
    "        \"CLASSIFICATION_JOB\": {\n",
    "            \"categories\": [{\"name\": predicted_category}]\n",
    "        }\n",
    "    }\n",
    "    kili.append_to_labels(\n",
    "        json_response=inference_label,\n",
    "        label_asset_id=asset_id,\n",
    "        label_type='INFERENCE',\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, human labelers can annotate a subsample of the assets pushed to Kili. \n",
    "\n",
    "Note : you can even automatically [prioritize assets](https://cloud.kili-technology.com/docs/python-graphql-api/recipes/import_assets/#setting-up-a-labeling-priority) to be reviewed by a human by using the model's uncertainty. When the model is unsure of its predictions, it may indicate wrong labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:00<00:00, 47.98it/s]\n"
     ]
    }
   ],
   "source": [
    "assets = kili.assets(project_id=project_id, fields=['id'])\n",
    "ground_truths = ['BLACK', 'GREY', 'WHITE', 'RED']\n",
    "\n",
    "for i, asset in enumerate(assets):\n",
    "    human_label = {\n",
    "        \"CLASSIFICATION_JOB\": {\n",
    "            \"categories\": [{\"name\": ground_truths[i]}]\n",
    "        }\n",
    "    }\n",
    "    kili.append_to_labels(\n",
    "        json_response=human_label,\n",
    "        label_asset_id=asset['id'],\n",
    "        label_type='DEFAULT',\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now fetch the agreement between the human and the model, for human labels :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'labelOf': {'id': 'ckpqjze9j00vsac9k897j2w73'},\n",
       "  'inferenceMark': 0.5,\n",
       "  'id': 'ckpqjzfml00xsac9k1i8scw4q'},\n",
       " {'labelOf': {'id': 'ckpqjzen000w9ac9kcxd8deo3'},\n",
       "  'inferenceMark': 1,\n",
       "  'id': 'ckpqjzfta00yaac9khdtcgxf6'},\n",
       " {'labelOf': {'id': 'ckpqjzeys00wqac9k4mdt2h7k'},\n",
       "  'inferenceMark': 0.5,\n",
       "  'id': 'ckpqjzg0b00ytac9k671ycpjw'},\n",
       " {'labelOf': {'id': 'ckpqjzf9e00x8ac9k14krfy0k'},\n",
       "  'inferenceMark': 1,\n",
       "  'id': 'ckpqjzg8x00zcac9k7bznhuyv'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = kili.labels(project_id=project_id,\n",
    "            fields=['inferenceMark', 'id', 'labelOf.id'], type_in=['DEFAULT'])\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For testing\n",
    "assert labels[0]['inferenceMark'] < 1\n",
    "assert labels[1]['inferenceMark'] == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This allows you to identify problems :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model predicted RED but the human predicted BLACK\n",
      "The model predicted GREY but the human predicted WHITE\n"
     ]
    }
   ],
   "source": [
    "for label in labels:\n",
    "    if label['inferenceMark'] < 1:\n",
    "        inference_label = kili.labels(asset_id=label['labelOf']['id'], type_in=['INFERENCE'])[0]\n",
    "        human_label = kili.labels(label_id=label['id'])[0]\n",
    "        inference_category = inference_label['jsonResponse']['CLASSIFICATION_JOB']['categories'][0]['name']\n",
    "        human_category = human_label['jsonResponse']['CLASSIFICATION_JOB']['categories'][0]['name']\n",
    "        print(f'The model predicted {inference_category} but the human predicted {human_category}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also find the assets with most disagreement directly from the interface with the filter \"Human/Model IOU\". Low IOU indicates low agreement : \n",
    "\n",
    "![inference](https://storage.googleapis.com/label-public-staging/recipes/inference/inference_filter.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Use case 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can invert the previous use case. We start by having a human labeled dataset, and we insert model predictions, to simulate testing our model on test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_id = kili.create_project(\n",
    "    title='Project demo inference 2',\n",
    "    input_type='IMAGE',\n",
    "    json_interface=json_interface\n",
    ")['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "labeled_assets = [\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/black_car.jpg\",\n",
    "        'external_id': 'black_car.jpg'\n",
    "    },\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/grey_car.jpg\",\n",
    "        'external_id': 'grey_car.jpg'\n",
    "    },\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/white_car.jpg\",\n",
    "        'external_id': 'white_car.jpg'\n",
    "    },\n",
    "    {\n",
    "        'url': \"https://storage.googleapis.com/label-public-staging/recipes/inference/red_car.jpg\",\n",
    "        'external_id': 'red_car.jpg'\n",
    "    }\n",
    "]\n",
    "ground_truths = ['BLACK', 'GREY', 'WHITE', 'RED']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00, 15.22it/s]\n",
      "100%|██████████| 2/2 [00:00<00:00, 38.46it/s]\n",
      "100%|██████████| 3/3 [00:00<00:00, 53.30it/s]\n",
      "100%|██████████| 4/4 [00:00<00:00, 79.20it/s]\n"
     ]
    }
   ],
   "source": [
    "for i, (asset, category) in enumerate(zip(labeled_assets, ground_truths)):\n",
    "    kili.append_many_to_dataset(\n",
    "        project_id=project_id,\n",
    "        content_array=[asset['url']],\n",
    "        external_id_array=[asset['external_id']]\n",
    "    )\n",
    "    asset_id = kili.assets(project_id=project_id, fields=['id'])[i]['id']\n",
    "    human_label = {\n",
    "        \"CLASSIFICATION_JOB\": {\n",
    "            \"categories\": [{\"name\": category}]\n",
    "        }\n",
    "    }\n",
    "    kili.append_to_labels(\n",
    "        json_response=human_label,\n",
    "        label_asset_id=asset_id,\n",
    "        label_type='DEFAULT',\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then our model is fit using maybe 80% of the training data. We can then run it against the remaining 20%, and upload its predictions to Kili :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:00<00:00, 65.84it/s]\n"
     ]
    }
   ],
   "source": [
    "assets = kili.assets(project_id=project_id, fields=['id'])\n",
    "\n",
    "for i, asset in enumerate(assets):\n",
    "    test_label = {\n",
    "        \"CLASSIFICATION_JOB\": {\n",
    "            \"categories\": [{\"name\": machine_learning_model(asset)}]\n",
    "        }\n",
    "    }\n",
    "    kili.append_to_labels(\n",
    "        json_response=test_label,\n",
    "        label_asset_id=asset['id'],\n",
    "        label_type='INFERENCE',\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'labelOf': {'id': 'ckpqjzgrp010aac9k3a8c6oy7'},\n",
       "  'inferenceMark': 1,\n",
       "  'id': 'ckpqjzgw2010hac9ked5609mq'},\n",
       " {'labelOf': {'id': 'ckpqjzh3k0110ac9k0j6r5eqw'},\n",
       "  'inferenceMark': 1,\n",
       "  'id': 'ckpqjzh7d0116ac9kgk9gbq8f'},\n",
       " {'labelOf': {'id': 'ckpqjzhek011pac9k6ppf3zf0'},\n",
       "  'inferenceMark': 0.5,\n",
       "  'id': 'ckpqjzhir011wac9k7y9jarir'},\n",
       " {'labelOf': {'id': 'ckpqjzhqj012gac9kgtbv58f4'},\n",
       "  'inferenceMark': 0.5,\n",
       "  'id': 'ckpqjzhue012mac9k7vof5432'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = kili.labels(project_id=project_id,\n",
    "            fields=['inferenceMark', 'id', 'labelOf.id'], type_in=['DEFAULT'])\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For testing\n",
    "assert labels[0]['inferenceMark'] == 1\n",
    "assert labels[2]['inferenceMark'] < 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The human predicted WHITE but the model predicted GREY\n",
      "The human predicted RED but the model predicted WHITE\n"
     ]
    }
   ],
   "source": [
    "for label in labels:\n",
    "    if label['inferenceMark'] < 1:\n",
    "        inference_label = kili.labels(asset_id=label['labelOf']['id'], type_in=['INFERENCE'])[0]\n",
    "        human_label = kili.labels(label_id=label['id'])[0]\n",
    "        inference_category = inference_label['jsonResponse']['CLASSIFICATION_JOB']['categories'][0]['name']\n",
    "        human_category = human_label['jsonResponse']['CLASSIFICATION_JOB']['categories'][0]['name']\n",
    "        print(f'The human predicted {human_category} but the model predicted {inference_category}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also find the assets where the prediction and the human disagree most directly from the interface with the filter \"Human/Model IOU\" : \n",
    "\n",
    "![inference](https://storage.googleapis.com/label-public-staging/recipes/inference/inference_test_filter.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we accomplished the following:\n",
    "\n",
    "We introduced the concept of Kili inference labels. We showed how to make use of such labels, in two practical use cases.\n",
    "\n",
    "You can also visit the Kili website or Kili documentation for more info!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}