examples/notebooks/multioutput_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Working with Multi-Output models\n",
    "\n",
    "In this notebook, we'll be performing multi-output tasks with the help of DFFML. DFFML makes it so that users can use a range of different models to perform multi-output tasks with the usual worklow, but instantiating models with a list of targets/labels!",
    "\n",
    "[![Video Explanation Thumbnail](http://img.youtube.com/vi/7whKBThZDN4/0.jpg)](http://www.youtube.com/watch?v=7whKBThZDN4)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Import Packages**\n",
    "\n",
    "Let us import dffml and other packages that we might need."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import asyncio\n",
    "import nest_asyncio\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_linnerud\n",
    "from sklearn.datasets import make_multilabel_classification\n",
    "\n",
    "from dffml import *"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "To use asyncio in a notebook, we need to use `nest_asycio.apply()`"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "nest_asyncio.apply()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Multi-Output Regression\n",
    "\n",
    "To perform multi-output regression tasks, we can select any scikit regression model and pass our multi-output dataset onto the model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Build our Dataset\n",
    "\n",
    "For our dataset, we will be generating a random multi-output dataset "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "X, y = load_linnerud(return_X_y=True)\n",
    "\n",
    "data = np.concatenate((X, y), axis=1)\n",
    "\n",
    "print(X.shape, y.shape)\n",
    "print(data)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "(20, 3) (20, 3)\n",
      "[[  5. 162.  60. 191.  36.  50.]\n",
      " [  2. 110.  60. 189.  37.  52.]\n",
      " [ 12. 101. 101. 193.  38.  58.]\n",
      " [ 12. 105.  37. 162.  35.  62.]\n",
      " [ 13. 155.  58. 189.  35.  46.]\n",
      " [  4. 101.  42. 182.  36.  56.]\n",
      " [  8. 101.  38. 211.  38.  56.]\n",
      " [  6. 125.  40. 167.  34.  60.]\n",
      " [ 15. 200.  40. 176.  31.  74.]\n",
      " [ 17. 251. 250. 154.  33.  56.]\n",
      " [ 17. 120.  38. 169.  34.  50.]\n",
      " [ 13. 210. 115. 166.  33.  52.]\n",
      " [ 14. 215. 105. 154.  34.  64.]\n",
      " [  1.  50.  50. 247.  46.  50.]\n",
      " [  6.  70.  31. 193.  36.  46.]\n",
      " [ 12. 210. 120. 202.  37.  62.]\n",
      " [  4.  60.  25. 176.  37.  54.]\n",
      " [ 11. 230.  80. 157.  32.  52.]\n",
      " [ 15. 225.  73. 156.  33.  54.]\n",
      " [  2. 110.  43. 138.  33.  68.]]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "data = [\n",
    "    {\"x1\": row[0], \"x2\": row[1], \"x3\": row[2], \"y1\": row[3], \"y2\": row[4], \"y3\": row[5]}\n",
    "    for row in data\n",
    "]\n",
    "data"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[{'x1': 5.0, 'x2': 162.0, 'x3': 60.0, 'y1': 191.0, 'y2': 36.0, 'y3': 50.0},\n",
       " {'x1': 2.0, 'x2': 110.0, 'x3': 60.0, 'y1': 189.0, 'y2': 37.0, 'y3': 52.0},\n",
       " {'x1': 12.0, 'x2': 101.0, 'x3': 101.0, 'y1': 193.0, 'y2': 38.0, 'y3': 58.0},\n",
       " {'x1': 12.0, 'x2': 105.0, 'x3': 37.0, 'y1': 162.0, 'y2': 35.0, 'y3': 62.0},\n",
       " {'x1': 13.0, 'x2': 155.0, 'x3': 58.0, 'y1': 189.0, 'y2': 35.0, 'y3': 46.0},\n",
       " {'x1': 4.0, 'x2': 101.0, 'x3': 42.0, 'y1': 182.0, 'y2': 36.0, 'y3': 56.0},\n",
       " {'x1': 8.0, 'x2': 101.0, 'x3': 38.0, 'y1': 211.0, 'y2': 38.0, 'y3': 56.0},\n",
       " {'x1': 6.0, 'x2': 125.0, 'x3': 40.0, 'y1': 167.0, 'y2': 34.0, 'y3': 60.0},\n",
       " {'x1': 15.0, 'x2': 200.0, 'x3': 40.0, 'y1': 176.0, 'y2': 31.0, 'y3': 74.0},\n",
       " {'x1': 17.0, 'x2': 251.0, 'x3': 250.0, 'y1': 154.0, 'y2': 33.0, 'y3': 56.0},\n",
       " {'x1': 17.0, 'x2': 120.0, 'x3': 38.0, 'y1': 169.0, 'y2': 34.0, 'y3': 50.0},\n",
       " {'x1': 13.0, 'x2': 210.0, 'x3': 115.0, 'y1': 166.0, 'y2': 33.0, 'y3': 52.0},\n",
       " {'x1': 14.0, 'x2': 215.0, 'x3': 105.0, 'y1': 154.0, 'y2': 34.0, 'y3': 64.0},\n",
       " {'x1': 1.0, 'x2': 50.0, 'x3': 50.0, 'y1': 247.0, 'y2': 46.0, 'y3': 50.0},\n",
       " {'x1': 6.0, 'x2': 70.0, 'x3': 31.0, 'y1': 193.0, 'y2': 36.0, 'y3': 46.0},\n",
       " {'x1': 12.0, 'x2': 210.0, 'x3': 120.0, 'y1': 202.0, 'y2': 37.0, 'y3': 62.0},\n",
       " {'x1': 4.0, 'x2': 60.0, 'x3': 25.0, 'y1': 176.0, 'y2': 37.0, 'y3': 54.0},\n",
       " {'x1': 11.0, 'x2': 230.0, 'x3': 80.0, 'y1': 157.0, 'y2': 32.0, 'y3': 52.0},\n",
       " {'x1': 15.0, 'x2': 225.0, 'x3': 73.0, 'y1': 156.0, 'y2': 33.0, 'y3': 54.0},\n",
       " {'x1': 2.0, 'x2': 110.0, 'x3': 43.0, 'y1': 138.0, 'y2': 33.0, 'y3': 68.0}]"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {
    "scrolled": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Instantiate our Model with parameters\n",
    "Dffml makes it quite easy to load multiple models dynamically using the `Model.load()` function. All the entrypoints for models available in DFFML can be found at the [Model Plugins Page](../../plugins/dffml_model.rst).\n",
    "\n",
    "**Note:** All(and only) [Scikit Regressors and Classifiers](../../plugins/dffml_model.rst#dffml-model-scikit) support multioutput.\n",
    "\n",
    "After that, you just have to parameterize the loaded model and it is ready to train!\n",
    "\n",
    "Note that we will set `predict` in the model config to `Features`, a list of features ie. the targets, instead of a single feature. This sets the output targets in the model causing it to work as a Multi-Output model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "ScikitRidgeModel = Model.load(\"scikitridge\")\n",
    "\n",
    "features = Features(\n",
    "    Feature(\"x1\", int, 1), Feature(\"x2\", int, 1), Feature(\"x3\", int, 1),\n",
    ")\n",
    "predict_features = Features(\n",
    "    Feature(\"y1\", int, 1), Feature(\"y2\", int, 1), Feature(\"y3\", int, 1),\n",
    ")\n",
    "\n",
    "multi_reg_model = ScikitRidgeModel(\n",
    "    features=features, predict=predict_features, location=\"scikitridgemulti\",\n",
    ")\n",
    "print(type(predict_features))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'dffml.feature.feature.Features'>\n"
     ]
    }
   ],
   "metadata": {
    "scrolled": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Train our Model\n",
    "Finally, our model is ready to be trained using the `high-level` API. Let's make sure to pass each record as a parameter by simply using the unpacking operator(*)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "await train(multi_reg_model, *data)"
   ],
   "outputs": [],
   "metadata": {
    "scrolled": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Test our Models\n",
    "\n",
    "To test our model, we'll use the `score()` function in the `high-level` API.\n",
    "\n",
    "Just like models, the scorers can be dynamically loaded in a similar fashion. \n",
    "\n",
    "We will evaluate our model using the Mean Squared Error Scorer by passing `\"meansqrerr\"` to `AccuracyScorer.load()`, which is an entrypoint for a scikit scorer.\n",
    "\n",
    "All the available entrypoints for scorers can be found at the [Scorers Plugin Page](../../plugins/dffml_accuracy.rst)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "MeanSquaredError = AccuracyScorer.load(\"meansqrerr\")\n",
    "\n",
    "scorer = MeanSquaredError()\n",
    "\n",
    "print(\"Score:\", await score(multi_reg_model, scorer, predict_features, *data))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Score: 158.02450547009664\n"
     ]
    }
   ],
   "metadata": {
    "scrolled": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Make predictions using our Model.\n",
    "\n",
    "Let's make predictions and see what they look like for each model using the `predict` function in the `high-level` API.\n",
    "Note that the `predict` function returns an asynciterator of a `Record` object that contains a tuple of record.key, features and predictions."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "async for i, features, prediction in predict(multi_reg_model, *data):\n",
    "    print(prediction)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'y1': {'confidence': nan, 'value': 176.16484295610107}, 'y2': {'confidence': nan, 'value': 35.054840701238405}, 'y3': {'confidence': nan, 'value': 57.09000135700629}}\n",
      "{'y1': {'confidence': nan, 'value': 188.91063061276134}, 'y2': {'confidence': nan, 'value': 37.56282467492615}, 'y3': {'confidence': nan, 'value': 54.90134303600152}}\n",
      "{'y1': {'confidence': nan, 'value': 189.9534236494583}, 'y2': {'confidence': nan, 'value': 37.709031672727995}, 'y3': {'confidence': nan, 'value': 53.32601213195409}}\n",
      "{'y1': {'confidence': nan, 'value': 183.12565674452532}, 'y2': {'confidence': nan, 'value': 35.757565130397566}, 'y3': {'confidence': nan, 'value': 55.37951699532168}}\n",
      "{'y1': {'confidence': nan, 'value': 173.71664101067168}, 'y2': {'confidence': nan, 'value': 34.190412250345105}, 'y3': {'confidence': nan, 'value': 56.86332544362222}}\n",
      "{'y1': {'confidence': nan, 'value': 188.24891042818894}, 'y2': {'confidence': nan, 'value': 37.1498878097515}, 'y3': {'confidence': nan, 'value': 55.05551171914713}}\n",
      "{'y1': {'confidence': nan, 'value': 185.9832838429502}, 'y2': {'confidence': nan, 'value': 36.49249789985963}, 'y3': {'confidence': nan, 'value': 55.177648143397576}}\n",
      "{'y1': {'confidence': nan, 'value': 181.88881353329703}, 'y2': {'confidence': nan, 'value': 35.85249351395624}, 'y3': {'confidence': nan, 'value': 56.125241718843995}}\n",
      "{'y1': {'confidence': nan, 'value': 161.29353272098604}, 'y2': {'confidence': nan, 'value': 31.59790186825947}, 'y3': {'confidence': nan, 'value': 59.28698314880402}}\n",
      "{'y1': {'confidence': nan, 'value': 168.78382349070674}, 'y2': {'confidence': nan, 'value': 35.14015097438437}, 'y3': {'confidence': nan, 'value': 55.246088161502314}}\n",
      "{'y1': {'confidence': nan, 'value': 177.58499810260847}, 'y2': {'confidence': nan, 'value': 34.49820471614462}, 'y3': {'confidence': nan, 'value': 55.985845402357114}}\n",
      "{'y1': {'confidence': nan, 'value': 167.04250441212406}, 'y2': {'confidence': nan, 'value': 33.56470984175445}, 'y3': {'confidence': nan, 'value': 57.49566250529646}}\n",
      "{'y1': {'confidence': nan, 'value': 164.5494400870963}, 'y2': {'confidence': nan, 'value': 32.946828909890485}, 'y3': {'confidence': nan, 'value': 58.00146809453469}}\n",
      "{'y1': {'confidence': nan, 'value': 201.5214637228238}, 'y2': {'confidence': nan, 'value': 39.84126139242746}, 'y3': {'confidence': nan, 'value': 52.67320665194464}}\n",
      "{'y1': {'confidence': nan, 'value': 193.03036486026033}, 'y2': {'confidence': nan, 'value': 37.82070841168854}, 'y3': {'confidence': nan, 'value': 54.07885873309051}}\n",
      "{'y1': {'confidence': nan, 'value': 167.98119561913504}, 'y2': {'confidence': nan, 'value': 33.84093335998922}, 'y3': {'confidence': nan, 'value': 57.34729123889391}}\n",
      "{'y1': {'confidence': nan, 'value': 195.59664386575733}, 'y2': {'confidence': nan, 'value': 38.32927601046475}, 'y3': {'confidence': nan, 'value': 53.833189096823645}}\n",
      "{'y1': {'confidence': nan, 'value': 160.37561584969586}, 'y2': {'confidence': nan, 'value': 32.05130196894592}, 'y3': {'confidence': nan, 'value': 59.36513951595868}}\n",
      "{'y1': {'confidence': nan, 'value': 158.91979326107125}, 'y2': {'confidence': nan, 'value': 31.511817391159646}, 'y3': {'confidence': nan, 'value': 59.36551593805333}}\n",
      "{'y1': {'confidence': nan, 'value': 187.3284212297808}, 'y2': {'confidence': nan, 'value': 37.08735150168848}, 'y3': {'confidence': nan, 'value': 55.402150967446254}}\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Multi-Output Classifier\n",
    "\n",
    "To perform multi-output classification tasks, we can select any scikit classification model and pass our multi-output dataset onto the model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Build our Dataset\n",
    "We again utilize sklearn to build our multi-output dataset for our classification task."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "X, y = make_multilabel_classification(n_classes=3, n_features=3, random_state=1)\n",
    "data = np.concatenate((X, y), axis=1)\n",
    "data = [\n",
    "    {\"x1\": row[0], \"x2\": row[1], \"x3\": row[2], \"y1\": row[3], \"y2\": row[4], \"y3\": row[5]}\n",
    "    for row in data\n",
    "]\n",
    "data[0:5]"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[{'x1': 12.0, 'x2': 11.0, 'x3': 20.0, 'y1': 1.0, 'y2': 1.0, 'y3': 0.0},\n",
       " {'x1': 21.0, 'x2': 13.0, 'x3': 17.0, 'y1': 0.0, 'y2': 0.0, 'y3': 0.0},\n",
       " {'x1': 22.0, 'x2': 13.0, 'x3': 19.0, 'y1': 1.0, 'y2': 0.0, 'y3': 0.0},\n",
       " {'x1': 23.0, 'x2': 12.0, 'x3': 20.0, 'y1': 0.0, 'y2': 0.0, 'y3': 0.0},\n",
       " {'x1': 7.0, 'x2': 18.0, 'x3': 33.0, 'y1': 1.0, 'y2': 1.0, 'y3': 0.0}]"
      ]
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Instantiate our Model with parameters\n",
    "Dffml makes it quite easy to load multiple models dynamically using the `Model.load()` function. All the entrypoints for models available in DFFML can be found at the [Model Plugins Page](../../plugins/dffml_model.rst). After that, you just have to parameterize the loaded model and it is ready to train!\n",
    "\n",
    "Note that we will set `predict` in the model config to `Features`, a list of features ie. the targets, instead of a single feature. This sets the output targets in the model causing it to work as a Multi-Output model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "ScikitETCModel = Model.load(\"scikitetc\")\n",
    "\n",
    "features = Features(\n",
    "    Feature(\"x1\", int, 1), Feature(\"x2\", int, 1), Feature(\"x3\", int, 1),\n",
    ")\n",
    "predict_features = Features(\n",
    "    Feature(\"y1\", int, 1), Feature(\"y2\", int, 1), Feature(\"y3\", int, 1),\n",
    ")\n",
    "\n",
    "multi_classif_model = ScikitETCModel(\n",
    "    features=features, predict=predict_features, location=\"scikitetcmulti\",\n",
    ")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Train our Model\n",
    "Finally, our model is ready to be trained using the `high-level` API. Let's make sure to pass each record as a parameter by simply using the unpacking operator(*)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "await train(multi_classif_model, *data)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Test our Models\n",
    "\n",
    "To test our model, we'll again use the `score()` function in the `high-level` API.\n",
    "\n",
    "This time, we evaluate our model using the Accuracy Scorer by passing `\"acscore\"` to `AccuracyScorer.load()`, which is also an entrypoint for a scikit scorer.\n",
    "\n",
    "All the available entrypoints for scorers can be found at the [Scorers Plugin Page](../../plugins/dffml_accuracy.rst)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "Accuracy = AccuracyScorer.load(\"acscore\")\n",
    "\n",
    "scorer = Accuracy()\n",
    "\n",
    "print(\"Accuracy:\", await score(multi_classif_model, scorer, predict_features, *data))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Make predictions using our Model.\n",
    "\n",
    "Let's make predictions and see what they look like using the `predict` function in the `high-level` API.\n",
    "Note that the `predict` function returns an asynciterator of a `Record` object that contains a tuple of record.key, features and predictions."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "async for i, features, prediction in predict(multi_classif_model, *data):\n",
    "    print(prediction)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}\n",
      "{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}