1-training_ref_index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A MODNet tutorial - Training the refractive index\n",
    "\n",
    "This notebook goes trough the main functions and objects implemented by MODNet. Based on a dataset containing ~3,700 entries of type (mp_id, structure, refractive index) taken from the MaterialsProject (MP), a `MODNetModel` is trained. The workflow can be devided in two parts. First, the creation of a `MODData` object which stores the information concerning this particular dataset: the materials, the targets and optimal features. Second, a `MODNetModel` is trained which can later be used for predictions on unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import dependencies\n",
    "import sys\n",
    "from modnet.models import MODNetModel\n",
    "from modnet.preprocessing import MODData\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading the dataset\n",
    "\n",
    "In this example the dataset is a dataframe saved as a pickle. But it can be any format as long as you can retreive the structures and targets (and the mpids optionally for fast featurization)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3735 datapoints\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>structure</th>\n",
       "      <th>ref_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mp-755998</th>\n",
       "      <td>[[2.06202807e-06 2.06349574e+00 2.69529553e+00...</td>\n",
       "      <td>2.439110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-13602</th>\n",
       "      <td>[[0.97627791 4.96510018 6.65949814] O, [1.0206...</td>\n",
       "      <td>1.945601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-22467</th>\n",
       "      <td>[[3.24119011 0.84967668 3.36565113] O, [3.2411...</td>\n",
       "      <td>2.283458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-23364</th>\n",
       "      <td>[[1.23672715 1.23672715 0.81639113] Li, [3.710...</td>\n",
       "      <td>1.611454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-540621</th>\n",
       "      <td>[[2.11356961 8.25584606 4.0428252 ] Sr, [6.474...</td>\n",
       "      <td>1.747648</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   structure  ref_index\n",
       "mp-755998  [[2.06202807e-06 2.06349574e+00 2.69529553e+00...   2.439110\n",
       "mp-13602   [[0.97627791 4.96510018 6.65949814] O, [1.0206...   1.945601\n",
       "mp-22467   [[3.24119011 0.84967668 3.36565113] O, [3.2411...   2.283458\n",
       "mp-23364   [[1.23672715 1.23672715 0.81639113] Li, [3.710...   1.611454\n",
       "mp-540621  [[2.11356961 8.25584606 4.0428252 ] Sr, [6.474...   1.747648"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_pickle('data/df_ref.pkl')\n",
    "print('{} datapoints'.format(len(df)))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a quick look at the dataset in terms of target distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ref_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3735.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.956727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.436022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.236632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.658203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.890101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.138779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.120923</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ref_index\n",
       "count  3735.000000\n",
       "mean      1.956727\n",
       "std       0.436022\n",
       "min       1.236632\n",
       "25%       1.658203\n",
       "50%       1.890101\n",
       "75%       2.138779\n",
       "max       6.120923"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fa165265e80>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjH0lEQVR4nO3dfVRVdb7H8fcJFEklJgMHOxgYXgUUkEe9Kx1LKCcVNRVMU3wKr9bN8s60mNWsxnE1Qd3ulEzc22Wut2GsuczYKjAzG/Nh9PpEJ8VGScMHJkAGSTRQQHnY9w+X504juA8o5/Dwea3lWpx99t6/7y/yfPzt32/vYzEMw0BEROQm7nB1ASIi0vUpLERExJTCQkRETCksRETElMJCRERMubu6gM5yzz33EBAQ4OoyRES6lZKSEr755psbtvfYsAgICMBms7m6DBGRbiU6OrrV7boMJSIiphQWIiJiSmEhIiKmOm3OYsmSJWzevBlfX1+OHj0KQHV1NcnJyZSUlBAQEMAf/vAHvve97wGQnp7O+vXrcXNzIzMzk0ceeQSAzz//nEWLFlFfX8+jjz7KunXrsFgsnVW2iLSisbGRsrIyGhoaXF2K3Cb9+vXDarXSp08fxw4wOsmf/vQn4/PPPzdCQ0Pt23784x8b6enphmEYRnp6uvH8888bhmEYx44dM8LCwoyGhgbj9OnTxrBhw4ympibDMAwjJibG2Ldvn9HS0mJMnjzZ2LJli0PtR0VF3eYeifRep0+fNqqqqoyWlhZXlyK3QUtLi1FVVWWcPn36hvfa+uzstMtQEyZM4O677/7Otvz8fFJSUgBISUkhLy/Pvn3u3Ll4eHgQGBhIUFAQBQUFVFRUUFNTw7hx47BYLCxcuNB+jIg4T0NDA4MGDdKovoewWCwMGjSoXSNFp85ZVFZW4ufnB4Cfnx/nzp0DoLy8HH9/f/t+VquV8vJyysvLsVqtN2wXEedTUPQs7f19don7LIxWnpJusVja3N6W7OxssrOzAaiqqrp9BYqI9HJODYvBgwdTUVGBn58fFRUV+Pr6AtdGDKWlpfb9ysrKGDJkCFarlbKyshu2tyU1NZXU1FSg7RtLROTWBaR9dFvPV5IxxXQfNzc3Ro8eTVNTE4GBgWzYsAFvb+8296+qqmLq1KlcvXqVzMxMxo8f3/H6SkrYt28f8+bNA8Bms/Hb3/6WzMzMDp+zo+dZs2YNAwYM4Ec/+tEttd1eTg2LxMREcnJySEtLIycnh+nTp9u3z5s3j9WrV3P27FmKi4uJjY3Fzc2NgQMHcuDAAeLi4vjtb3/LP//zPzuzZKe61b+AjvyFE+muPD09KSwsBK7NeWZlZfHCCy+0uf/27dsZOXIkOTk5N7zX3NyMm5ubw22XlJTwu9/9zh4W0dHRt+UfpLfrPM7QaXMWjz/+OOPGjePEiRNYrVbWr19PWloa27ZtY/jw4Wzbto20tDQAQkNDSUpKIiQkhMmTJ5OVlWX/Rf7Hf/wHy5YtIygoiPvvv58f/vCHnVWyiHQT48aNs89fnjp1ismTJxMVFcX48eM5fvw4hYWFPP/882zZsoWIiAjq6+sZMGAAL774InFxcezfv5+1a9cSExPDqFGjSE1NtV/2PnnyJPHx8YSHhxMZGcmpU6dIS0tjz549RERE8Prrr7Nr1y6mTp1KS0sLAQEBXLx40V5bUFAQlZWVVFVVMWvWLGJiYoiJiWHv3r039OP6eeDaiGHJkiVMnDiRYcOGfWe08Ytf/IIRI0YQHx/PiRMn7Ntb63tTUxMxMTHs2rULgJ/85Cc3DVVHddrI4n/+539a3b59+/ZWt7/wwgutdig6Otp+n4aISHNzM9u3b2fp0qXAtcvPb731FsOHD+fgwYOsXLmSHTt2sHbtWmw2G2+++SYAly9fZtSoUaxduxaAkJAQXnzxRQAWLFjA5s2bmTZtGvPnzyctLY2ZM2fS0NBAS0sLGRkZvPbaa2zevBnA/kF8xx13MH36dD744AMWL17MwYMHCQgIYPDgwcybN4/nnnuOBx54gK+//ppHHnmEL7/88qZ9O378ODt37qS2tpYRI0awYsUKvvjiC3Jzczl8+DBNTU1ERkYSFRV1077/5je/Yfbs2WRmZrJ161YOHjx4y//du8QEt4iImfr6eiIiIigpKSEqKoqEhAQuXbrEvn37mDNnjn2/K1eutHq8m5sbs2bNsr/euXMnr776KnV1dVRXVxMaGsrEiRMpLy9n5syZwLUb18wkJyezdu1aFi9eTG5uLsnJyQB8+umnFBUV2ferqamhtraWgQMHtnmuKVOm4OHhgYeHB76+vlRWVrJnzx5mzpzJnXfeCVy7bA/ctO+hoaEsWLCAadOmsX//fvr27WvaDzMKCxHpFq7PWXz77bdMnTqVrKwsFi1ahLe3t30u42b69etnv7zd0NDAypUrsdls+Pv7s2bNGhoaGlpdgWlm3LhxnDx5kqqqKvLy8vjpT38KQEtLC/v378fT09Phc3l4eNh/dnNzo6mpCWh9FWhLS8tN+/7nP/8Zb29vKisr29GbtiksbrPbvUpERL7rrrvuIjMzk+nTp7NixQoCAwPZuHEjc+bMwTAMvvjiC8LDw296jus3o91zzz1cunSJ9957j9mzZ+Pl5YXVaiUvL48ZM2Zw5coVmpubGThwILW1ta2ey2KxMHPmTFavXk1wcDCDBg0C4OGHH+bNN9/kxz/+MQCFhYVERES0u78TJkxg0aJFpKWl0dTUxIcffsjy5cvx8vJqs+/vv/8+58+fZ/fu3UydOpWCgoKbrhxzhMJCRNrN1SvvxowZQ3h4OLm5ubz77rusWLGCl156icbGRubOnWsaFt7e3jz55JOMHj2agIAAYmJi7O9t2LCB5cuX8+KLL9KnTx82btxIWFgY7u7uhIeHs2jRIsaMGfOd8yUnJxMTE8NvfvMb+7bMzEyeeuopwsLCaGpqYsKECbz11lvt7mtkZCTJyclERERw3333fWcJcGt9v/fee0lLS2P79u34+/vz9NNPs2rVqlZXhbWHxejIuKsbiI6OdsmXH7lyZOHqv8DSc3355ZcEBwe7ugy5zVr7vbb12alHlIuIiCmFhYiImFJYiIhDeugV616rvb9PhYWImOrXrx/nz59XYPQQhmFw/vx5h+4juU6roUTE1PWHeuppzj3H9W/Kc5TCQkRM9enTh8DAQFeXIS6ky1AiImJKYSEiIqYUFiIiYkphISIiphQWIiJiSmEhIiKmFBYiImJKYSEiIqYUFiIiYkphISIiphQWIiJiSmEhIiKmFBYiImJKYSEiIqYUFiIiYkphISIiphQWIiJiSmEhIiKmFBYiImJKYSEiIqYUFiIiYkphISIiplwSFq+//jqhoaGMGjWKxx9/nIaGBqqrq0lISGD48OEkJCRw4cIF+/7p6ekEBQUxYsQIPvnkE1eULCLSq7k7u8Hy8nIyMzMpKirC09OTpKQkcnNzKSoqYtKkSaSlpZGRkUFGRgavvPIKRUVF5ObmcuzYMc6ePUt8fDxfffUVbm5uzi69ywtI+6jDx5ZkTLmNlYhIT+OSkUVTUxP19fU0NTVRV1fHkCFDyM/PJyUlBYCUlBTy8vIAyM/PZ+7cuXh4eBAYGEhQUBAFBQWuKFtEpNdyeljce++9/OhHP2Lo0KH4+flx11138fDDD1NZWYmfnx8Afn5+nDt3Drg2EvH397cfb7VaKS8vb/Xc2dnZREdHEx0dTVVVVed3RkSkl3B6WFy4cIH8/HzOnDnD2bNnuXz5Mu+8806b+xuGccM2i8XS6r6pqanYbDZsNhs+Pj63rWYRkd7O6WHx6aefEhgYiI+PD3369OGxxx5j3759DB48mIqKCgAqKirw9fUFro0kSktL7ceXlZUxZMgQZ5ctItKrOT0shg4dyoEDB6irq8MwDLZv305wcDCJiYnk5OQAkJOTw/Tp0wFITEwkNzeXK1eucObMGYqLi4mNjXV22SIivZrTV0PFxcUxe/ZsIiMjcXd3Z8yYMaSmpnLp0iWSkpJYv349Q4cOZePGjQCEhoaSlJRESEgI7u7uZGVlaSWUiIiTWYzWJgV6gOjoaGw2m9PbvZXlq66kpbMiAm1/duoObhERMaWwEBERUwoLERExpbAQERFTCgsRETGlsBAREVMKCxERMaWwEBERUwoLERExpbAQERFTCgsRETGlsBAREVMKCxERMaWwEBERUwoLERExpbAQERFTCgsRETGlsBAREVMKCxERMaWwEBERUwoLERExpbAQERFTCgsRETGlsBAREVMKCxERMaWwEBERUwoLERExpbAQERFTCgsRETGlsBAREVMOhcXRo0c7uw4REenCHAqLf/qnfyI2NpZ///d/5+LFi51ckoiIdDUOhcX//u//8u6771JaWkp0dDTz5s1j27ZtnV2biIh0EQ7PWQwfPpyXXnqJV155hT/96U8888wzjBw5kvfff7/djV68eJHZs2czcuRIgoOD2b9/P9XV1SQkJDB8+HASEhK4cOGCff/09HSCgoIYMWIEn3zySbvbExGRW+NQWHzxxRc899xzBAcHs2PHDj788EO+/PJLduzYwXPPPdfuRletWsXkyZM5fvw4R44cITg4mIyMDCZNmkRxcTGTJk0iIyMDgKKiInJzczl27Bhbt25l5cqVNDc3t7tNERHpOIfC4umnnyYyMpIjR46QlZVFZGQkAEOGDOGll15qV4M1NTXs3r2bpUuXAtC3b1+8vb3Jz88nJSUFgJSUFPLy8gDIz89n7ty5eHh4EBgYSFBQEAUFBe1qU0REbo27Iztt2bIFT09P3NzcAGhpaaGhoYE777yTBQsWtKvB06dP4+Pjw+LFizly5AhRUVGsW7eOyspK/Pz8APDz8+PcuXMAlJeXM3bsWPvxVquV8vLyVs+dnZ1NdnY2AFVVVe2qS0RE2ubQyCI+Pp76+nr767q6OuLj4zvUYFNTE4cOHWLFihUcPnyY/v372y85tcYwjBu2WSyWVvdNTU3FZrNhs9nw8fHpUH0iInIjh8KioaGBAQMG2F8PGDCAurq6DjVotVqxWq3ExcUBMHv2bA4dOsTgwYOpqKgAoKKiAl9fX/v+paWl9uPLysoYMmRIh9oWEZGOcSgs+vfvz6FDh+yvP//8czw9PTvU4Pe//338/f05ceIEANu3byckJITExERycnIAyMnJYfr06QAkJiaSm5vLlStXOHPmDMXFxcTGxnaobRER6RiH5izeeOMN5syZY/8XfUVFBb///e873OivfvUr5s+fz9WrVxk2bBhvv/02LS0tJCUlsX79eoYOHcrGjRsBCA0NJSkpiZCQENzd3cnKyrLPnYiIiHNYjNYmBVrR2NjIiRMnMAyDkSNH0qdPn86u7ZZER0djs9mc3m5A2kdOb/N2KMmY4uoSRKQLaOuz06GRBcBnn31GSUkJTU1NHD58GICFCxfevgpFRKTLcigsFixYwKlTp4iIiLBfArJYLAoLEZFewqGwsNlsFBUVtblkVUREejaHVkONGjWKv/71r51di4iIdFEOjSy++eYbQkJCiI2NxcPDw75906ZNnVaYiIh0HQ6FxZo1azq5DBER6cocCosf/OAH/OUvf6G4uJj4+Hjq6ur05FcRkV7EoTmLX//618yePZvly5cD1x7uN2PGjM6sS0REuhCHwiIrK4u9e/fi5eUFXPsipOtPhRURkZ7PobDw8PCgb9++9tdNTU1aRisi0os4FBY/+MEPePnll6mvr2fbtm3MmTOHadOmdXZtIiLSRTgUFhkZGfj4+DB69Gj+8z//k0cffbTd35AnIiLdl0Oroe644w6efPJJnnzyyc6uR0REuiCHwiIwMLDVOYrTp0/f9oJERKTrcfjZUNc1NDSwceNGqqurO60oERHpWhyasxg0aJD9z7333suzzz7Ljh07Ors2ERHpIhwaWfztV6q2tLRgs9mora3ttKJERKRrcSgs/uVf/uX/D3B3JyAggD/84Q+dVpSIiHQtDoXFzp07O7sOERHpwhwKi1/+8pc3fX/16tW3pRgREemaHF4N9dlnn5GYmAjAhx9+yIQJE/D39+/U4kREpGtw+MuPDh06xMCBA4Fr328xZ84c/uu//qtTixMRka7BoaWzX3/99XceJNi3b19KSko6qyYREeliHBpZLFiwgNjYWGbOnInFYuGDDz5g4cKFnV2biIh0EQ6FxQsvvMAPf/hD9uzZA8Dbb7/NmDFjOrUwERHpOhy6DAVQV1eHl5cXq1atwmq1cubMmc6sS0REuhCHwuLnP/85r7zyCunp6QA0NjbyxBNPdGphIiLSdTgUFh988AGbNm2if//+AAwZMkSP+xAR6UUcCou+fftisVjsjym/fPlypxYlIiJdi0NhkZSUxPLly7l48SK//vWviY+P1xchiYj0IqaroQzDIDk5mePHj+Pl5cWJEydYu3YtCQkJzqhPRES6ANOwsFgszJgxg88//1wBISLSSzl0GWrs2LF89tlnnV2LiIh0UQ6Fxc6dOxk7diz3338/YWFhjB49mrCwsFtquLm5mTFjxjB16lQAqqurSUhIYPjw4SQkJHDhwgX7vunp6QQFBTFixAg++eSTW2pXRETa76aXob7++muGDh3Kxx9/fNsbXrduHcHBwdTU1ACQkZHBpEmTSEtLIyMjg4yMDF555RWKiorIzc3l2LFjnD17lvj4eL766ivc3Nxue00iItK6m44sZsyYAcB9993H6tWrue+++77zp6PKysr46KOPWLZsmX1bfn4+KSkpAKSkpJCXl2ffPnfuXDw8PAgMDCQoKIiCgoIOty0iIu1305GFYRj2n0+fPn3bGn322Wd59dVXv3NjX2VlJX5+fgD4+flx7tw5AMrLyxk7dqx9P6vVSnl5eavnzc7OJjs7G4CqqqrbVm9vEJD2UYePLcmYchsrEZGu6KYji+s34f39z7di8+bN+Pr6EhUV5dD+fxtYZrWkpqZis9mw2Wz4+PjcUp0iIvL/bjqyOHLkCF5eXhiGQX19PV5eXsC1D3CLxWKfb2iPvXv3smnTJrZs2UJDQwM1NTU88cQTDB48mIqKCvz8/KioqMDX1xe4NpIoLS21H19WVsaQIUPa3a6IiHTcTUcWzc3N1NTUUFtbS1NTEzU1NfbXHQkKuLayqaysjJKSEnJzc3nooYd45513SExMJCcnB4CcnBymT58OQGJiIrm5uVy5coUzZ85QXFxMbGxsh9oWEZGOcej7LJwhLS2NpKQk1q9fz9ChQ9m4cSMAoaGhJCUlERISgru7O1lZWVoJJSLiZBajtUmBHiA6Ohqbzeb0dm9lori70gS3SM/R1menw19+JCIivZfCQkRETCksRETElMJCRERMKSxERMSUwkJEREwpLERExJTCQkRETCksRETElMJCRERMKSxERMSUwkJEREwpLERExJTCQkRETCksRETElMJCRERMKSxERMSUwkJEREwpLERExJTCQkRETCksRETElMJCRERMKSxERMSUwkJEREwpLERExJTCQkRETLm7uoCuKCDtI1eXICLSpWhkISIiphQWIiJiSmEhIiKmFBYiImJKYSEiIqYUFiIiYsrpYVFaWsqDDz5IcHAwoaGhrFu3DoDq6moSEhIYPnw4CQkJXLhwwX5Meno6QUFBjBgxgk8++cTZJYuI9HpODwt3d3f+7d/+jS+//JIDBw6QlZVFUVERGRkZTJo0ieLiYiZNmkRGRgYARUVF5ObmcuzYMbZu3crKlStpbm52dtkiIr2a08PCz8+PyMhIAAYOHEhwcDDl5eXk5+eTkpICQEpKCnl5eQDk5+czd+5cPDw8CAwMJCgoiIKCAmeXLSLSq7l0zqKkpITDhw8TFxdHZWUlfn5+wLVAOXfuHADl5eX4+/vbj7FarZSXl7d6vuzsbKKjo4mOjqaqqqrzOyAi0ku4LCwuXbrErFmzeOONN/Dy8mpzP8MwbthmsVha3Tc1NRWbzYbNZsPHx+e21Soi0tu5JCwaGxuZNWsW8+fP57HHHgNg8ODBVFRUAFBRUYGvry9wbSRRWlpqP7asrIwhQ4Y4v2gRkV7M6WFhGAZLly4lODiY1atX27cnJiaSk5MDQE5ODtOnT7dvz83N5cqVK5w5c4bi4mJiY2OdXbaISK/m9KfO7t27lw0bNjB69GgiIiIAePnll0lLSyMpKYn169czdOhQNm7cCEBoaChJSUmEhITg7u5OVlYWbm5uzi5bRKRXsxitTQr0ANHR0dhstg4dq0eUt09JxhRXlyAit0lbn526g1tEREwpLERExJTCQkRETCksRETElMJCRERMKSxERMSUwkJEREw5/aY86Xlu5b4U3aMh0j1oZCEiIqYUFiIiYkphISIiphQWIiJiSmEhIiKmFBYiImJKYSEiIqYUFiIiYkphISIiphQWIiJiSmEhIiKmFBYiImJKYSEiIqb01FlxKT2xVqR70MhCRERMKSxERMSUwkJEREwpLERExJQmuKXbupXJcdAEuUh7aGQhIiKmFBYiImJKYSEiIqYUFiIiYkoT3CIdoDvPpbfRyEJEREx1m5HF1q1bWbVqFc3NzSxbtoy0tDRXlyTd3K0uvXVFuxqViKt0i7Bobm7mqaeeYtu2bVitVmJiYkhMTCQkJMTVpYn0Ggq53q1bhEVBQQFBQUEMGzYMgLlz55Kfn6+wkF5HNyJ2Hz0tXLtFWJSXl+Pv729/bbVaOXjw4A37ZWdnk52dDcDRo0eJjo7uUHv3dKzMLqGqqgofHx9Xl+ESvbXv7el3dPTPOtzOrfy9uJV229LVf9+d+d+rM/teUlLS6vZuERaGYdywzWKx3LAtNTWV1NRUAKKjo7HZbJ1eW1fTW/sNvbfv6nfv44q+d4vVUFarldLSUvvrsrIyhgwZ4sKKRER6l24RFjExMRQXF3PmzBmuXr1Kbm4uiYmJri5LRKTX6BaXodzd3XnzzTd55JFHaG5uZsmSJYSGht70mOuXo3qb3tpv6L19V797H1f03WK0NiEgIiLyN7rFZSgREXEthYWIiJjqcWGxZMkSfH19GTVqlKtLcarS0lIefPBBgoODCQ0NZd26da4uySkaGhqIjY0lPDyc0NBQfvaz27+evytrbm5mzJgxTJ061dWlOFVAQACjR48mIiKiw/dTdUcXL15k9uzZjBw5kuDgYPbv3++0tnvcnMXu3bsZMGAACxcu5OjRo64ux2kqKiqoqKggMjKS2tpaoqKiyMvL6/F3uRuGweXLlxkwYACNjY088MADrFu3jrFjx7q6NKf45S9/ic1mo6amhs2bN7u6HKcJCAjAZrNxzz3d+Rba9ktJSWH8+PEsW7aMq1evUldXh7e3t1Pa7nEjiwkTJnD33Xe7ugyn8/PzIzIyEoCBAwcSHBxMeXm5i6vqfBaLhQEDBgDQ2NhIY2Njqzds9kRlZWV89NFHLFu2zNWliBPU1NSwe/duli5dCkDfvn2dFhTQA8NCrt2uf/jwYeLi4lxdilM0NzcTERGBr68vCQkJvabfzz77LK+++ip33NH7/hpbLBYefvhhoqKi7I/46elOnz6Nj48PixcvZsyYMSxbtozLly87rf3e939ZD3fp0iVmzZrFG2+8gZeXl6vLcQo3NzcKCwspKyujoKCgV1x+3Lx5M76+vkRFRbm6FJfYu3cvhw4d4uOPPyYrK4vdu3e7uqRO19TUxKFDh1ixYgWHDx+mf//+ZGRkOK19hUUP0tjYyKxZs5g/fz6PPfaYq8txOm9vbyZOnMjWrVtdXUqn27t3L5s2bSIgIIC5c+eyY8cOnnjiCVeX5TTXH/fj6+vLzJkzKSgocHFFnc9qtWK1Wu0j59mzZ3Po0CGnta+w6CEMw2Dp0qUEBwezevVqV5fjNFVVVVy8eBGA+vp6Pv30U0aOHOnaopwgPT2dsrIySkpKyM3N5aGHHuKdd95xdVlOcfnyZWpra+0///GPf+wVqx+///3v4+/vz4kTJwDYvn27UxewdIvHfbTH448/zq5du/jmm2+wWq38/Oc/t08I9WR79+5lw4YN9uWEAC+//DKPPvqoawvrZBUVFaSkpNDc3ExLSwtJSUm9bhlpb1NZWcnMmTOBa5dm5s2bx+TJk11clXP86le/Yv78+Vy9epVhw4bx9ttvO63tHrd0VkREbj9dhhIREVMKCxERMaWwEBERUwoLERExpbAQERFTCgvpltzc3IiIiGDUqFFMmzbNfq9FW6qqqoiLi2PMmDHs2bPnltouKSnhd7/7nf21zWbjmWeeuaVzdvQ8a9as4bXXXrvltuHa40NauxO6qqqq1yxNlbYpLKRb8vT0pLCwkKNHj3L33XeTlZV10/23b9/OyJEjOXz4MOPHj//Oe83Nze1q++/DIjo6mszMzHadozW36zwdUV1dzYEDB5gwYcIN7/n4+ODn58fevXtdUJl0FQoL6fbGjRtnf8LuqVOnmDx5MlFRUYwfP57jx49TWFjI888/z5YtW4iIiKC+vp4BAwbw4osvEhcXx/79+1m7di0xMTGMGjWK1NRUrt9+dPLkSeLj4wkPDycyMpJTp06RlpbGnj17iIiI4PXXX2fXrl1MnTqVlpYWAgICvjPKCQoKorKykqqqKmbNmkVMTAwxMTGtfvBePw9cGzEsWbKEiRMnMmzYsO+EyC9+8QtGjBhBfHy8/W7etvre1NRETEwMu3btAuAnP/kJL7zwwg1tv/feezcdPcyYMYN3333X8V+K9DyGSDfUv39/wzAMo6mpyZg9e7bx8ccfG4ZhGA899JDx1VdfGYZhGAcOHDAefPBBwzAM4+233zaeeuop+/GA8fvf/97++vz58/afn3jiCWPTpk2GYRhGbGys8f777xuGYRj19fXG5cuXjZ07dxpTpkyx7/+3r5955hnjv//7v+3tT5o0yTAMw3j88ceNPXv2GIZhGH/5y1+MkSNH3tCnvz3Pz372M2PcuHFGQ0ODUVVVZdx9993G1atXDZvNZowaNcq4fPmy8e233xr333+/8a//+q837fvRo0eNkSNHGn/84x+NiIgI48qVKze0vXDhQnufW1NWVmaMGjWqzfel5+txj/uQ3qG+vp6IiAhKSkqIiooiISGBS5cusW/fPubMmWPf78qVK60e7+bmxqxZs+yvd+7cyauvvkpdXR3V1dWEhoYyceJEysvL7Y+W6Nevn2ldycnJrF27lsWLF5Obm0tycjIAn376KUVFRfb9ampqqK2tZeDAgW2ea8qUKXh4eODh4YGvry+VlZXs2bOHmTNncueddwKQmJgIcNO+h4aGsmDBAqZNm8b+/fvp27fvDW1VVFTg4+PTZi2+vr6cPXvWtP/ScykspFu6Pmfx7bffMnXqVLKysli0aBHe3t4UFhaaHt+vXz/c3NyAa1/NunLlSmw2G/7+/qxZs4aGhgb7paj2GDduHCdPnqSqqoq8vDx++tOfAtDS0sL+/fvx9PR0+FweHh72n93c3GhqagJo9cudWlpabtr3P//5z3h7e1NZWdnq+56enjQ0NABw8OBBli9fDsDatWtJTEykoaGhXbVLz6M5C+nW7rrrLjIzM3nttdfw9PQkMDCQjRs3AteexHvkyBHTc1z/kLznnnu4dOkS7733HgBeXl5YrVby8vKAa/9Sr6urY+DAgfannv49i8XCzJkzWb16NcHBwQwaNAiAhx9+mDfffNO+nyOB1poJEybwwQcfUF9fT21tLR9++KG91rb6/v7773P+/Hl2797NM8880+rKseDgYE6ePAlAXFwchYWFFBYW2kcuX331Va94squ0TWEh3d6YMWMIDw8nNzeXd999l/Xr1xMeHk5oaCj5+fmmx3t7e/Pkk08yevRoZsyYQUxMjP29DRs2kJmZSVhYGP/4j//IX//6V8LCwnB3dyc8PJzXX3/9hvMlJyfzzjvv2C9BAWRmZmKz2QgLCyMkJIS33nqrQ32NjIwkOTmZiIgIZs2a9Z2VXa31/ZtvviEtLY3169fzD//wDzz99NOsWrXqhvNOmTLFPgnemp07dzJlypQO1Sw9g546KyIAPPDAA2zevLnV73WeMGEC+fn5fO9733N+YdIlKCxEBLg2V+Hp6UlYWNh3tldVVbF3715mzJjhmsKkS1BYiIiIKc1ZiIiIKYWFiIiYUliIiIgphYWIiJhSWIiIiKn/AyYmxUnNzgdYAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(facecolor=\"w\")\n",
    "ax.hist(df[\"ref_index\"], bins=20, density=False, label=\"Refractive index\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_xlabel(\"Refractive index (-)\")\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Creating a MODData instance\n",
    "\n",
    "### (a) structure, mpid, target creation\n",
    "\n",
    "The first step is to gather all data in a `MODData` object. Here we put the structures, targets, ids, and property names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:44:11,849 - modnet - INFO - Loaded Matminer2023Featurizer featurizer.\n"
     ]
    }
   ],
   "source": [
    "md = MODData(materials = df['structure'],\n",
    "             targets = df['ref_index'].values,\n",
    "             structure_ids = df.index, #will be used to identify compounds\n",
    "             target_names = ['refractive_index'] # you can choose any name\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (b) Featurizing the data\n",
    "Structures wil be converted into features, using the matminer library. This can take some time. Fortunately, the MODData has an integrated database containing the features of many materials from the MP. By enabling fast featurization (`fast=True`) they are directly retreived from this database and not computed from the structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:44:11,892 - modnet - INFO - Computing features, this can take time...\n",
      "2023-01-24 14:44:11,894 - modnet - INFO - Fast featurization on, retrieving from database...\n",
      "2023-01-24 14:44:13,934 - modnet - INFO - Retrieved features for 3735 out of 3735 materials\n",
      "2023-01-24 14:44:15,236 - modnet - INFO - Data has successfully been featurized!\n"
     ]
    }
   ],
   "source": [
    "md.featurize(fast=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AtomicOrbitals|HOMO_character</th>\n",
       "      <th>AtomicOrbitals|HOMO_element</th>\n",
       "      <th>AtomicOrbitals|HOMO_energy</th>\n",
       "      <th>AtomicOrbitals|LUMO_character</th>\n",
       "      <th>AtomicOrbitals|LUMO_element</th>\n",
       "      <th>AtomicOrbitals|LUMO_energy</th>\n",
       "      <th>AtomicOrbitals|gap_AO</th>\n",
       "      <th>AtomicPackingEfficiency|mean simul. packing efficiency</th>\n",
       "      <th>AtomicPackingEfficiency|mean abs simul. packing efficiency</th>\n",
       "      <th>AtomicPackingEfficiency|dist from 1 clusters |APE| &lt; 0.010</th>\n",
       "      <th>...</th>\n",
       "      <th>BondFractions|Co - O bond frac.</th>\n",
       "      <th>BondFractions|Bi - O bond frac.</th>\n",
       "      <th>BondFractions|Sc - Sc bond frac.</th>\n",
       "      <th>BondFractions|Co - Co bond frac.</th>\n",
       "      <th>BondFractions|O - Y bond frac.</th>\n",
       "      <th>BondFractions|Nb - O bond frac.</th>\n",
       "      <th>CoulombMatrix|coulomb matrix eig 123</th>\n",
       "      <th>BondFractions|C - O bond frac.</th>\n",
       "      <th>BondFractions|Li - P bond frac.</th>\n",
       "      <th>SineCoulombMatrix|sine coulomb matrix eig 122</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mp-755998</th>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>-0.266297</td>\n",
       "      <td>1.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>-0.162391</td>\n",
       "      <td>0.103906</td>\n",
       "      <td>0.012571</td>\n",
       "      <td>0.012571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-13602</th>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-0.338381</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-0.338381</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.020275</td>\n",
       "      <td>0.026709</td>\n",
       "      <td>0.015990</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-22467</th>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-0.338381</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-0.338381</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.025676</td>\n",
       "      <td>0.025813</td>\n",
       "      <td>0.024296</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-23364</th>\n",
       "      <td>2.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>-0.320380</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>-0.105540</td>\n",
       "      <td>0.214840</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.612372</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mp-540621</th>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-0.338381</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-0.338381</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.001310</td>\n",
       "      <td>0.050161</td>\n",
       "      <td>0.024298</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1359 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           AtomicOrbitals|HOMO_character  AtomicOrbitals|HOMO_element  \\\n",
       "id                                                                      \n",
       "mp-755998                            2.0                          7.0   \n",
       "mp-13602                             2.0                          8.0   \n",
       "mp-22467                             2.0                          8.0   \n",
       "mp-23364                             2.0                         17.0   \n",
       "mp-540621                            2.0                          8.0   \n",
       "\n",
       "           AtomicOrbitals|HOMO_energy  AtomicOrbitals|LUMO_character  \\\n",
       "id                                                                     \n",
       "mp-755998                   -0.266297                            1.0   \n",
       "mp-13602                    -0.338381                            2.0   \n",
       "mp-22467                    -0.338381                            2.0   \n",
       "mp-23364                    -0.320380                            1.0   \n",
       "mp-540621                   -0.338381                            2.0   \n",
       "\n",
       "           AtomicOrbitals|LUMO_element  AtomicOrbitals|LUMO_energy  \\\n",
       "id                                                                   \n",
       "mp-755998                         40.0                   -0.162391   \n",
       "mp-13602                           8.0                   -0.338381   \n",
       "mp-22467                           8.0                   -0.338381   \n",
       "mp-23364                           3.0                   -0.105540   \n",
       "mp-540621                          8.0                   -0.338381   \n",
       "\n",
       "           AtomicOrbitals|gap_AO  \\\n",
       "id                                 \n",
       "mp-755998               0.103906   \n",
       "mp-13602                0.000000   \n",
       "mp-22467                0.000000   \n",
       "mp-23364                0.214840   \n",
       "mp-540621               0.000000   \n",
       "\n",
       "           AtomicPackingEfficiency|mean simul. packing efficiency  \\\n",
       "id                                                                  \n",
       "mp-755998                                           0.012571        \n",
       "mp-13602                                           -0.020275        \n",
       "mp-22467                                           -0.025676        \n",
       "mp-23364                                            0.000000        \n",
       "mp-540621                                          -0.001310        \n",
       "\n",
       "           AtomicPackingEfficiency|mean abs simul. packing efficiency  \\\n",
       "id                                                                      \n",
       "mp-755998                                           0.012571            \n",
       "mp-13602                                            0.026709            \n",
       "mp-22467                                            0.025813            \n",
       "mp-23364                                            0.000000            \n",
       "mp-540621                                           0.050161            \n",
       "\n",
       "           AtomicPackingEfficiency|dist from 1 clusters |APE| < 0.010  ...  \\\n",
       "id                                                                     ...   \n",
       "mp-755998                                           0.000000           ...   \n",
       "mp-13602                                            0.015990           ...   \n",
       "mp-22467                                            0.024296           ...   \n",
       "mp-23364                                            0.612372           ...   \n",
       "mp-540621                                           0.024298           ...   \n",
       "\n",
       "           BondFractions|Co - O bond frac.  BondFractions|Bi - O bond frac.  \\\n",
       "id                                                                            \n",
       "mp-755998                              0.0                              0.0   \n",
       "mp-13602                               0.0                              0.0   \n",
       "mp-22467                               0.0                              0.0   \n",
       "mp-23364                               0.0                              0.0   \n",
       "mp-540621                              0.0                              0.0   \n",
       "\n",
       "           BondFractions|Sc - Sc bond frac.  BondFractions|Co - Co bond frac.  \\\n",
       "id                                                                              \n",
       "mp-755998                               0.0                               0.0   \n",
       "mp-13602                                0.0                               0.0   \n",
       "mp-22467                                0.0                               0.0   \n",
       "mp-23364                                0.0                               0.0   \n",
       "mp-540621                               0.0                               0.0   \n",
       "\n",
       "           BondFractions|O - Y bond frac.  BondFractions|Nb - O bond frac.  \\\n",
       "id                                                                           \n",
       "mp-755998                             0.0                              0.0   \n",
       "mp-13602                              0.0                              0.0   \n",
       "mp-22467                              0.0                              0.0   \n",
       "mp-23364                              0.0                              0.0   \n",
       "mp-540621                             0.0                              0.0   \n",
       "\n",
       "           CoulombMatrix|coulomb matrix eig 123  \\\n",
       "id                                                \n",
       "mp-755998                                   0.0   \n",
       "mp-13602                                    0.0   \n",
       "mp-22467                                    0.0   \n",
       "mp-23364                                    0.0   \n",
       "mp-540621                                   0.0   \n",
       "\n",
       "           BondFractions|C - O bond frac.  BondFractions|Li - P bond frac.  \\\n",
       "id                                                                           \n",
       "mp-755998                             0.0                              0.0   \n",
       "mp-13602                              0.0                              0.0   \n",
       "mp-22467                              0.0                              0.0   \n",
       "mp-23364                              0.0                              0.0   \n",
       "mp-540621                             0.0                              0.0   \n",
       "\n",
       "           SineCoulombMatrix|sine coulomb matrix eig 122  \n",
       "id                                                        \n",
       "mp-755998                                            0.0  \n",
       "mp-13602                                             0.0  \n",
       "mp-22467                                             0.0  \n",
       "mp-23364                                             0.0  \n",
       "mp-540621                                            0.0  \n",
       "\n",
       "[5 rows x 1359 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md.get_featurized_df().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (c) Computing the optimal features\n",
    "\n",
    "This runs the feature selection algorithm. First, the mutual information is computed, followed by the iterative selection based on relevance and redundancy.\n",
    "\n",
    "This step takes a little bit of time, but is normally run only once before being saved.\n",
    "\n",
    "Here, we use the precomputed cross-NMI, which corresponds to the NMI *between* features. This can either be run on the current dataset (`use_precomputed_cross_nmi=True`) or retreived from modnet itself as computed from a large set of data (`use_precomputed_cross_nmi=True`).\n",
    "We use the second option as it is faster. However, it will warn you that some features are not in the precomputed cross-NMI. This is normal as the dataset may contain more complex structures than used in the precomputed file. For simplicity of the tutorial (and even in most cases), it can be ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:44:15,437 - modnet - INFO - Loading cross NMI from 'Features_cross' file.\n",
      "2023-01-24 14:44:15,493 - modnet - WARNING - Feature mismatch between precomputed `Features_cross` and `df_featurized`. Missing columns: {'CoulombMatrix|coulomb matrix eig 121', 'SineCoulombMatrix|sine coulomb matrix eig 50', 'CoulombMatrix|coulomb matrix eig 33', 'SineCoulombMatrix|sine coulomb matrix eig 58', 'XRDPowderPattern|xrd_116', 'SineCoulombMatrix|sine coulomb matrix eig 42', 'SineCoulombMatrix|sine coulomb matrix eig 26', 'CoulombMatrix|coulomb matrix eig 14', 'CoulombMatrix|coulomb matrix eig 13', 'SineCoulombMatrix|sine coulomb matrix eig 114', 'CoulombMatrix|coulomb matrix eig 55', 'SineCoulombMatrix|sine coulomb matrix eig 77', 'CoulombMatrix|coulomb matrix eig 61', 'CoulombMatrix|coulomb matrix eig 96', 'BondFractions|Ta - Ta bond frac.', 'BondFractions|Y - Y bond frac.', 'XRDPowderPattern|xrd_90', 'XRDPowderPattern|xrd_42', 'SineCoulombMatrix|sine coulomb matrix eig 76', 'CoulombMatrix|coulomb matrix eig 97', 'BondFractions|O - Y bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 86', 'XRDPowderPattern|xrd_56', 'CoulombMatrix|coulomb matrix eig 45', 'SineCoulombMatrix|sine coulomb matrix eig 37', 'CoulombMatrix|coulomb matrix eig 37', 'SineCoulombMatrix|sine coulomb matrix eig 36', 'XRDPowderPattern|xrd_108', 'BondFractions|Ru - Ru bond frac.', 'CoulombMatrix|coulomb matrix eig 20', 'BondFractions|Pb - Pb bond frac.', 'CoulombMatrix|coulomb matrix eig 54', 'CoulombMatrix|coulomb matrix eig 48', 'SineCoulombMatrix|sine coulomb matrix eig 21', 'BondFractions|Pd - Pd bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 19', 'SineCoulombMatrix|sine coulomb matrix eig 110', 'BondFractions|W - W bond frac.', 'CoulombMatrix|coulomb matrix eig 31', 'BondFractions|O - Sb bond frac.', 'BondFractions|Nd - Nd bond frac.', 'CoulombMatrix|coulomb matrix eig 114', 'CoulombMatrix|coulomb matrix eig 104', 'SineCoulombMatrix|sine coulomb matrix eig 49', 'BondFractions|Sr - Sr bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 57', 'SineCoulombMatrix|sine coulomb matrix eig 99', 'SineCoulombMatrix|sine coulomb matrix eig 93', 'SineCoulombMatrix|sine coulomb matrix eig 85', 'XRDPowderPattern|xrd_84', 'CoulombMatrix|coulomb matrix eig 94', 'CoulombMatrix|coulomb matrix eig 71', 'CoulombMatrix|coulomb matrix eig 35', 'XRDPowderPattern|xrd_28', 'BondFractions|Bi - Bi bond frac.', 'BondFractions|As - As bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 88', 'CoulombMatrix|coulomb matrix eig 16', 'SineCoulombMatrix|sine coulomb matrix eig 83', 'SineCoulombMatrix|sine coulomb matrix eig 89', 'CoulombMatrix|coulomb matrix eig 21', 'CoulombMatrix|coulomb matrix eig 105', 'XRDPowderPattern|xrd_44', 'BondFractions|Ba - Ba bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 24', 'CoulombMatrix|coulomb matrix eig 89', 'BondFractions|Nb - Nb bond frac.', 'XRDPowderPattern|xrd_60', 'XRDPowderPattern|xrd_32', 'SineCoulombMatrix|sine coulomb matrix eig 34', 'BondFractions|O - W bond frac.', 'BondFractions|O - Sr bond frac.', 'BondFractions|Te - Te bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 81', 'CoulombMatrix|coulomb matrix eig 12', 'XRDPowderPattern|xrd_88', 'XRDPowderPattern|xrd_68', 'SineCoulombMatrix|sine coulomb matrix eig 39', 'SineCoulombMatrix|sine coulomb matrix eig 54', 'BondFractions|U - U bond frac.', 'CoulombMatrix|coulomb matrix eig 38', 'CoulombMatrix|coulomb matrix eig 23', 'CoulombMatrix|coulomb matrix eig 26', 'CoulombMatrix|coulomb matrix eig 81', 'CoulombMatrix|coulomb matrix eig 80', 'CoulombMatrix|coulomb matrix eig 91', 'CoulombMatrix|coulomb matrix eig 112', 'XRDPowderPattern|xrd_118', 'BondFractions|Pr - Pr bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 14', 'CoulombMatrix|coulomb matrix eig 119', 'SineCoulombMatrix|sine coulomb matrix eig 61', 'BondFractions|Re - Re bond frac.', 'CoulombMatrix|coulomb matrix eig 65', 'XRDPowderPattern|xrd_58', 'SineCoulombMatrix|sine coulomb matrix eig 62', 'XRDPowderPattern|xrd_86', 'XRDPowderPattern|xrd_78', 'SineCoulombMatrix|sine coulomb matrix eig 82', 'XRDPowderPattern|xrd_98', 'SineCoulombMatrix|sine coulomb matrix eig 96', 'SineCoulombMatrix|sine coulomb matrix eig 75', 'SineCoulombMatrix|sine coulomb matrix eig 92', 'SineCoulombMatrix|sine coulomb matrix eig 123', 'CoulombMatrix|coulomb matrix eig 19', 'CoulombMatrix|coulomb matrix eig 92', 'CoulombMatrix|coulomb matrix eig 109', 'SineCoulombMatrix|sine coulomb matrix eig 73', 'XRDPowderPattern|xrd_4', 'XRDPowderPattern|xrd_12', 'CoulombMatrix|coulomb matrix eig 126', 'XRDPowderPattern|xrd_2', 'CoulombMatrix|coulomb matrix eig 127', 'SineCoulombMatrix|sine coulomb matrix eig 22', 'CoulombMatrix|coulomb matrix eig 66', 'XRDPowderPattern|xrd_6', 'SineCoulombMatrix|sine coulomb matrix eig 43', 'CoulombMatrix|coulomb matrix eig 107', 'XRDPowderPattern|xrd_26', 'CoulombMatrix|coulomb matrix eig 120', 'CoulombMatrix|coulomb matrix eig 63', 'SineCoulombMatrix|sine coulomb matrix eig 125', 'SineCoulombMatrix|sine coulomb matrix eig 25', 'CoulombMatrix|coulomb matrix eig 111', 'SineCoulombMatrix|sine coulomb matrix eig 104', 'XRDPowderPattern|xrd_80', 'XRDPowderPattern|xrd_62', 'CoulombMatrix|coulomb matrix eig 88', 'SineCoulombMatrix|sine coulomb matrix eig 52', 'CoulombMatrix|coulomb matrix eig 56', 'SineCoulombMatrix|sine coulomb matrix eig 94', 'CoulombMatrix|coulomb matrix eig 106', 'CoulombMatrix|coulomb matrix eig 115', 'BondFractions|Gd - Gd bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 66', 'BondFractions|Yb - Yb bond frac.', 'CoulombMatrix|coulomb matrix eig 90', 'XRDPowderPattern|xrd_22', 'SineCoulombMatrix|sine coulomb matrix eig 55', 'BondFractions|Hg - Hg bond frac.', 'XRDPowderPattern|xrd_100', 'CoulombMatrix|coulomb matrix eig 78', 'SineCoulombMatrix|sine coulomb matrix eig 20', 'SineCoulombMatrix|sine coulomb matrix eig 91', 'SineCoulombMatrix|sine coulomb matrix eig 108', 'CoulombMatrix|coulomb matrix eig 49', 'CoulombMatrix|coulomb matrix eig 125', 'CoulombMatrix|coulomb matrix eig 102', 'BondFractions|Tl - Tl bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 47', 'CoulombMatrix|coulomb matrix eig 15', 'CoulombMatrix|coulomb matrix eig 76', 'CoulombMatrix|coulomb matrix eig 74', 'XRDPowderPattern|xrd_114', 'XRDPowderPattern|xrd_10', 'CoulombMatrix|coulomb matrix eig 62', 'CoulombMatrix|coulomb matrix eig 69', 'XRDPowderPattern|xrd_36', 'Miedema|Miedema_deltaH_ss_min', 'BondFractions|Sn - Sn bond frac.', 'XRDPowderPattern|xrd_48', 'SineCoulombMatrix|sine coulomb matrix eig 53', 'XRDPowderPattern|xrd_82', 'SineCoulombMatrix|sine coulomb matrix eig 126', 'SineCoulombMatrix|sine coulomb matrix eig 41', 'CoulombMatrix|coulomb matrix eig 28', 'SineCoulombMatrix|sine coulomb matrix eig 59', 'CoulombMatrix|coulomb matrix eig 68', 'BondFractions|Ho - Ho bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 33', 'CoulombMatrix|coulomb matrix eig 86', 'XRDPowderPattern|xrd_20', 'CoulombMatrix|coulomb matrix eig 22', 'SineCoulombMatrix|sine coulomb matrix eig 118', 'BondFractions|O - Sn bond frac.', 'CoulombMatrix|coulomb matrix eig 42', 'XRDPowderPattern|xrd_46', 'SineCoulombMatrix|sine coulomb matrix eig 65', 'CoulombMatrix|coulomb matrix eig 59', 'CoulombMatrix|coulomb matrix eig 122', 'BondFractions|Sm - Sm bond frac.', 'XRDPowderPattern|xrd_124', 'CoulombMatrix|coulomb matrix eig 57', 'BondFractions|O - Te bond frac.', 'XRDPowderPattern|xrd_76', 'SineCoulombMatrix|sine coulomb matrix eig 29', 'BondFractions|H - O bond frac.', 'BondFractions|H - N bond frac.', 'BondFractions|Br - Br bond frac.', 'BondFractions|Rh - Rh bond frac.', 'CoulombMatrix|coulomb matrix eig 40', 'CoulombMatrix|coulomb matrix eig 73', 'XRDPowderPattern|xrd_40', 'BondFractions|Mo - Mo bond frac.', 'XRDPowderPattern|xrd_16', 'CoulombMatrix|coulomb matrix eig 84', 'XRDPowderPattern|xrd_52', 'SineCoulombMatrix|sine coulomb matrix eig 48', 'SineCoulombMatrix|sine coulomb matrix eig 105', 'CoulombMatrix|coulomb matrix eig 95', 'SineCoulombMatrix|sine coulomb matrix eig 103', 'CoulombMatrix|coulomb matrix eig 24', 'CoulombMatrix|coulomb matrix eig 43', 'SineCoulombMatrix|sine coulomb matrix eig 45', 'CoulombMatrix|coulomb matrix eig 32', 'SineCoulombMatrix|sine coulomb matrix eig 115', 'SineCoulombMatrix|sine coulomb matrix eig 35', 'BondFractions|Pt - Pt bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 17', 'SineCoulombMatrix|sine coulomb matrix eig 27', 'XRDPowderPattern|xrd_72', 'SineCoulombMatrix|sine coulomb matrix eig 106', 'BondFractions|Tb - Tb bond frac.', 'CoulombMatrix|coulomb matrix eig 47', 'CoulombMatrix|coulomb matrix eig 98', 'CoulombMatrix|coulomb matrix eig 77', 'XRDPowderPattern|xrd_104', 'SineCoulombMatrix|sine coulomb matrix eig 67', 'SineCoulombMatrix|sine coulomb matrix eig 74', 'CoulombMatrix|coulomb matrix eig 44', 'SineCoulombMatrix|sine coulomb matrix eig 44', 'SineCoulombMatrix|sine coulomb matrix eig 101', 'SineCoulombMatrix|sine coulomb matrix eig 56', 'SineCoulombMatrix|sine coulomb matrix eig 64', 'XRDPowderPattern|xrd_96', 'SineCoulombMatrix|sine coulomb matrix eig 113', 'CoulombMatrix|coulomb matrix eig 39', 'CoulombMatrix|coulomb matrix eig 123', 'SineCoulombMatrix|sine coulomb matrix eig 100', 'BondFractions|Sb - Sb bond frac.', 'XRDPowderPattern|xrd_120', 'CoulombMatrix|coulomb matrix eig 83', 'SineCoulombMatrix|sine coulomb matrix eig 60', 'SineCoulombMatrix|sine coulomb matrix eig 121', 'SineCoulombMatrix|sine coulomb matrix eig 112', 'SineCoulombMatrix|sine coulomb matrix eig 79', 'BondFractions|O - Rb bond frac.', 'CoulombMatrix|coulomb matrix eig 11', 'SineCoulombMatrix|sine coulomb matrix eig 63', 'SineCoulombMatrix|sine coulomb matrix eig 90', 'XRDPowderPattern|xrd_106', 'XRDPowderPattern|xrd_126', 'CoulombMatrix|coulomb matrix eig 100', 'XRDPowderPattern|xrd_18', 'SineCoulombMatrix|sine coulomb matrix eig 31', 'XRDPowderPattern|xrd_66', 'CoulombMatrix|coulomb matrix eig 29', 'XRDPowderPattern|xrd_102', 'BondFractions|Cd - Cd bond frac.', 'XRDPowderPattern|xrd_24', 'BondFractions|Hf - Hf bond frac.', 'CoulombMatrix|coulomb matrix eig 30', 'CoulombMatrix|coulomb matrix eig 60', 'BondFractions|Zr - Zr bond frac.', 'CoulombMatrix|coulomb matrix eig 85', 'SineCoulombMatrix|sine coulomb matrix eig 116', 'BondFractions|Au - Au bond frac.', 'CoulombMatrix|coulomb matrix eig 99', 'CoulombMatrix|coulomb matrix eig 51', 'SineCoulombMatrix|sine coulomb matrix eig 23', 'BondFractions|Tm - Tm bond frac.', 'CoulombMatrix|coulomb matrix eig 18', 'SineCoulombMatrix|sine coulomb matrix eig 11', 'XRDPowderPattern|xrd_8', 'SineCoulombMatrix|sine coulomb matrix eig 68', 'BondFractions|La - La bond frac.', 'CoulombMatrix|coulomb matrix eig 93', 'CoulombMatrix|coulomb matrix eig 113', 'BondFractions|Dy - Dy bond frac.', 'BondFractions|Ge - Ge bond frac.', 'CoulombMatrix|coulomb matrix eig 108', 'SineCoulombMatrix|sine coulomb matrix eig 102', 'CoulombMatrix|coulomb matrix eig 124', 'BondFractions|Bi - O bond frac.', 'BondFractions|Er - Er bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 117', 'SineCoulombMatrix|sine coulomb matrix eig 107', 'SineCoulombMatrix|sine coulomb matrix eig 12', 'BondFractions|Cs - Cs bond frac.', 'XRDPowderPattern|xrd_94', 'Miedema|Miedema_deltaH_amor', 'SineCoulombMatrix|sine coulomb matrix eig 87', 'SineCoulombMatrix|sine coulomb matrix eig 122', 'CoulombMatrix|coulomb matrix eig 79', 'CoulombMatrix|coulomb matrix eig 82', 'SineCoulombMatrix|sine coulomb matrix eig 119', 'CoulombMatrix|coulomb matrix eig 87', 'SineCoulombMatrix|sine coulomb matrix eig 51', 'CoulombMatrix|coulomb matrix eig 27', 'CoulombMatrix|coulomb matrix eig 110', 'XRDPowderPattern|xrd_92', 'BondFractions|As - O bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 84', 'SineCoulombMatrix|sine coulomb matrix eig 13', 'BondFractions|Ag - Ag bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 71', 'SineCoulombMatrix|sine coulomb matrix eig 16', 'BondFractions|In - In bond frac.', 'BondFractions|H - H bond frac.', 'BondFractions|Ce - Ce bond frac.', 'CoulombMatrix|coulomb matrix eig 36', 'CoulombMatrix|coulomb matrix eig 117', 'CoulombMatrix|coulomb matrix eig 67', 'SineCoulombMatrix|sine coulomb matrix eig 69', 'XRDPowderPattern|xrd_64', 'BondFractions|Lu - Lu bond frac.', 'XRDPowderPattern|xrd_74', 'CoulombMatrix|coulomb matrix eig 101', 'SineCoulombMatrix|sine coulomb matrix eig 120', 'SineCoulombMatrix|sine coulomb matrix eig 98', 'CoulombMatrix|coulomb matrix eig 64', 'CoulombMatrix|coulomb matrix eig 41', 'SineCoulombMatrix|sine coulomb matrix eig 127', 'SineCoulombMatrix|sine coulomb matrix eig 72', 'SineCoulombMatrix|sine coulomb matrix eig 18', 'BondFractions|Rb - Rb bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 95', 'BondFractions|Ga - Ga bond frac.', 'CoulombMatrix|coulomb matrix eig 52', 'XRDPowderPattern|xrd_50', 'SineCoulombMatrix|sine coulomb matrix eig 80', 'XRDPowderPattern|xrd_54', 'CoulombMatrix|coulomb matrix eig 75', 'BondFractions|I - I bond frac.', 'BondFractions|Nb - O bond frac.', 'CoulombMatrix|coulomb matrix eig 72', 'SineCoulombMatrix|sine coulomb matrix eig 70', 'SineCoulombMatrix|sine coulomb matrix eig 30', 'XRDPowderPattern|xrd_110', 'CoulombMatrix|coulomb matrix eig 25', 'SineCoulombMatrix|sine coulomb matrix eig 40', 'XRDPowderPattern|xrd_70', 'SineCoulombMatrix|sine coulomb matrix eig 109', 'XRDPowderPattern|xrd_38', 'SineCoulombMatrix|sine coulomb matrix eig 124', 'SineCoulombMatrix|sine coulomb matrix eig 46', 'XRDPowderPattern|xrd_34', 'XRDPowderPattern|xrd_30', 'BondFractions|Mo - O bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 111', 'CoulombMatrix|coulomb matrix eig 53', 'CoulombMatrix|coulomb matrix eig 116', 'SineCoulombMatrix|sine coulomb matrix eig 32', 'XRDPowderPattern|xrd_122', 'SineCoulombMatrix|sine coulomb matrix eig 15', 'CoulombMatrix|coulomb matrix eig 17', 'SineCoulombMatrix|sine coulomb matrix eig 38', 'CoulombMatrix|coulomb matrix eig 118', 'BondFractions|Se - Se bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 28', 'CoulombMatrix|coulomb matrix eig 58', 'CoulombMatrix|coulomb matrix eig 50', 'BondFractions|Eu - Eu bond frac.', 'BondFractions|Ir - Ir bond frac.', 'XRDPowderPattern|xrd_14', 'CoulombMatrix|coulomb matrix eig 103', 'CoulombMatrix|coulomb matrix eig 70', 'SineCoulombMatrix|sine coulomb matrix eig 78', 'BondFractions|Ba - O bond frac.', 'SineCoulombMatrix|sine coulomb matrix eig 97', 'CoulombMatrix|coulomb matrix eig 46', 'BondFractions|La - O bond frac.', 'XRDPowderPattern|xrd_112', 'CoulombMatrix|coulomb matrix eig 34'}\n",
      "2023-01-24 14:44:15,501 - modnet - INFO - Starting target 1/1: refractive_index ...\n",
      "2023-01-24 14:44:15,502 - modnet - INFO - Computing mutual information between features and target...\n",
      "2023-01-24 14:44:38,546 - modnet - INFO - Computing optimal features...\n",
      "2023-01-24 14:45:00,441 - modnet - INFO - Selected 50/200 features...\n",
      "2023-01-24 14:45:21,201 - modnet - INFO - Selected 100/200 features...\n",
      "2023-01-24 14:45:39,151 - modnet - INFO - Selected 150/200 features...\n",
      "2023-01-24 14:45:54,857 - modnet - INFO - Done with target 1/1: refractive_index.\n",
      "2023-01-24 14:45:54,858 - modnet - INFO - Merging all features...\n",
      "2023-01-24 14:45:54,858 - modnet - INFO - Done.\n"
     ]
    }
   ],
   "source": [
    "md.feature_selection(n=200,\n",
    "                    n_samples=500,\n",
    "                    use_precomputed_cross_nmi=True,\n",
    "                    ) # Here we use precomputed cross_nmi to save time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now look at the most important features. This can be done by either looking at *relevant* features (highest target NMI) or *Relevant-nonRedundant (RR)* features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DensityFeatures|density                         0.103335\n",
       "ValenceOrbital|frac p valence electrons         0.087323\n",
       "ElementProperty|MagpieData maximum GSbandgap    0.081005\n",
       "ElementProperty|MagpieData range GSbandgap      0.080530\n",
       "ElementProperty|MagpieData mean NdValence       0.080490\n",
       "ElementProperty|MagpieData mean AtomicWeight    0.079362\n",
       "ValenceOrbital|avg d valence electrons          0.078763\n",
       "ElementProperty|MagpieData mean Number          0.077673\n",
       "ElementProperty|MagpieData mean Row             0.075266\n",
       "ElementProperty|MagpieData avg_dev Number       0.075012\n",
       "Name: refractive_index, dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# relevant features\n",
    "md.target_nmi.nlargest(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['DensityFeatures|density',\n",
       " 'BondFractions|N - N bond frac.',\n",
       " 'OPSiteFingerprint|std_dev pentagonal planar CN_5',\n",
       " 'ElementProperty|MagpieData mode GSbandgap',\n",
       " 'ElementProperty|MagpieData avg_dev Number',\n",
       " 'AGNIFingerPrint|std_dev AGNI dir=x eta=1.23e+00',\n",
       " 'ElementProperty|MagpieData maximum NdValence',\n",
       " 'ElementProperty|MagpieData range Column',\n",
       " 'OPSiteFingerprint|std_dev sgl_bd CN_1',\n",
       " 'AGNIFingerPrint|mean AGNI eta=1.23e+00']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Relevant-unRedundant (RR) features\n",
    "md.get_optimal_descriptors()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (d) Saving the MODData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:45:58,449 - modnet - INFO - Data successfully saved as out/ref_index.mdt!\n"
     ]
    }
   ],
   "source": [
    "md.save('out/ref_index.mdt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. MODNet model\n",
    "The previously created MODData is loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:46:00,153 - modnet - INFO - Loaded <modnet.preprocessing.MODData object at 0x7fa1643f7e50> object, created with modnet version 0.2.0~develop\n"
     ]
    }
   ],
   "source": [
    "md = MODData.load('out/ref_index.mdt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (a) Creating the `MODNetModel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 200)]             0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 128)               25728     \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 64)                8256      \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 32)                2080      \n",
      "                                                                 \n",
      " refractive_index (Dense)    (None, 1)                 33        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 36,097\n",
      "Trainable params: 36,097\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:46:00.252445: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "model = MODNetModel([[['refractive_index']]],{'refractive_index':1},\n",
    "                    n_feat=200, # number of optimal features to use\n",
    "                    num_neurons=[[128],[64],[32],[]], # architecture of the neural network\n",
    "                    act='elu'\n",
    "                   )\n",
    "model.model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (b) Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the user provides hand-chosen hyperparameters and calls fit to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ppdebreuck/anaconda3/envs/modnet-develop/lib/python3.8/site-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(Adam, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0: loss: 0.296, val_loss:0.149 val_mae:0.149\n",
      "epoch 1: loss: 0.139, val_loss:0.152 val_mae:0.152\n",
      "epoch 2: loss: 0.131, val_loss:0.117 val_mae:0.117\n",
      "epoch 3: loss: 0.112, val_loss:0.107 val_mae:0.107\n",
      "epoch 4: loss: 0.105, val_loss:0.109 val_mae:0.109\n",
      "epoch 5: loss: 0.101, val_loss:0.102 val_mae:0.102\n",
      "epoch 6: loss: 0.102, val_loss:0.135 val_mae:0.135\n",
      "epoch 7: loss: 0.105, val_loss:0.108 val_mae:0.108\n",
      "epoch 8: loss: 0.091, val_loss:0.091 val_mae:0.091\n",
      "epoch 9: loss: 0.095, val_loss:0.121 val_mae:0.121\n",
      "epoch 10: loss: 0.097, val_loss:0.090 val_mae:0.090\n",
      "epoch 11: loss: 0.093, val_loss:0.091 val_mae:0.091\n",
      "epoch 12: loss: 0.087, val_loss:0.082 val_mae:0.082\n",
      "epoch 13: loss: 0.079, val_loss:0.104 val_mae:0.104\n",
      "epoch 14: loss: 0.081, val_loss:0.077 val_mae:0.077\n",
      "epoch 15: loss: 0.077, val_loss:0.075 val_mae:0.075\n",
      "epoch 16: loss: 0.074, val_loss:0.081 val_mae:0.081\n",
      "epoch 17: loss: 0.083, val_loss:0.088 val_mae:0.088\n",
      "epoch 18: loss: 0.078, val_loss:0.080 val_mae:0.080\n",
      "epoch 19: loss: 0.075, val_loss:0.079 val_mae:0.079\n",
      "epoch 20: loss: 0.075, val_loss:0.080 val_mae:0.080\n",
      "epoch 21: loss: 0.075, val_loss:0.083 val_mae:0.083\n",
      "epoch 22: loss: 0.069, val_loss:0.081 val_mae:0.081\n",
      "epoch 23: loss: 0.079, val_loss:0.096 val_mae:0.096\n",
      "epoch 24: loss: 0.079, val_loss:0.085 val_mae:0.085\n",
      "epoch 25: loss: 0.071, val_loss:0.073 val_mae:0.073\n",
      "epoch 26: loss: 0.070, val_loss:0.085 val_mae:0.085\n",
      "epoch 27: loss: 0.074, val_loss:0.082 val_mae:0.082\n",
      "epoch 28: loss: 0.068, val_loss:0.075 val_mae:0.075\n",
      "epoch 29: loss: 0.069, val_loss:0.072 val_mae:0.072\n",
      "epoch 30: loss: 0.066, val_loss:0.073 val_mae:0.073\n",
      "epoch 31: loss: 0.064, val_loss:0.074 val_mae:0.074\n",
      "epoch 32: loss: 0.067, val_loss:0.086 val_mae:0.086\n",
      "epoch 33: loss: 0.067, val_loss:0.070 val_mae:0.070\n",
      "epoch 34: loss: 0.061, val_loss:0.070 val_mae:0.070\n",
      "epoch 35: loss: 0.063, val_loss:0.073 val_mae:0.073\n",
      "epoch 36: loss: 0.066, val_loss:0.080 val_mae:0.080\n",
      "epoch 37: loss: 0.061, val_loss:0.072 val_mae:0.072\n",
      "epoch 38: loss: 0.063, val_loss:0.095 val_mae:0.095\n",
      "epoch 39: loss: 0.067, val_loss:0.077 val_mae:0.077\n",
      "epoch 40: loss: 0.064, val_loss:0.067 val_mae:0.067\n",
      "epoch 41: loss: 0.066, val_loss:0.074 val_mae:0.074\n",
      "epoch 42: loss: 0.062, val_loss:0.075 val_mae:0.075\n",
      "epoch 43: loss: 0.059, val_loss:0.083 val_mae:0.083\n",
      "epoch 44: loss: 0.059, val_loss:0.065 val_mae:0.065\n",
      "epoch 45: loss: 0.060, val_loss:0.081 val_mae:0.081\n",
      "epoch 46: loss: 0.064, val_loss:0.068 val_mae:0.068\n",
      "epoch 47: loss: 0.060, val_loss:0.066 val_mae:0.066\n",
      "epoch 48: loss: 0.062, val_loss:0.078 val_mae:0.078\n",
      "epoch 49: loss: 0.060, val_loss:0.076 val_mae:0.076\n",
      "epoch 50: loss: 0.058, val_loss:0.071 val_mae:0.071\n",
      "epoch 51: loss: 0.056, val_loss:0.070 val_mae:0.070\n",
      "epoch 52: loss: 0.060, val_loss:0.070 val_mae:0.070\n",
      "epoch 53: loss: 0.062, val_loss:0.072 val_mae:0.072\n",
      "epoch 54: loss: 0.059, val_loss:0.077 val_mae:0.077\n",
      "epoch 55: loss: 0.056, val_loss:0.071 val_mae:0.071\n",
      "epoch 56: loss: 0.058, val_loss:0.069 val_mae:0.069\n",
      "epoch 57: loss: 0.057, val_loss:0.082 val_mae:0.082\n",
      "epoch 58: loss: 0.056, val_loss:0.066 val_mae:0.066\n",
      "epoch 59: loss: 0.058, val_loss:0.090 val_mae:0.090\n",
      "epoch 60: loss: 0.058, val_loss:0.070 val_mae:0.070\n",
      "epoch 61: loss: 0.053, val_loss:0.072 val_mae:0.072\n",
      "epoch 62: loss: 0.053, val_loss:0.065 val_mae:0.065\n",
      "epoch 63: loss: 0.056, val_loss:0.074 val_mae:0.074\n",
      "epoch 64: loss: 0.055, val_loss:0.078 val_mae:0.078\n",
      "epoch 65: loss: 0.055, val_loss:0.065 val_mae:0.065\n",
      "epoch 66: loss: 0.052, val_loss:0.074 val_mae:0.074\n",
      "epoch 67: loss: 0.056, val_loss:0.068 val_mae:0.068\n",
      "epoch 68: loss: 0.056, val_loss:0.065 val_mae:0.065\n",
      "epoch 69: loss: 0.052, val_loss:0.070 val_mae:0.070\n",
      "epoch 70: loss: 0.053, val_loss:0.068 val_mae:0.068\n",
      "epoch 71: loss: 0.051, val_loss:0.068 val_mae:0.068\n",
      "epoch 72: loss: 0.054, val_loss:0.067 val_mae:0.067\n",
      "epoch 73: loss: 0.056, val_loss:0.064 val_mae:0.064\n",
      "epoch 74: loss: 0.052, val_loss:0.064 val_mae:0.064\n",
      "epoch 75: loss: 0.049, val_loss:0.065 val_mae:0.065\n",
      "epoch 76: loss: 0.052, val_loss:0.066 val_mae:0.066\n",
      "epoch 77: loss: 0.052, val_loss:0.067 val_mae:0.067\n",
      "epoch 78: loss: 0.051, val_loss:0.069 val_mae:0.069\n",
      "epoch 79: loss: 0.050, val_loss:0.069 val_mae:0.069\n",
      "epoch 80: loss: 0.051, val_loss:0.069 val_mae:0.069\n",
      "epoch 81: loss: 0.053, val_loss:0.067 val_mae:0.067\n",
      "epoch 82: loss: 0.052, val_loss:0.077 val_mae:0.077\n",
      "epoch 83: loss: 0.052, val_loss:0.065 val_mae:0.065\n",
      "epoch 84: loss: 0.050, val_loss:0.066 val_mae:0.066\n",
      "epoch 85: loss: 0.048, val_loss:0.069 val_mae:0.069\n",
      "epoch 86: loss: 0.054, val_loss:0.068 val_mae:0.068\n",
      "epoch 87: loss: 0.048, val_loss:0.065 val_mae:0.065\n",
      "epoch 88: loss: 0.048, val_loss:0.063 val_mae:0.063\n",
      "epoch 89: loss: 0.051, val_loss:0.070 val_mae:0.070\n",
      "epoch 90: loss: 0.049, val_loss:0.064 val_mae:0.064\n",
      "epoch 91: loss: 0.047, val_loss:0.068 val_mae:0.068\n",
      "epoch 92: loss: 0.053, val_loss:0.066 val_mae:0.066\n",
      "epoch 93: loss: 0.051, val_loss:0.064 val_mae:0.064\n",
      "epoch 94: loss: 0.048, val_loss:0.062 val_mae:0.062\n",
      "epoch 95: loss: 0.048, val_loss:0.066 val_mae:0.066\n",
      "epoch 96: loss: 0.052, val_loss:0.066 val_mae:0.066\n",
      "epoch 97: loss: 0.051, val_loss:0.065 val_mae:0.065\n",
      "epoch 98: loss: 0.046, val_loss:0.069 val_mae:0.069\n",
      "epoch 99: loss: 0.048, val_loss:0.069 val_mae:0.069\n",
      "epoch 100: loss: 0.045, val_loss:0.072 val_mae:0.072\n",
      "epoch 101: loss: 0.049, val_loss:0.064 val_mae:0.064\n",
      "epoch 102: loss: 0.048, val_loss:0.065 val_mae:0.065\n",
      "epoch 103: loss: 0.046, val_loss:0.068 val_mae:0.068\n",
      "epoch 104: loss: 0.046, val_loss:0.065 val_mae:0.065\n",
      "epoch 105: loss: 0.047, val_loss:0.067 val_mae:0.067\n",
      "epoch 106: loss: 0.044, val_loss:0.068 val_mae:0.068\n",
      "epoch 107: loss: 0.047, val_loss:0.066 val_mae:0.066\n",
      "epoch 108: loss: 0.050, val_loss:0.066 val_mae:0.066\n",
      "epoch 109: loss: 0.044, val_loss:0.066 val_mae:0.066\n",
      "epoch 110: loss: 0.044, val_loss:0.062 val_mae:0.062\n",
      "epoch 111: loss: 0.048, val_loss:0.067 val_mae:0.067\n",
      "epoch 112: loss: 0.046, val_loss:0.065 val_mae:0.065\n",
      "epoch 113: loss: 0.045, val_loss:0.062 val_mae:0.062\n",
      "epoch 114: loss: 0.043, val_loss:0.059 val_mae:0.059\n",
      "epoch 115: loss: 0.042, val_loss:0.064 val_mae:0.064\n",
      "epoch 116: loss: 0.047, val_loss:0.065 val_mae:0.065\n",
      "epoch 117: loss: 0.048, val_loss:0.066 val_mae:0.066\n",
      "epoch 118: loss: 0.048, val_loss:0.071 val_mae:0.071\n",
      "epoch 119: loss: 0.048, val_loss:0.063 val_mae:0.063\n",
      "epoch 120: loss: 0.047, val_loss:0.067 val_mae:0.067\n",
      "epoch 121: loss: 0.048, val_loss:0.071 val_mae:0.071\n",
      "epoch 122: loss: 0.043, val_loss:0.064 val_mae:0.064\n",
      "epoch 123: loss: 0.043, val_loss:0.063 val_mae:0.063\n",
      "epoch 124: loss: 0.046, val_loss:0.064 val_mae:0.064\n",
      "epoch 125: loss: 0.045, val_loss:0.071 val_mae:0.071\n",
      "epoch 126: loss: 0.047, val_loss:0.067 val_mae:0.067\n",
      "epoch 127: loss: 0.046, val_loss:0.063 val_mae:0.063\n",
      "epoch 128: loss: 0.048, val_loss:0.064 val_mae:0.064\n",
      "epoch 129: loss: 0.042, val_loss:0.063 val_mae:0.063\n",
      "epoch 130: loss: 0.042, val_loss:0.064 val_mae:0.064\n",
      "epoch 131: loss: 0.043, val_loss:0.067 val_mae:0.067\n",
      "epoch 132: loss: 0.042, val_loss:0.063 val_mae:0.063\n",
      "epoch 133: loss: 0.045, val_loss:0.061 val_mae:0.061\n",
      "epoch 134: loss: 0.042, val_loss:0.064 val_mae:0.064\n",
      "epoch 135: loss: 0.046, val_loss:0.071 val_mae:0.071\n",
      "epoch 136: loss: 0.044, val_loss:0.066 val_mae:0.066\n",
      "epoch 137: loss: 0.042, val_loss:0.067 val_mae:0.067\n",
      "epoch 138: loss: 0.044, val_loss:0.064 val_mae:0.064\n",
      "epoch 139: loss: 0.042, val_loss:0.062 val_mae:0.062\n",
      "epoch 140: loss: 0.038, val_loss:0.062 val_mae:0.062\n",
      "epoch 141: loss: 0.041, val_loss:0.064 val_mae:0.064\n",
      "epoch 142: loss: 0.050, val_loss:0.075 val_mae:0.075\n",
      "epoch 143: loss: 0.046, val_loss:0.062 val_mae:0.062\n",
      "epoch 144: loss: 0.039, val_loss:0.066 val_mae:0.066\n",
      "epoch 145: loss: 0.044, val_loss:0.076 val_mae:0.076\n",
      "epoch 146: loss: 0.043, val_loss:0.064 val_mae:0.064\n",
      "epoch 147: loss: 0.041, val_loss:0.073 val_mae:0.073\n",
      "epoch 148: loss: 0.042, val_loss:0.059 val_mae:0.059\n",
      "epoch 149: loss: 0.041, val_loss:0.076 val_mae:0.076\n",
      "epoch 150: loss: 0.042, val_loss:0.065 val_mae:0.065\n",
      "epoch 151: loss: 0.041, val_loss:0.067 val_mae:0.067\n",
      "epoch 152: loss: 0.040, val_loss:0.060 val_mae:0.060\n",
      "epoch 153: loss: 0.041, val_loss:0.073 val_mae:0.073\n",
      "epoch 154: loss: 0.041, val_loss:0.060 val_mae:0.060\n",
      "epoch 155: loss: 0.038, val_loss:0.063 val_mae:0.063\n",
      "epoch 156: loss: 0.042, val_loss:0.063 val_mae:0.063\n",
      "epoch 157: loss: 0.045, val_loss:0.066 val_mae:0.066\n",
      "epoch 158: loss: 0.042, val_loss:0.063 val_mae:0.063\n",
      "epoch 159: loss: 0.039, val_loss:0.062 val_mae:0.062\n",
      "epoch 160: loss: 0.038, val_loss:0.065 val_mae:0.065\n",
      "epoch 161: loss: 0.039, val_loss:0.060 val_mae:0.060\n",
      "epoch 162: loss: 0.040, val_loss:0.062 val_mae:0.062\n",
      "epoch 163: loss: 0.040, val_loss:0.061 val_mae:0.061\n",
      "epoch 164: loss: 0.039, val_loss:0.060 val_mae:0.060\n",
      "epoch 165: loss: 0.042, val_loss:0.063 val_mae:0.063\n",
      "epoch 166: loss: 0.038, val_loss:0.059 val_mae:0.059\n",
      "epoch 167: loss: 0.039, val_loss:0.061 val_mae:0.061\n",
      "epoch 168: loss: 0.038, val_loss:0.064 val_mae:0.064\n",
      "epoch 169: loss: 0.037, val_loss:0.064 val_mae:0.064\n",
      "epoch 170: loss: 0.037, val_loss:0.061 val_mae:0.061\n",
      "epoch 171: loss: 0.043, val_loss:0.059 val_mae:0.059\n",
      "epoch 172: loss: 0.040, val_loss:0.060 val_mae:0.060\n",
      "epoch 173: loss: 0.041, val_loss:0.064 val_mae:0.064\n",
      "epoch 174: loss: 0.038, val_loss:0.062 val_mae:0.062\n",
      "epoch 175: loss: 0.038, val_loss:0.064 val_mae:0.064\n",
      "epoch 176: loss: 0.041, val_loss:0.060 val_mae:0.060\n",
      "epoch 177: loss: 0.037, val_loss:0.060 val_mae:0.060\n",
      "epoch 178: loss: 0.040, val_loss:0.059 val_mae:0.059\n",
      "epoch 179: loss: 0.036, val_loss:0.060 val_mae:0.060\n",
      "epoch 180: loss: 0.036, val_loss:0.059 val_mae:0.059\n",
      "epoch 181: loss: 0.041, val_loss:0.063 val_mae:0.063\n",
      "epoch 182: loss: 0.038, val_loss:0.060 val_mae:0.060\n",
      "epoch 183: loss: 0.037, val_loss:0.062 val_mae:0.062\n",
      "epoch 184: loss: 0.040, val_loss:0.065 val_mae:0.065\n",
      "epoch 185: loss: 0.039, val_loss:0.061 val_mae:0.061\n",
      "epoch 186: loss: 0.038, val_loss:0.059 val_mae:0.059\n",
      "epoch 187: loss: 0.043, val_loss:0.069 val_mae:0.069\n",
      "epoch 188: loss: 0.037, val_loss:0.063 val_mae:0.063\n",
      "epoch 189: loss: 0.039, val_loss:0.065 val_mae:0.065\n",
      "epoch 190: loss: 0.038, val_loss:0.061 val_mae:0.061\n",
      "epoch 191: loss: 0.036, val_loss:0.061 val_mae:0.061\n",
      "epoch 192: loss: 0.035, val_loss:0.064 val_mae:0.064\n",
      "epoch 193: loss: 0.037, val_loss:0.062 val_mae:0.062\n",
      "epoch 194: loss: 0.036, val_loss:0.063 val_mae:0.063\n",
      "epoch 195: loss: 0.037, val_loss:0.060 val_mae:0.060\n",
      "epoch 196: loss: 0.040, val_loss:0.067 val_mae:0.067\n",
      "epoch 197: loss: 0.037, val_loss:0.070 val_mae:0.070\n",
      "epoch 198: loss: 0.038, val_loss:0.067 val_mae:0.067\n",
      "epoch 199: loss: 0.039, val_loss:0.064 val_mae:0.064\n",
      "epoch 200: loss: 0.038, val_loss:0.059 val_mae:0.059\n",
      "epoch 201: loss: 0.035, val_loss:0.061 val_mae:0.061\n",
      "epoch 202: loss: 0.037, val_loss:0.066 val_mae:0.066\n",
      "epoch 203: loss: 0.039, val_loss:0.074 val_mae:0.074\n",
      "epoch 204: loss: 0.041, val_loss:0.061 val_mae:0.061\n",
      "epoch 205: loss: 0.033, val_loss:0.058 val_mae:0.058\n",
      "epoch 206: loss: 0.033, val_loss:0.059 val_mae:0.059\n",
      "epoch 207: loss: 0.035, val_loss:0.059 val_mae:0.059\n",
      "epoch 208: loss: 0.037, val_loss:0.060 val_mae:0.060\n",
      "epoch 209: loss: 0.038, val_loss:0.058 val_mae:0.058\n",
      "epoch 210: loss: 0.033, val_loss:0.060 val_mae:0.060\n",
      "epoch 211: loss: 0.035, val_loss:0.059 val_mae:0.059\n",
      "epoch 212: loss: 0.035, val_loss:0.062 val_mae:0.062\n",
      "epoch 213: loss: 0.033, val_loss:0.063 val_mae:0.063\n",
      "epoch 214: loss: 0.033, val_loss:0.060 val_mae:0.060\n",
      "epoch 215: loss: 0.036, val_loss:0.059 val_mae:0.059\n",
      "epoch 216: loss: 0.035, val_loss:0.064 val_mae:0.064\n",
      "epoch 217: loss: 0.035, val_loss:0.061 val_mae:0.061\n",
      "epoch 218: loss: 0.036, val_loss:0.064 val_mae:0.064\n",
      "epoch 219: loss: 0.035, val_loss:0.057 val_mae:0.057\n",
      "epoch 220: loss: 0.032, val_loss:0.063 val_mae:0.063\n",
      "epoch 221: loss: 0.035, val_loss:0.061 val_mae:0.061\n",
      "epoch 222: loss: 0.033, val_loss:0.060 val_mae:0.060\n",
      "epoch 223: loss: 0.033, val_loss:0.059 val_mae:0.059\n",
      "epoch 224: loss: 0.034, val_loss:0.060 val_mae:0.060\n",
      "epoch 225: loss: 0.035, val_loss:0.066 val_mae:0.066\n",
      "epoch 226: loss: 0.033, val_loss:0.062 val_mae:0.062\n",
      "epoch 227: loss: 0.034, val_loss:0.063 val_mae:0.063\n",
      "epoch 228: loss: 0.036, val_loss:0.058 val_mae:0.058\n",
      "epoch 229: loss: 0.034, val_loss:0.060 val_mae:0.060\n",
      "epoch 230: loss: 0.035, val_loss:0.059 val_mae:0.059\n",
      "epoch 231: loss: 0.033, val_loss:0.056 val_mae:0.056\n",
      "epoch 232: loss: 0.033, val_loss:0.062 val_mae:0.062\n",
      "epoch 233: loss: 0.034, val_loss:0.059 val_mae:0.059\n",
      "epoch 234: loss: 0.037, val_loss:0.066 val_mae:0.066\n",
      "epoch 235: loss: 0.035, val_loss:0.061 val_mae:0.061\n",
      "epoch 236: loss: 0.033, val_loss:0.057 val_mae:0.057\n",
      "epoch 237: loss: 0.033, val_loss:0.069 val_mae:0.069\n",
      "epoch 238: loss: 0.042, val_loss:0.060 val_mae:0.060\n",
      "epoch 239: loss: 0.032, val_loss:0.062 val_mae:0.062\n",
      "epoch 240: loss: 0.031, val_loss:0.064 val_mae:0.064\n",
      "epoch 241: loss: 0.034, val_loss:0.059 val_mae:0.059\n",
      "epoch 242: loss: 0.036, val_loss:0.061 val_mae:0.061\n",
      "epoch 243: loss: 0.031, val_loss:0.062 val_mae:0.062\n",
      "epoch 244: loss: 0.036, val_loss:0.062 val_mae:0.062\n",
      "epoch 245: loss: 0.037, val_loss:0.059 val_mae:0.059\n",
      "epoch 246: loss: 0.037, val_loss:0.076 val_mae:0.076\n",
      "epoch 247: loss: 0.038, val_loss:0.060 val_mae:0.060\n",
      "epoch 248: loss: 0.032, val_loss:0.063 val_mae:0.063\n",
      "epoch 249: loss: 0.036, val_loss:0.059 val_mae:0.059\n",
      "epoch 250: loss: 0.034, val_loss:0.061 val_mae:0.061\n",
      "epoch 251: loss: 0.032, val_loss:0.059 val_mae:0.059\n",
      "epoch 252: loss: 0.031, val_loss:0.062 val_mae:0.062\n",
      "epoch 253: loss: 0.033, val_loss:0.063 val_mae:0.063\n",
      "epoch 254: loss: 0.037, val_loss:0.058 val_mae:0.058\n",
      "epoch 255: loss: 0.031, val_loss:0.054 val_mae:0.054\n",
      "epoch 256: loss: 0.030, val_loss:0.059 val_mae:0.059\n",
      "epoch 257: loss: 0.030, val_loss:0.059 val_mae:0.059\n",
      "epoch 258: loss: 0.032, val_loss:0.059 val_mae:0.059\n",
      "epoch 259: loss: 0.031, val_loss:0.058 val_mae:0.058\n",
      "epoch 260: loss: 0.032, val_loss:0.062 val_mae:0.062\n",
      "epoch 261: loss: 0.034, val_loss:0.056 val_mae:0.056\n",
      "epoch 262: loss: 0.032, val_loss:0.058 val_mae:0.058\n",
      "epoch 263: loss: 0.031, val_loss:0.060 val_mae:0.060\n",
      "epoch 264: loss: 0.036, val_loss:0.064 val_mae:0.064\n",
      "epoch 265: loss: 0.034, val_loss:0.054 val_mae:0.054\n",
      "epoch 266: loss: 0.031, val_loss:0.058 val_mae:0.058\n",
      "epoch 267: loss: 0.032, val_loss:0.057 val_mae:0.057\n",
      "epoch 268: loss: 0.031, val_loss:0.062 val_mae:0.062\n",
      "epoch 269: loss: 0.031, val_loss:0.060 val_mae:0.060\n",
      "epoch 270: loss: 0.032, val_loss:0.056 val_mae:0.056\n",
      "epoch 271: loss: 0.030, val_loss:0.058 val_mae:0.058\n",
      "epoch 272: loss: 0.035, val_loss:0.059 val_mae:0.059\n",
      "epoch 273: loss: 0.031, val_loss:0.063 val_mae:0.063\n",
      "epoch 274: loss: 0.032, val_loss:0.056 val_mae:0.056\n",
      "epoch 275: loss: 0.030, val_loss:0.060 val_mae:0.060\n",
      "epoch 276: loss: 0.033, val_loss:0.070 val_mae:0.070\n",
      "epoch 277: loss: 0.034, val_loss:0.056 val_mae:0.056\n",
      "epoch 278: loss: 0.030, val_loss:0.057 val_mae:0.057\n",
      "epoch 279: loss: 0.029, val_loss:0.058 val_mae:0.058\n",
      "epoch 280: loss: 0.031, val_loss:0.056 val_mae:0.056\n",
      "epoch 281: loss: 0.029, val_loss:0.062 val_mae:0.062\n",
      "epoch 282: loss: 0.028, val_loss:0.063 val_mae:0.063\n",
      "epoch 283: loss: 0.032, val_loss:0.055 val_mae:0.055\n",
      "epoch 284: loss: 0.030, val_loss:0.059 val_mae:0.059\n",
      "epoch 285: loss: 0.029, val_loss:0.058 val_mae:0.058\n",
      "epoch 286: loss: 0.028, val_loss:0.056 val_mae:0.056\n",
      "epoch 287: loss: 0.029, val_loss:0.055 val_mae:0.055\n",
      "epoch 288: loss: 0.034, val_loss:0.057 val_mae:0.057\n",
      "epoch 289: loss: 0.029, val_loss:0.056 val_mae:0.056\n",
      "epoch 290: loss: 0.029, val_loss:0.058 val_mae:0.058\n",
      "epoch 291: loss: 0.029, val_loss:0.059 val_mae:0.059\n",
      "epoch 292: loss: 0.029, val_loss:0.063 val_mae:0.063\n",
      "epoch 293: loss: 0.030, val_loss:0.061 val_mae:0.061\n",
      "epoch 294: loss: 0.031, val_loss:0.058 val_mae:0.058\n",
      "epoch 295: loss: 0.031, val_loss:0.056 val_mae:0.056\n",
      "epoch 296: loss: 0.028, val_loss:0.056 val_mae:0.056\n",
      "epoch 297: loss: 0.031, val_loss:0.061 val_mae:0.061\n",
      "epoch 298: loss: 0.031, val_loss:0.061 val_mae:0.061\n",
      "epoch 299: loss: 0.030, val_loss:0.056 val_mae:0.056\n"
     ]
    }
   ],
   "source": [
    "model.fit(md,val_fraction=0.1,\n",
    "          val_key='refractive_index',\n",
    "          loss='mae', lr=0.001, epochs = 300,\n",
    "          batch_size = 64, xscale='minmax',\n",
    "          yscale=None,\n",
    "          verbose=1\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fit the model a second time with a lower learning rate to fine-tune the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ppdebreuck/anaconda3/envs/modnet-develop/lib/python3.8/site-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(Adam, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0: loss: 0.002, val_loss:0.020 val_mae:0.058\n",
      "epoch 1: loss: 0.002, val_loss:0.019 val_mae:0.056\n",
      "epoch 2: loss: 0.001, val_loss:0.020 val_mae:0.056\n",
      "epoch 3: loss: 0.002, val_loss:0.019 val_mae:0.055\n",
      "epoch 4: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 5: loss: 0.001, val_loss:0.020 val_mae:0.056\n",
      "epoch 6: loss: 0.001, val_loss:0.019 val_mae:0.054\n",
      "epoch 7: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 8: loss: 0.002, val_loss:0.020 val_mae:0.056\n",
      "epoch 9: loss: 0.002, val_loss:0.019 val_mae:0.058\n",
      "epoch 10: loss: 0.002, val_loss:0.018 val_mae:0.054\n",
      "epoch 11: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 12: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 13: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 14: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 15: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 16: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 17: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 18: loss: 0.001, val_loss:0.019 val_mae:0.059\n",
      "epoch 19: loss: 0.002, val_loss:0.020 val_mae:0.056\n",
      "epoch 20: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 21: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 22: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 23: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 24: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 25: loss: 0.001, val_loss:0.018 val_mae:0.056\n",
      "epoch 26: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 27: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 28: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 29: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 30: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 31: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 32: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 33: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 34: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 35: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 36: loss: 0.001, val_loss:0.018 val_mae:0.056\n",
      "epoch 37: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 38: loss: 0.001, val_loss:0.019 val_mae:0.059\n",
      "epoch 39: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 40: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 41: loss: 0.001, val_loss:0.019 val_mae:0.059\n",
      "epoch 42: loss: 0.001, val_loss:0.019 val_mae:0.058\n",
      "epoch 43: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 44: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 45: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 46: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 47: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 48: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 49: loss: 0.001, val_loss:0.019 val_mae:0.058\n",
      "epoch 50: loss: 0.001, val_loss:0.018 val_mae:0.057\n",
      "epoch 51: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 52: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 53: loss: 0.001, val_loss:0.020 val_mae:0.059\n",
      "epoch 54: loss: 0.001, val_loss:0.020 val_mae:0.057\n",
      "epoch 55: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 56: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 57: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 58: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 59: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 60: loss: 0.001, val_loss:0.018 val_mae:0.056\n",
      "epoch 61: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 62: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 63: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 64: loss: 0.001, val_loss:0.020 val_mae:0.057\n",
      "epoch 65: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 66: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 67: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 68: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 69: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 70: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 71: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 72: loss: 0.001, val_loss:0.019 val_mae:0.058\n",
      "epoch 73: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 74: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 75: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 76: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 77: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 78: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 79: loss: 0.001, val_loss:0.018 val_mae:0.056\n",
      "epoch 80: loss: 0.001, val_loss:0.018 val_mae:0.055\n",
      "epoch 81: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 82: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 83: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 84: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 85: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 86: loss: 0.001, val_loss:0.018 val_mae:0.056\n",
      "epoch 87: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 88: loss: 0.001, val_loss:0.019 val_mae:0.056\n",
      "epoch 89: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 90: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 91: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 92: loss: 0.001, val_loss:0.019 val_mae:0.057\n",
      "epoch 93: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 94: loss: 0.001, val_loss:0.019 val_mae:0.059\n",
      "epoch 95: loss: 0.001, val_loss:0.021 val_mae:0.058\n",
      "epoch 96: loss: 0.001, val_loss:0.019 val_mae:0.055\n",
      "epoch 97: loss: 0.001, val_loss:0.020 val_mae:0.056\n",
      "epoch 98: loss: 0.001, val_loss:0.020 val_mae:0.058\n",
      "epoch 99: loss: 0.001, val_loss:0.020 val_mae:0.057\n"
     ]
    }
   ],
   "source": [
    "# we fit a secno\n",
    "model.fit(md,\n",
    "          val_fraction=0.1,\n",
    "          val_key='refractive_index',\n",
    "          lr=0.0005,\n",
    "          epochs = 100,\n",
    "          batch_size = 128,\n",
    "          xscale='minmax',\n",
    "          yscale=None,\n",
    "          verbose=1\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Saving the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-01-24 14:46:37,092 - modnet - INFO - Model successfully saved as out/MODNet_refractive_index!\n"
     ]
    }
   ],
   "source": [
    "model.save('out/MODNet_refractive_index')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Predicting on unseen data\n",
    "\n",
    "See \"2-predicting_ref_index\" notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (modnet-develop)",
   "language": "python",
   "name": "modnet-develop"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}