Notes2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JAX Learning Notes\n",
    "--------------------\n",
    "# Tutorial JAX 101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JAX As Accelerated Numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x2e5817b3460>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6t0lEQVR4nO3dfXSU9Z3//9cESSKaDHeBCRi5bcU0KoICQdcCYqFaKt96PF2UIhZRWXBVWBforxaBY6OLPb1RF6prQQ9S2+5XRdSyX0DURUOxUKqAsCXLjYYENYEZDCXEZH5/xImZZG6umbmuua655vk4Z84xk+ua+Uw8dd79fN43nmAwGBQAAICL5Ni9AAAAALMR4AAAANchwAEAAK5DgAMAAFyHAAcAALgOAQ4AAHAdAhwAAOA6BDgAAMB1zrF7AXZoaWnRsWPHVFBQII/HY/dyAACAAcFgUKdOnVK/fv2UkxN7jyYrA5xjx46ppKTE7mUAAIAkfPTRR7rgggtiXpOVAU5BQYGk1j9QYWGhzasBAABGBAIBlZSUtH2Px5KVAU7oWKqwsJAABwCADGMkvcT2JOOBAwfK4/F0esydOzfi9WvWrOl0bX5+fppXDQAAnMz2HZz33ntPzc3NbT/v2bNH1113nW6++eao9xQWFurAgQNtP5MoDAAA2rM9wCkqKgr7+ZFHHtGQIUP0zW9+M+o9Ho9HPp/P6qUBAIAMZfsRVXtnz57V2rVr9cMf/jDmrsznn3+uAQMGqKSkRDfeeKP27t0b83UbGxsVCATCHgAAwL0cFeC8/PLLOnnypGbOnBn1mosuuki/+c1vtH79eq1du1YtLS0aO3asPv7446j3VFRUyOv1tj0oEQcAwN08wWAwaPciQiZNmqTc3Fxt2LDB8D1NTU26+OKLNW3aNC1fvjziNY2NjWpsbGz7OVRm5vf7qaICACBDBAIBeb1eQ9/ftufghBw5ckSbN2/Wiy++mNB9Xbt21eWXX66DBw9GvSYvL095eXmpLhEAAGQIxxxRrV69Wn369NENN9yQ0H3Nzc364IMPVFxcbNHKAABApnHEDk5LS4tWr16t2267TeecE76kGTNmqH///qqoqJAkLVu2TGPGjNHQoUN18uRJrVixQkeOHNEdd9xhx9IBAEA7zS1B7ThUr09OnVGfgnyNGtRTXXLS387FEQHO5s2bdfToUf3whz/s9LujR4+GDdQ6ceKEZs+erdraWvXo0UMjR47Uu+++q9LS0nQuGQAAdLBxT42WbtinGv+ZtueKvflaMqVUk8vSe9LiqCTjdEkkSQkAAMS3cU+N5qzdpY5BRWjvZuX0ESkHOYl8fzsmBwcAAGSm5paglm7Y1ym4kdT23NIN+9Tckr49FQIcAACQkh2H6sOOpToKSqrxn9GOQ/VpWxMBDgAASMknp6IHN8lcZwYCHAAAkJI+BfmmXmcGAhwAAJCSUYN6qtibr2jF4B61VlONGtQzbWsiwAEAACnpkuPRkimt7Vo6Bjmhn5dMKU1rPxwCHAAAkLLJZcVaOX2EfN7wYyifN9+UEvFEOaLRHwAAyHyTy4p1XamPTsYAAMBduuR4VD6kl93L4IgKAAC4DwEOAABwHY6oAACwmVMmcLsJAQ4AADZy0gRuN+GICgAAm4QmcHec41TrP6M5a3dp454am1aW+QhwAACwgRMncLsJAQ4AADZw4gRuNyHAAQDABk6cwO0mBDgAANjAiRO43YQABwAAGzhxArebEOAAAGADJ07gdhMCHAAAbOK0CdxuQqM/AABs5KQJ3G5CgAMAgM2cMoHbTTiiAgAArkOAAwAAXIcABwAAuA4BDgAAcB0CHAAA4DoEOAAAwHUIcAAAgOsQ4AAAANchwAEAAK5DgAMAAFyHAAcAALgOAQ4AAHAdAhwAAOA6BDgAAMB1CHAAAIDrnGP3AgAAMENzS1A7DtXrk1Nn1KcgX6MG9VSXHI/dy4JNCHAAABlv454aLd2wTzX+M23PFXvztWRKqSaXFdu4MtjF9iOqhx56SB6PJ+wxbNiwmPf84Q9/0LBhw5Sfn69LLrlEr7/+eppWCwBwmo17ajRn7a6w4EaSav1nNGftLm3cU2PTymAn2wMcSfrGN76hmpqatse2bduiXvvuu+9q2rRpmjVrlv7yl79o6tSpmjp1qvbs2ZPGFQMAnKC5JailG/YpGOF3oeeWbtin5pZIV8DNHHFEdc4558jn8xm69pe//KUmT56sBx54QJK0fPlybdq0SU888YRWrVpl5TIBAA7QPtfms1ONnXZu2gtKqvGf0Y5D9Sof0suSNZDv40yOCHD+9re/qV+/fsrPz1d5ebkqKip04YUXRry2srJS8+fPD3tu0qRJevnll9OwUgCAnSLl2hjxyanErk90DeT7OI/tR1SjR4/WmjVrtHHjRq1cuVKHDh3SP/zDP+jUqVMRr6+trVXfvn3Dnuvbt69qa2ujvkdjY6MCgUDYAwCQWaLl2hjRpyDf0jWQ7+M8tgc43/72t3XzzTfr0ksv1aRJk/T666/r5MmT+v3vf2/ae1RUVMjr9bY9SkpKTHttAID1YuXaxOJR6+7KqEE9LV0D+T7OY3uA01H37t319a9/XQcPHoz4e5/Pp+PHj4c9d/z48Zg5PIsXL5bf7297fPTRR6auGQBgrR2H6hPeuQllxCyZUmpKfky8NbTP94H9HBfgfP7556qqqlJxceRzzPLycm3ZsiXsuU2bNqm8vDzqa+bl5amwsDDsAQDIHMnk0Pi8+Vo5fYRpeTFG12Bmvg+SZ3uS8b/8y79oypQpGjBggI4dO6YlS5aoS5cumjZtmiRpxowZ6t+/vyoqKiRJ9957r775zW/qZz/7mW644Qa98MIL+vOf/6ynnnrKzo8BALCQ0RyaB2+4WL0L8iypbDK6BrPyfZAa2wOcjz/+WNOmTVNdXZ2Kiop09dVXa/v27SoqKpIkHT16VDk5X200jR07VuvWrdOPf/xj/ehHP9LXvvY1vfzyyyorK7PrIwAADEiltHrUoJ4q9uar1n8mYg6MR607NjOvGmRZubbRNZiR74PUeYLBYNZlQwUCAXm9Xvn9fo6rACANzCitDlUwSQoLMELhjJnHUU5eQzZL5PvbcTk4AAB3Mau0enJZsVZOHyGfN/wIyOxcG6evAcawg8MODgBYprklqKsffSNq9VHoWGfbwgmGj5ac0EXYCWvIRol8f9uegwMAcK9ESquNjlLokuMxdexCMpywBsTGERUAwDKUVsMuBDgAAMtQWg27EOAAACwTKq2Olp1i5igFoD0CHACAZbrkeLRkSqkkdQpyzB6lALRHgAMAsBSl1bADVVQAAMtNLivWdaU+SquRNgQ4AIC0SLW0mt4zSAQBDgDA8cwY9YDsQg4OAMDRzBr1gOxCgAMAcKzmlqCWbtgXcXp36LmlG/apuSXrpg4hDgIcAIBjJTLqAWiPAAcA4FiMekCyCHAAAI7FqAckiwAHAOBYjHpAsghwAACOZfaoh+aWoCqr6rR+d7Uqq+pITnYxTzAYzLp/u4FAQF6vV36/X4WFhXYvBwAQR7Q+OA/eUKoe5+Uaav5HL53Ml8j3NwEOAQ4ApE0q3Yg73nuioVHLX/vQUMAS6qXT8Qsv9M7MxMoMBDhxEOAAyDbpHnMQ6f027as1bQclkYCluSWoqx99I2q5uUetgz+3LZzA6AeHS+T7m1ENAOBy6T6aifR+3bt11cnTTZ2uDXUjTmQHxWjzv+tKfeqS40mol04qs7LgLCQZA4CLpXvMQbT3ixTcSMl1I95eVRczYJHCm//RSyc7EeAAgEule8xBrPeLJZFuxBv31Gjuul2GXnfTvlpJ9NLJVgQ4AOBS6R5zEO/94om3gxLaHTr598i7QR2t331MzS1BeulkKQIcAHCpdB/NpPo6sXZQktkdqms4qx2H6k3vpYPMQIADAC6V7qOZZF/HI8lXmKeWYDBqA75kd4dCQdfksmKtnD5CPm/4Gn3efErEXYoqKgBwqdDRTK3/TMSdj1B5tFlHM/HeLxKPWo/KznzRolv/409tz3es8kp2d6h90DW5rFjXlfrSWi4P+7CDAwAule6jGSPv171b17DnvV/+3LHKqsZ/Rnev3aVfbv6bmluCCe8ORcur6ZLjUfmQXrpxeH+VD+lFcONiBDgA4GLpPpqJ9X6rpo/Qzh9fp9/OHqNf/uNwPT9rtPLP6RLz9X6++X901SNbdKKhMWaicHvk1UCikzGdjAFkBSd0Mu74fpVVdZr29HZDr+eRdOc1g/TU24ckKeYRGPOl3ItOxgCAMKGjGSe9X6J5Na/8tUZP3jJCy1/r3JX5H6+8UAN7dyOvBm0IcAAAtkgkrybUs6fHebnatnACicKIiwAHAGCLZKquPjl1Ju27UchMJBkDAGzRvurKKMYpwCgCHACAbdqqrgpjBy6MU0CiCHAAALaaXFasdxZN0P0Tvxbx95R9IxkEOAAA23XJ8ejeiV/XqukjVMw4BZiAJGMAyBLp7oWTDMYpwCwEOACQBTbuqdHSDZ37xzixIR5VUjADR1QA4HIb99RoztpdnaZx1/rPaM7aXdq4p8amlQHWsT3Aqaio0JVXXqmCggL16dNHU6dO1YEDB2Les2bNGnk8nrBHfj6lgwDQUXNLUEs37IvYZyb03NIN+9TcknVTe+Bytgc4b731lubOnavt27dr06ZNampq0re+9S01NDTEvK+wsFA1NTVtjyNHjqRpxQCQOXYcqu+0c9NeqEPwjkP16VsUkAa25+Bs3Lgx7Oc1a9aoT58+2rlzp6655pqo93k8Hvl8PquXBwAZzei8p0TnQgFOZ/sOTkd+v1+S1LNn7GZOn3/+uQYMGKCSkhLdeOON2rt3b9RrGxsbFQgEwh4A4FTNLUFVVtVp/e5qVVbVpXR8ZLTzLx2C4Ta27+C019LSovvuu09XXXWVysrKol530UUX6Te/+Y0uvfRS+f1+PfbYYxo7dqz27t2rCy64oNP1FRUVWrp0qZVLB4CERCvZNrvaKd68J49a+8zQIRhu4wkGg47JLJszZ47++Mc/atu2bREDlWiampp08cUXa9q0aVq+fHmn3zc2NqqxsbHt50AgoJKSEvn9fhUWFpqydgAwKloQ893LivXU24c6BSKhDjDJNrsLVVFJCnvtVF8XSLdAICCv12vo+9sxR1Tz5s3Tq6++qq1btyYU3EhS165ddfnll+vgwYMRf5+Xl6fCwsKwBwDYIVbJ9q8jBDdS6tVObfOe6BCMLGL7EVUwGNQ999yjl156SW+++aYGDRqU8Gs0Nzfrgw8+0PXXX2/BCgFkC6s7/Rop2Y6mfbVTMk3w6BCMbGN7gDN37lytW7dO69evV0FBgWprayVJXq9X5557riRpxowZ6t+/vyoqKiRJy5Yt05gxYzR06FCdPHlSK1as0JEjR3THHXfY9jkAZDYjuS+pBkDxSraNSKXaiQ7ByCa2BzgrV66UJI0bNy7s+dWrV2vmzJmSpKNHjyon56vTtBMnTmj27Nmqra1Vjx49NHLkSL377rsqLS1N17IBuEjo2KjjLkqo0+/K6SMkKeXkXzNKsal2AoxxVJJxuiSSpATA3Zpbgrr60Tei7qx4JHm7dZX/dFPKyb+VVXWa9vT2pNYZqnbatnBCSsdKmTBwE4gmke9v23dwAMBORjr9njzdFPV3HrXu7FxX6osbKMQr2Y4m9KpLppSmFIxk0sBNIFWOqaICADukemyUyKiDLjkeLZnSepTeMUzxfPm465pBKu5Q7dTzvFw9eUtq1U4M3ES2IcABkNXMymkxGijFK9lefH2p/r9vD1NBfpe239U1nNXy1/YlHYQwcBPZiCMqAFkt2WOjjhIJlGKVbFe8vk+/fvtQp3tq/Gd099pd+uFVA3VdqS8sdyZeXk0iAzepsoJbEOAAyGqhY6M5a3fJo86dfoOSukdJMg5dk8yog0gl26+/XxMxuGnvN+8c1m/eOdyWOyPFr+5i4CayEUdUALJerGOjVdNH6JHvXSIpct6MlHryr9S6C/Pj9XsMX1/75Y7O3Qbyahi4iWzEDg4AKH6n35XTR3TaKfGZWIG041C96hvOGr4+1nFax+ouBm4iGxHgAMCXYnX6tXrUgdnHQx3zamIdw0nm7EIBTkKAAwAGWTnqwKrjoVDgFDqGs3IXCnASAhwAcIDQMVKqs6o6ah84MXAT2YQABwAcoH01lxndaKLl1TBwE9mCKioAaKe5JajKqjqt312tyqq6tDa/Cx0jdexkHIknyj+3/5m8GmQzdnAA4EtOmNUU6RjpREOjlr/2YcTcGalzHxzyagCmiTNNHICkr2Y1pTox3CqxuhUzIRzZgmniAJCAeLOaEpkYbpVYuTPk1QCdEeAAyHpOmdWUTTsx2fRZYQ8CHABZzwmzmpyQ/5Mu2fRZYR+qqABkPbtnNYXyf+LNlHKDbPqssBcBDoCsF2qyF+2AxKPWHQYrZjXFy/+RWvN/0lmubpVs+qywHwEOgKwXarInde4pE5JIT5lEeukkkv+T6bLps8J+5OAAgL5qsrfoxQ908nRT2O+83boafp1E80uM5vW8c/CzjE/EdUKuE7IHOzgA0E7H4EaS/KebDOWHJJNfYjSv54mtB3X1o29kdI6K3blOyC4EOACgr/JDIjGSH5Jsfkm8/J/2Mj0R185cJ2QfAhwAUOr5IcnebyT/p/1rSJmbiBvrszI/C2YjwAEApZ4fksr9ofwfn4Ehm5meiBvts/q8+baPw4C7kGQMAEo9PyTV+0NDNn++6YCe2FoV93UyORE30kDRTE+ghvOwgwMASj0/xIz8ki45Hl01tMjQej871WioDN2pQvOzbhzeX+VDehHcwHQEOACg1PNDzMovMZJ0nOORlr/2oe59YbemPb0946urACsQ4ADAl1LNDzEjv8RI0nHHDZtMr64CrOAJBoOZt7eZokAgIK/XK7/fr8LCQruXAyANEpleneqkazMmZUdqGJjj6RzchHjUGkhtWziB4x64ViLf3wQ4BDiA62Xq9Or2gdJnpxq1/LUP497z29ljVD6kVxpWB6RfIt/fHFEBcLVMnl7dPhG3d0GeoXsyuboKMBMBDgDXctP0asYcAIkhwAHgWm6aXs2YAyAxBDgAXMtN06sZcwAkhgAHgGu57ViHMQeAcYxqAOAIZpRWdxQ61qn1n4mYhxMqrc6kYx3GHADGEOAAsJ1VZdyhY505a3fJI4UFOZl8rBOqrgIQHUdUAGxldRk3xzpAdmIHB4Bt4pVxe9Raxn1dqS+lXRaOdYDs44gdnCeffFIDBw5Ufn6+Ro8erR07dsS8/g9/+IOGDRum/Px8XXLJJXr99dfTtFIAZkpnGTfTq4HsYnuA87vf/U7z58/XkiVLtGvXLl122WWaNGmSPvnkk4jXv/vuu5o2bZpmzZqlv/zlL5o6daqmTp2qPXv2pHnlAFJlVxl3c0tQlVV1Wr+7WpVVdUk1+jPjNQBYx/ZZVKNHj9aVV16pJ554QpLU0tKikpIS3XPPPVq0aFGn67///e+roaFBr776attzY8aM0fDhw7Vq1SpD78ksKsAZKqvqNO3p7XGvizdfKZEKLDMSmu2YbWVFlRmQaRL5/rY1B+fs2bPauXOnFi9e3PZcTk6OJk6cqMrKyoj3VFZWav78+WHPTZo0SS+//HLU92lsbFRjY2Pbz4FAILWFAzCFGWXcsYKNjnk3JxrOau66XZ3eK5TQbCTpOJQUncprJCpTh4UCdrI1wPnss8/U3Nysvn37hj3ft29f7d+/P+I9tbW1Ea+vra2N+j4VFRVaunRp6gsGYKpUy7hjBRt3r92l7t266uTpprbnczxKKaE5XUnR7dkRUAFuYHsOTjosXrxYfr+/7fHRRx/ZvSQga3XMXbmu1JdUGbeRQZrtgxtJipUmYyShOd2zrdw0LBRIN1t3cHr37q0uXbro+PHjYc8fP35cPp8v4j0+ny+h6yUpLy9PeXl5qS8YQEpiHbVsWzghoRyTeMFGsmIlNKc7KTqRgIrGf0A4W3dwcnNzNXLkSG3ZsqXtuZaWFm3ZskXl5eUR7ykvLw+7XpI2bdoU9XoAzhCvod+mfbUJlXFbNSAz1lyqdM+2ctOwUCDdbD+imj9/vp5++mk9++yz+vDDDzVnzhw1NDTo9ttvlyTNmDEjLAn53nvv1caNG/Wzn/1M+/fv10MPPaQ///nPmjdvnl0fAUAcVhy1mD0g06PW3aRYCc2hpOhooZeR10iE24aFAulke4Dz/e9/X4899ph+8pOfaPjw4dq9e7c2btzYlkh89OhR1dR81ap97NixWrdunZ566ilddtll+s///E+9/PLLKisrs+sjAIjDityVeMFGIozOpQolRbe/J9HXSES6AyrATWzvg2MH+uAA4czusdLx9Wr9f9f9v/9r3Pt++Y/DdePw/obfJ3TsJUWujoomxxOecOzkPjjRPmPo3w5VVMgmiXx/E+AQ4CDLmf1lHen1ep6Xq/qGs3HvjdfQz+j79ejWVSdON0UtPX/ylsvV47y8lAK6dDbeow8O0IoAJw4CHKBVtB4rye4OxHq9WP+hCTX027ZwQlJBQqRgY9O+WlcFBXQyBghw4iLAAVq/MK9+9I2ouTGJBh3xXi8aK49aCAoAd8mYUQ0A7GN2jxWjfWl6ntdV9Q1fNeDzWbirEpogDiD7EOAAWcrsHitGr3vwO9+QrzA/43ZV2A0CMgsBDpClzO6xYvQ6X2G+o3dVsiGfB8gGBDhAljJjkreVr2eHSNVKHQd2hjDsEnA22xv9AbCH2U3r0t0Ez2zRRklECm4khl0CTkeAA2SxyWXFSU3yTtfrpUusURKxmD09HIB5OKICstzksmJdV+ozLYHW7NdLR3JvqpPJGXYJOA8BDgDTy6nNer10dfBNNUBh2CXgPBxRAXCkaDkxoeTejXtqotyZuGQDFIZdAs5FgAPAcs0tQVVW1Wn97mpVVtXFTcqNlRNjRXJvMpPJMyFxGshmHFEBsFQyx0xmd1mOJ1QBNmftrogDOoPqXC5uZQdmAKkjwAFgmWjDN+P1kDG7e3JHkRKXQxVgHYOxUCBjZuI0AOsR4ACwRLxjJo9aj5muK/V1ChTM7p7cXrwdpViBjJM7MAMIRw4OAEskcszUUbycmGSTe40kLocqwG4c3l/lQ3qxSwNkKAIcAJZI5ZjJiq7I6U5cBmAvAhwAlkj1mMnsrsip7CgByDzk4ACwRLzhm5Lk8Uj/b29N2/Udd2TM7IpsdeIyAGchwAFgifal19EEg9Lqd49o9btHopaOm9UV2crEZQDOwxEVgKgSbdDX8V7vubmaedVAeQxsuFjRobg9qxKXATgTOzgAIkplDlSke+OJVzqeqnjN/CS6EgNuwg4OgE5SmQMV7V4jrE70NTtxGYBzsYMDIIzRBn0ThvXVziMnwpJ/9eXvUi20tjLR18zEZQDORYADIIzRcuoxFZtV3/DVbKZib77+8cqSpHZuOrI60desxGUAzkWAAyCM0d2T9sGN1Hp89fPNf0vpvT1qPS4i0RdAqsjBARAm2d2TVI+lSPQFYCYCHCDDpVLKHUm8cupUeCT1Oi9XM8cOUM/zcsN+R6IvADNxRAVksFRKuaOJVU5thof/T5kmlxXrwe98g0RfAJbxBIPBrJssFwgE5PV65ff7VVhYaPdygKSEyrE7/g84FCKkuhuycU+NHnplr2oDjW3P9ezWVfWnm2LcFV2OR3pi2ghdfyk7NACSk8j3N0dUQJqYdZTU3BLUOwc/06L/+0EaJmOH76jkntNF3bt1Ter4qiUo1fj/btpRGgDEwhEVkAZmHSUZ7RDcvmFeMuXQ0XaHjge+GpyZzPHV8tc+bPvnVI/SACAWdnAAi6XSFdjI68SSTMM8I43+enTrqr6FeQm/dntWz54CkN3YwQEsZLQrcLzZS7FeJ5ZkSr6NNPo7cbpJz98xWjkejz45dUa9z8/Tgt/v1vFAo+E1Wj17CkB2YwcHsJDRrsDxZi/Fe52OUpmMbXTX57PPG1U+pJduHN5fVw3trYe++4229zbK6tlTALIXAQ5gIaPBQrzrEjlqSqVhXnNLUJ+daox/oTrvDkUbZGmElbOnAGQnjqgACxk9Iop3XSJHTb4kk3eNJjCHximMHNBDlVV1YX1sOg6y/OxUY1hicTRWz54CkH0IcAATNbcEw5rXjRzQQ8XefNX6z0TMTTE6eynUXTja60hS925d9eS0ERozpFfCOzfRqqYirVeSvntZsb65YmtYMOQrzNe0URdqYO9u6lOQr+9c2k+S9B/bDqX8+QEgUQQ4gEmilYJ/97JiPfX2oU5l1YkcJcXqLhy685HvXaKrvtZbUudAK1aX4EQSmH3tPk/H62sDZ/Tzzf/T9nOoDDzeupk9BcAKtuXgHD58WLNmzdKgQYN07rnnasiQIVqyZInOnj0b875x48bJ4/GEPe6+++40rRqILFYp+FNvH9Kd1wzqlJuS6OylaDkuHV9n454aXf3oG5r29Hbd+8JuTXt6u65+9I2o5dhGE5gfvOFivfXAeL3y1xpDwVCoDFySoXUDgJls28HZv3+/Wlpa9Otf/1pDhw7Vnj17NHv2bDU0NOixxx6Lee/s2bO1bNmytp+7detm9XKBqIyUgr/y1xq99cB47TxyIqXZSx1zXDq+TrSjplCwESmgMJrg27sgTzuPnDBczdW+DHzbwgkx1w0AZrMtwJk8ebImT57c9vPgwYN14MABrVy5Mm6A061bN/l8PquXCBhitBR855ETSXUV7qhLjifi6yTbcyeRROhEq506dlQ24/MDgBGOKhP3+/3q2TN+suHzzz+v3r17q6ysTIsXL9bp06djXt/Y2KhAIBD2AMxiVil4qpLtuRNKYI62l+KR1Ou8XNX6/264hLwjysABpJtjkowPHjyoxx9/PO7uzS233KIBAwaoX79+ev/997Vw4UIdOHBAL774YtR7KioqtHTpUrOXDEgyrxQ8VckGWrESmPXlz3UNZ3X/7/8qqXUqeKJzMikDB5Bupu/gLFq0qFMScMfH/v37w+6prq7W5MmTdfPNN2v27NkxX//OO+/UpEmTdMkll+jWW2/Vc889p5deeklVVVVR71m8eLH8fn/b46OPPjLlswKSsR2QZLsKJyKVQCuRJn2JBDfp+uwA0JHpOzgLFizQzJkzY14zePDgtn8+duyYxo8fr7Fjx+qpp55K+P1Gjx4tqXUHaMiQIRGvycvLU15eaoMBgWiMlHCnoxQ6Xq+ceD1n2icw1wbOaPmre1Xf0BT1/eLt5FAGDsBOpgc4RUVFKioqMnRtdXW1xo8fr5EjR2r16tXKyUl8Q2n37t2SpOJiSk1hn9AOSMc+OMl2FU6GGYFWKIG5sqouZnAjtQY3D95wsXoX5OnwZw367Y6jqg18laOTzs8OAB3ZloNTXV2tcePGacCAAXrsscf06aeftv0uVCFVXV2ta6+9Vs8995xGjRqlqqoqrVu3Ttdff7169eql999/X/fff7+uueYaXXrppXZ9FEBS/BLudK3BjEArkdLxG4f3lyTNm/A1ysABOIZtAc6mTZt08OBBHTx4UBdccEHY74LB1v/v2dTUpAMHDrRVSeXm5mrz5s36xS9+oYaGBpWUlOimm27Sj3/847SvH3AqMwKtZPJ5opWvA4AdPMFQNJFFAoGAvF6v/H6/CgsL7V4OXCLaqIZMPKZpbgnq6kffiJvPs23hhLiBUyJjIwAglkS+vx1TJg5ksmQ6CDtJpCDEjMRpNwV9ADILOzjs4LiambsH0V4rtNsRrcleIrsddogVhEhKOkCJFvSF/gJOD/oAOA87OICM7R4YDYBivZb33FzDHYSdlqNiZOdp28IJCQeJyY6NAACzEODAlYx8cUvGdifivdbtVw00tCanjStIJAhJNDBLZGyE04I+AO7gqFlUgBnifXFL0qIXP9Cctbs6fQmHgpaNe2oMv9b63ccMrctp4wqSnV1lhFPmcwHIXgQ4cB0jX9wnTzfFDFqWbtjXdnwV77XqGs6q53ldbR/VkCgrgxCnzOcCkL0IcOA6qe4KtN+5MPpa/+fLZncdgxwnjyuwMghxynwuANmLAAeuY9auQCip1oiJpb6Iwyp93nzLq4WaW4KqrKrT+t3VqqyqU7PBaZhWBiGhsRGh1+n4upIzgz4A7kGSMVwn3tBJo0IVQ0YHWHbJ8aRlVEP7yq/Dn53+cgZU4mXcVg8JdcJ8LgDZiz449MFxpVDlk9T5izsoqXu3rvJHycPp2Lcm1mtJ6e3nEqlcvaNE12V1Mz46GQMwSyLf3wQ4BDiuFa+BXSJBixM68kYrV48k0eaCBCEAMgEBThwEONkj1hd3okGLnUFAvG7J0fx29hj6zABwDToZA1+KNeE60anbdk7LjleuHs07Bz9jNwZAViLAQVazM2hJRLKl709sPaj/u+tjknoBZB3KxOFIyZY+u1Uqpe8duzMDQDZgBweO44SE3mRZlaeTSuk7wy0BZCMCHDhKtEqhmnZDMp0a5FgZmMXqWWMEwy0BZBuOqOAYsQZbSq1f0qEZUU4TCsziDe9MRahxXqduyYV5ur6sr6HXYLglgGzBDg4cw0ilkBN3IYxMHH/olb2mHA9Fq/zacaher+85Hvd+hlsCyBYEOHCMWv/fTb0uXYwEZrWBRj3xxkHdO/FrKb9fpMqvREZKAEA24IgKjlHfcNbU68wSr6LL6LHPzzf/j2WVTAy3BIBw7ODAMXqen2fqdWYwkjicyLHP4hc/sKySieGWAPAVAhw4hq/QWKBg9LpURavoqu1Q0RU6HjLSafjE6SZt/986XTW0tyVrTrQ7MwC4FUdUcIxQoBBLcZrySIwkDocqutofDxlRWVVnyhqjCeXo3Di8v8qH9CK4AZCVCHDgGKFAwaPIeSQepS+PJF7icPu+MlLrzsm3DZZqJ97FBgCQKAIcOErUXi/e/LQ2+TOaONz+uumjBxq6p3ywNcdTAICvkIMDx3FCHonRxOH2140Z0kvdu3XVydNNUa8/P+8cfXLqjCqr6siNAQALEeDAkeye8p1MX5kuOR498r1LdPfaXVFf9/PGL3T/7/8qKXPmawFAJuKICogg2b4yk8uKtWr6CPkK45eyM+UbAKzjCQaDWZfxGAgE5PV65ff7VVhYaPdy4GDJDtAMTRWvDZzR8lf3qr4h8rFVaCdo28IJHFcBQByJfH9zRAVLhb7oM7UnS7L5QKEjtsqquqjBjcSUbwCwCgEOLJPs7ofTpJIPlEw1FgAgdeTgwBKhLsAde8lkW96J0Wqs3uflxZ15BQAwjh0cmC5eF2CPWrsAWzWTyUniVWOFzP1ta+VV+xLzTNztAgCnYAcHpku0C7CbxarGau/k6aZO/XOybbcLAMxEgAPTOT3vJN1HQaHuzH0NlI6313HmFQDAOI6oYLpkugCni12Jz5PLilWQ31W3/sefErqPKisASA47ODBdKO8k2pGMR+mbCt6e3YnPn33emPS9VFkBQGIIcGC6ZLsAWyle4rNk/VFQKjtWdux2AUAmI8CBJVKZCm5FjowTEp/j7WxFYtduFwBkOltzcAYOHKgjR46EPVdRUaFFixZFvefMmTNasGCBXnjhBTU2NmrSpEn693//d/Xt29fq5WYNs7oPJ9MF2KocGSckPod2tuas3SWPFLNsXLJvtwsA3MD2JONly5Zp9uzZbT8XFBTEvP7+++/Xa6+9pj/84Q/yer2aN2+evve97+mdd96xeqlZwewAI5EuwKEcmY5f/KEcmXg7P7E4JfE5tLPV8W/co1tXBRXeB8dHHxwASJrtAU5BQYF8Pp+ha/1+v5555hmtW7dOEyZMkCStXr1aF198sbZv364xY8ZYuVTXszLAiMfq5oDxGu6Fhl6m4ygo2s6WpIye2wUATmJ7Ds4jjzyiXr166fLLL9eKFSv0xRdfRL12586dampq0sSJE9ueGzZsmC688EJVVlZGva+xsVGBQCDsgXB2J+FanSPjtMTn0M7WjcP7q3xIL3XJ8UR8DgCQHFsDnH/+53/WCy+8oK1bt+quu+7ST3/6U/3rv/5r1Otra2uVm5ur7t27hz3ft29f1dbWRr2voqJCXq+37VFSUmLWR3ANu5Nw05Ejk0riMwAgs5h+RLVo0SI9+uijMa/58MMPNWzYMM2fP7/tuUsvvVS5ubm66667VFFRoby8xLq+xrJ48eKw9woEAgQ5HdidhJuuHJlkEp8BAJnH9ABnwYIFmjlzZsxrBg8eHPH50aNH64svvtDhw4d10UUXdfq9z+fT2bNndfLkybBdnOPHj8fM48nLyzM1YHIju5Nw05kjk0jiMwAgM5ke4BQVFamoqCipe3fv3q2cnBz16dMn4u9Hjhyprl27asuWLbrpppskSQcOHNDRo0dVXl6e9JphfxJurBLqdOTImFUaDwBwBtuqqCorK/WnP/1J48ePV0FBgSorK3X//fdr+vTp6tGjhySpurpa1157rZ577jmNGjVKXq9Xs2bN0vz589WzZ08VFhbqnnvuUXl5uesrqKz+ArY7wJCil1BbXS5t13wqAIB1PMFg0JYxxbt27dI//dM/af/+/WpsbNSgQYP0gx/8QPPnz287Tjp8+LAGDRqkrVu3aty4cZK+avT329/+NqzRn9FSc6k1B8fr9crv96uwsNCKj2eqdH4Bp+O94gVr6dhNCb3H5n21euadw51+H3o3ko8BwDkS+f62LcCxUyYFONF601j5BWxlgOGE3ZJIa4gkdCy3beEEjqsAwAES+f62vQ8OorOrN41V/VjsnuYdaw2RpGM+FQDAGgQ4DmZ3bxoz2d1IMN4aYrFyPhUAwBoEOA5md28aMzkhWIu3hmisnk8FADCf7bOoEJ3dvWnM5IRgLdHXTud8KgCAudjBcbBQb5poGTAetSboZsIXsBOCtURe2475VAAA8xDgOJjTBkSmwgnBWrw1tMd8KgDIbAQ4DueWAZFOCNaMrOGHVw3Ub2eP0baFEzLmbwsA6Iw+OA7vgxPillECTu2DQ+diAHA+Gv3FkYkBjps4IVhzwhoAAIlJ5PubKiqknROmeTthDQAA65CDAwAAXIcABwAAuA5HVHAUcmMAAGYgwIFjUN0EADALR1RwBCdMGgcAuAcBDmznhEnjAAB3IcCB7ZwwaRwA4C4EOLCdEyaNAwDchQAHtnPCpHEAgLtQRYW061gKPnJADxV781XrPxMxD8ej1uGiVk4aBwC4CwEO0ipaKfh3LyvWU28fkkcKC3LSNWkcAOAuHFEhbWKVgj/19iHdec0g+bzhx1A+b75WTh9BHxwAQELYwUFaxCsF90h65a81euuB8dp55ASdjAEAKSHAQVoYLQXfeeQEU74BACkjwMlS6Z75RCk4ACCdCHCykB0znygFBwCkE0nGWcaumU+jBvVUsTdf0faIPGoNsigFBwCYgQAni8RL9A3KuplPXXI8WjKlVJI6BTmUggMAzEaAk0XiJfpKrYm+T7zxN0vef3JZsVZOH0EpOADAcuTgZBGjCbw/3/w3XeQrsCTgmFxWrOtKfWlNcAYAZB8CnCySSALv0g37dF2pz5LAo0uOh1JwAIClOKLKIqFEXyNq/Ge041C9xSsCAMAaBDhZpH2irxH0pAEAZCoCnCwzuaxY90/8uqFr6UkDAMhUBDhZaN6EofIV5kX9PT1pAACZjgAnC3XJ8eih735DHtGTBgDgTgQ4WYqeNAAAN6NM3ETpHmCZKnrSAADcigDHJHYMsDQDPWkAAG7EEZUJ7BpgCQAAIrMtwHnzzTfl8XgiPt57772o940bN67T9XfffXcaVx4u3gBLyboBlgAAIDLbjqjGjh2rmprwnY0HH3xQW7Zs0RVXXBHz3tmzZ2vZsmVtP3fr1s2SNRoRb4BlUF91BeYoCACA9LAtwMnNzZXP52v7uampSevXr9c999wjjyd2kmu3bt3C7rWT0W6/dAUGACB9HJOD88orr6iurk6333573Guff/559e7dW2VlZVq8eLFOnz6dhhVGZrTbb6pdgZtbgqqsqtP63dWqrKrjyAsAgBgcU0X1zDPPaNKkSbrgggtiXnfLLbdowIAB6tevn95//30tXLhQBw4c0Isvvhj1nsbGRjU2Nrb9HAgETFt3aIBlrf9MxDwcj1p7y6TSFThTK7QAALCL6Ts4ixYtipo8HHrs378/7J6PP/5Y//Vf/6VZs2bFff0777xTkyZN0iWXXKJbb71Vzz33nF566SVVVVVFvaeiokJer7ftUVJSkvLnDGk/wNKKrsBUaAEAkDhPMBg09azj008/VV1dXcxrBg8erNzc3Lafly9frscff1zV1dXq2rVrQu/X0NCg888/Xxs3btSkSZMiXhNpB6ekpER+v1+FhYUJvV80VuyyNLcEdfWjb0RNYg7tDm1bOCFqAJVpzQcBAIgmEAjI6/Ua+v42/YiqqKhIRUVFhq8PBoNavXq1ZsyYkXBwI0m7d++WJBUXRw8i8vLylJcXfbikGazoCpxqhRZHWwCAbGV7kvEbb7yhQ4cO6Y477uj0u+rqag0bNkw7duyQJFVVVWn58uXauXOnDh8+rFdeeUUzZszQNddco0svvTTdS+8k1BX4xuH9VT6kV8o7JalUaHG0BQDIZrYHOM8884zGjh2rYcOGdfpdU1OTDhw40FYllZubq82bN+tb3/qWhg0bpgULFuimm27Shg0b0r3stEi2QovmgwCAbGd7FdW6deui/m7gwIFqnyJUUlKit956Kx3LcoRkK7RoPggAyHa27+AgumQrtGg+CADIdgQ4Dje5rFgrp4+Qzxt+DOXz5mvl9BERk4XT1XwQAACnsv2ICvElWqGVjuaDAAA4GQFOhghVaBm9dsmUUs1Zu0seKSzIMaP5IAAATscRlUslc7QFAIBbsIPjYlY0HwQAIBMQ4LhcIkdbAAC4BUdUAADAdQhwAACA6xDgAAAA1yHAAQAArkOAAwAAXIcABwAAuA4BDgAAcB0CHAAA4DoEOAAAwHUIcAAAgOswqsFFmluCzJ0CAEAEOK6xcU+Nlm7Ypxr/mbbnir35WjKllMnhAICswxGVC2zcU6M5a3eFBTeSVOs/ozlrd2njnhqbVgYAgD0IcDJcc0tQSzfsUzDC70LPLd2wT80tka4AAMCdCHAy3I5D9Z12btoLSqrxn9GOQ/XpWxQAADYjwMlwn5yKHtwkcx0AAG5AgJPh+hTkm3odAABuQICT4UYN6qlib76iFYN71FpNNWpQz3QuCwAAWxHgZLguOR4tmVIqSZ2CnNDPS6aU0g8HAJBVCHBcYHJZsVZOHyGfN/wYqsd5XfXkLZfTBwcAkHUIcFxiclmxHryhVD3Py217rr6hSctf+5A+OACArEOA4xIb99Ro7rpdqm84G/Y8zf4AANmIAMcFaPYHAEA4AhwXoNkfAADhCHBcgGZ/AACEI8BxAZr9AQAQjgDHBWj2BwBAOAIcF6DZHwAA4QhwXCJasz+fN18rp4+g2R8AIKucY/cCYJ7JZcW6rtSnHYfq9cmpM+pT0Hosxc4NACDbEOC4TJccj8qH9LJ7GQAA2IojKgAA4DoEOAAAwHUIcAAAgOtYFuA8/PDDGjt2rLp166bu3btHvObo0aO64YYb1K1bN/Xp00cPPPCAvvjii5ivW19fr1tvvVWFhYXq3r27Zs2apc8//9yCTwAAADKVZQHO2bNndfPNN2vOnDkRf9/c3KwbbrhBZ8+e1bvvvqtnn31Wa9as0U9+8pOYr3vrrbdq79692rRpk1599VW9/fbbuvPOO634CAAAIEN5gsGgpSOm16xZo/vuu08nT54Me/6Pf/yjvvOd7+jYsWPq27evJGnVqlVauHChPv30U+Xm5nZ6rQ8//FClpaV67733dMUVV0iSNm7cqOuvv14ff/yx+vXrZ2hNgUBAXq9Xfr9fhYWFqX1AAACQFol8f9uWg1NZWalLLrmkLbiRpEmTJikQCGjv3r1R7+nevXtbcCNJEydOVE5Ojv70pz9Ffa/GxkYFAoGwBwAAcC/bApza2tqw4EZS28+1tbVR7+nTp0/Yc+ecc4569uwZ9R5JqqiokNfrbXuUlJSkuHoAAOBkCQU4ixYtksfjifnYv3+/VWtN2uLFi+X3+9seH330kd1LAgAAFkqok/GCBQs0c+bMmNcMHjzY0Gv5fD7t2LEj7Lnjx4+3/S7aPZ988knYc1988YXq6+uj3iNJeXl5ysvLa/s5lHbEURUAAJkj9L1tJH04oQCnqKhIRUVFya2qg/Lycj388MP65JNP2o6dNm3apMLCQpWWlka95+TJk9q5c6dGjhwpSXrjjTfU0tKi0aNHG37vU6dOSRJHVQAAZKBTp07J6/XGvMayKqqjR4+qvr5er7zyilasWKH//u//liQNHTpU559/vpqbmzV8+HD169dP//Zv/6ba2lr94Ac/0B133KGf/vSnkqQdO3ZoxowZ2rJli/r37y9J+va3v63jx49r1apVampq0u23364rrrhC69atM7y2lpYWHTt2TAUFBfJ43DmIMhAIqKSkRB999BGVYhbg72st/r7W4u9rLf6+1gkGgzp16pT69eunnJzYWTaWDdv8yU9+omeffbbt58svv1yStHXrVo0bN05dunTRq6++qjlz5qi8vFznnXeebrvtNi1btqztntOnT+vAgQNqampqe+7555/XvHnzdO211yonJ0c33XSTfvWrXyW0tpycHF1wwQUpfsLMUFhYyP/ALMTf11r8fa3F39da/H2tEW/nJsTyPjiwB71+rMXf11r8fa3F39da/H2dgVlUAADAdQhwXCovL09LliwJqx6Defj7Wou/r7X4+1qLv68zcEQFAABchx0cAADgOgQ4AADAdQhwAACA6xDgAAAA1yHAyQKHDx/WrFmzNGjQIJ177rkaMmSIlixZorNnz9q9NNd4+OGHNXbsWHXr1k3du3e3ezkZ78knn9TAgQOVn5+v0aNHd5pbh+S8/fbbmjJlivr16yePx6OXX37Z7iW5SkVFha688koVFBSoT58+mjp1qg4cOGD3srIWAU4W2L9/v1paWvTrX/9ae/fu1c9//nOtWrVKP/rRj+xemmucPXtWN998s+bMmWP3UjLe7373O82fP19LlizRrl27dNlll2nSpEmdBu0icQ0NDbrsssv05JNP2r0UV3rrrbc0d+5cbd++XZs2bVJTU5O+9a1vqaGhwe6lZSXKxLPUihUrtHLlSv3v//6v3UtxlTVr1ui+++7TyZMn7V5Kxho9erSuvPJKPfHEE5JaZ8eVlJTonnvu0aJFi2xenXt4PB699NJLmjp1qt1Lca1PP/1Uffr00VtvvaVrrrnG7uVkHXZwspTf71fPnj3tXgYQ5uzZs9q5c6cmTpzY9lxOTo4mTpyoyspKG1cGJM7v90sS/621CQFOFjp48KAef/xx3XXXXXYvBQjz2Wefqbm5WX379g17vm/fvqqtrbVpVUDiWlpadN999+mqq65SWVmZ3cvJSgQ4GWzRokXyeDwxH/v37w+7p7q6WpMnT9bNN9+s2bNn27TyzJDM3xcAJGnu3Lnas2ePXnjhBbuXkrXOsXsBSN6CBQs0c+bMmNcMHjy47Z+PHTum8ePHa+zYsXrqqacsXl3mS/Tvi9T17t1bXbp00fHjx8OeP378uHw+n02rAhIzb948vfrqq3r77bd1wQUX2L2crEWAk8GKiopUVFRk6Nrq6mqNHz9eI0eO1OrVq5WTw+ZdPIn8fWGO3NxcjRw5Ulu2bGlLfm1padGWLVs0b948excHxBEMBnXPPffopZde0ptvvqlBgwbZvaSsRoCTBaqrqzVu3DgNGDBAjz32mD799NO23/H/is1x9OhR1dfX6+jRo2pubtbu3bslSUOHDtX5559v7+IyzPz583Xbbbfpiiuu0KhRo/SLX/xCDQ0Nuv322+1eWsb7/PPPdfDgwbafDx06pN27d6tnz5668MILbVyZO8ydO1fr1q3T+vXrVVBQ0JY35vV6de6559q8uuxDmXgWWLNmTdQvB/71m2PmzJl69tlnOz2/detWjRs3Lv0LynBPPPGEVqxYodraWg0fPly/+tWvNHr0aLuXlfHefPNNjR8/vtPzt912m9asWZP+BbmMx+OJ+Pzq1avjHnfDfAQ4AADAdUjEAAAArkOAAwAAXIcABwAAuA4BDgAAcB0CHAAA4DoEOAAAwHUIcAAAgOsQ4AAAANchwAEAAK5DgAMAAFyHAAcAALgOAQ4AAHCd/x/HESCpZdp6ZgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Linear Regression Example\n",
    "\n",
    "import jax\n",
    "from jax import numpy as jnp\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "xs = np.random.normal(size=(100,))\n",
    "noise = np.random.normal(scale=1, size=(100,))\n",
    "ys = xs * 3 - 1 + noise\n",
    "\n",
    "plt.scatter(xs, ys)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model(theta, x):\n",
    "    \"\"\"Computes wx + b on a batch of input x.\"\"\"\n",
    "    w, b = theta\n",
    "    return w * x + b\n",
    "def loss_fn(theta, x, y):\n",
    "    prediction = model(theta, x)\n",
    "    return jnp.mean((prediction-y)**2)\n",
    "@jax.jit\n",
    "def update(theta, x, y, lr=0.1):\n",
    "    return theta - lr * jax.grad(loss_fn)(theta, x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 10.60\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "loss: 0.89\n",
      "w: 3.04, b: -0.88\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABJNUlEQVR4nO3dfXhT5f0/8Pdppa3FNlAppEClBZxYiyIoUGAICLYTGGyOKcoApzg7cCp8XcGprPJ1lek2N3U4nIL7VUTdVKS67suDyJAiSqmjVJjFlofS8NDaBIp9IDm/P0pC05yTnCTn5CQn79d19dp655yTO53X8vZ++NyCKIoiiIiIiAwkRu8OEBEREamNAYeIiIgMhwGHiIiIDIcBh4iIiAyHAYeIiIgMhwGHiIiIDIcBh4iIiAyHAYeIiIgM5xK9O6AHh8OB48ePIykpCYIg6N0dIiIiUkAURZw5cwZ9+/ZFTIz3MZqoDDjHjx9Henq63t0gIiKiABw9ehT9+/f3ek1UBpykpCQAHX+g5ORknXtDRERESthsNqSnp7u+x72JyoDjnJZKTk5mwCEiIoowSpaXcJExERERGQ4DDhERERkOAw4REREZDgMOERERGQ4DDhERERkOAw4REREZDgMOERERGQ4DDhERERmO7gEnIyMDgiB4/CxcuFDy+rVr13pcm5CQEOJeExERUTjTvZLxZ599Brvd7vq9srISU6ZMwaxZs2TvSU5OxsGDB12/88BMIiIi6kz3gJOamur2+9NPP41Bgwbhpptukr1HEASYzWatu0ZEREQRSvcpqs7a2tpQXFyMn/70p15HZc6ePYsBAwYgPT0dM2bMwP79+70+t7W1FTabze2HiIiIjCusAs57772HpqYmzJ8/X/aaq666Cq+++io2bNiA4uJiOBwOjBkzBseOHZO9p6ioCCaTyfWTnp6uQe+JiIgI27cDy5cDdXW6dkMQRVHUtQed5ObmIi4uDhs3blR8T3t7O66++mrMnj0bK1askLymtbUVra2trt+dx61brVaeJk5ERKQGUQQmTQK2bev4vbAQeOIJVd/CZrPBZDIp+v7WfQ2O0+HDh7F582a88847ft3XrVs3XH/99aiurpa9Jj4+HvHx8cF2kYiIiKRUVQHXXOPe5mU2JhTCZopqzZo16N27N6ZOnerXfXa7Hfv27UNaWppGPSMiIiJZ99/vHm6Sk4HWVuCKK/TrE8JkBMfhcGDNmjWYN28eLrnEvUtz585Fv379UFRUBAB48sknMXr0aAwePBhNTU145plncPjwYdx77716dJ2IiCg61dcDffu6t736Kuzz5mN3TSNOnmlB76QEjMxMQWxM6Mu5hEXA2bx5M44cOYKf/vSnHq8dOXIEMTEXB5q++eYbLFiwABaLBT179sSIESOwc+dOZGVlhbLLRERE0euZZ4Bf/tK97ZtvUHrsWxSu3Ip6a4urOc2UgOXTs5CXHdqZlrBaZBwq/ixSIiIiogvOngWSktzbnngCKCxEaWU98ovL0TVUOMduVs0ZHnTI8ef7O2zW4BAREVEYe/NNz3Bz+DBQWAi7Q0ThxiqPcAPA1Va4sQp2R+jGVBhwiIiISN7580D//sAdd1xsu/POjm3hFxYS765pdJuW6koEUG9twe6aRo07e1FYrMEhIiKiMPTvfwPjx7u37d0LDBvm1nTyjHy4CeQ6NXAEh4iIiNyJInDzze7h5oYbALvdI9wAQO+kBEWPVXqdGhhwiIiI6KIDB4CYGGDr1ottpaXAZ591tEsYmZmCNFMC5DaDC+jYTTUyM0X17sphwCEiIqIOP/85cPXVF3/v3h1oaQFyc73eFhsjYPn0jnItXUOO8/fl07NCWg+HAYeIiCjanTgBCAKwatXFtr/+tWNbuMKjjvKy07BqznCYTe7TUGZTgipbxP3FRcZERETR7NlngUcecW9rbAR69vT7UXnZaZiSZWYlYyIiItJJczNw2WXubY89BqxYEdRjY2ME5Ay6PKhnqIEBh4iIKNq89RZw++3ubbW1wIABunRHCww4REREOrM7xNBM65w/D2RmAseOXWy7/XZg/Xr130tnDDhEREQ6Kq2sR+HGKu0PqNyxA/jud93b9uwBhg9X7z3CCHdRERER6cR5QGXXYw4s1hbkF5ejtLI++DcRRWDKFPdwM2xYR9E+g4YbgAGHiIhIFyE5oPLgwY7ifJs3X2z78MOO4xZkivYZhbE/HRERUZjS/IDKRYuAIUMu/n7ppR1F+773vcCeF2EYcIiIiHSg2QGVzqJ9L754sW31auDcOcVF+4yAAYeIiEgHmhxQ+fvfA2aze1tjI7BggR89MwYGHCIiIh2oekBlc3PHqM2SJRfbHn20Y4FxABWJjYABh4iISAeqHVD59797ViSuqQGeekqVfkYqBhwiIiKdBHVApd3eUbRv1qyLbT/+cceoTUaGNh2OICz0R0REpKOADqj85BNg3Dj3ts8/B0aM0LazEYQBh4iISGeKD6gUxY5t3v/618W2664DyssNX9fGXww4REREkeC//wWuusq97YMPgFtv1ac/YY5xj4iIKNz94hfu4SY+Hvj2W4YbLziCQ0REFK5OngT69HFv+8tfgPvu06c/EYQjOEREROHouec8w01DA8ONQhzBISIiCifnzgHdu7u3LV0KFBXp058IxYBDREQULv7xD+BHP3Jv+/rrjno35BcGHCIiIr21tXkehHnbbR1ViikgXINDRESkp9/9zjPcfPYZw02QOIJDRESkB1GULs5nt7Nonwr4FyQiIgq1Dz/0DDH5+fKhh/zGERwiIqJQEiTOmPrmG6BHj5B3xcgYE4mIiEKhqsoz3Fx/fceoDcON6jiCQ0REpLXBg4FDh9zbvvqqo500wYBDRESklYYGoFcvz3ZRDH1fogynqIiIiLQwd65nuNmyheEmRDiCQ0REhmB3iNhd04iTZ1rQOykBIzNTEBsjsaBXa+3tQFycZzuDTUgx4BARUcQrraxH4cYq1FtbXG1ppgQsn56FvOy00HXkueeAhx92b/vznzu2gFNI6T5F9etf/xqCILj9DBkyxOs9b7/9NoYMGYKEhAQMHToUH374YYh6S0RE4aa0sh75xeVu4QYALNYW5BeXo7SyXvtOiGLHDqmu4aa9neFGJ7oHHAC45pprUF9f7/rZsWOH7LU7d+7E7Nmzcc8992Dv3r2YOXMmZs6cicrKyhD2mIiI9GJ3iCg71IANFXX45KvT+PX7VZCa/HG2FW6sgt2h7vRQ5z58+epbnsX57ruvI/RcwokSvYTFX/6SSy6B2WxWdO0f//hH5OXl4ZFHHgEArFixAps2bcILL7yAl156SctuEhGRzqSmorwRAdRbW7C7phE5gy5XvQ+1K6d5XsCifWEhLEZwvvrqK/Tt2xcDBw7EXXfdhSNHjsheW1ZWhsmTJ7u15ebmoqysTPae1tZW2Gw2tx8iIoosclNRSpw84/893vrQv/Jzj3BzIDUDpfuOM9yECd1HcEaNGoW1a9fiqquuQn19PQoLC/Hd734XlZWVSEpK8rjeYrGgT58+bm19+vSBxWKRfY+ioiIUFhaq3nciIgoNu0NE4UbpqSgleiclqNaHGolRm4kL/oLalH4wb6zClCyzPru3yI3uIzjf+973MGvWLFx77bXIzc3Fhx9+iKamJrz11luqvceyZctgtVpdP0ePHlXt2UREpL3dNY0BjdwI6NhNNTIzJeg+VJTtR9mjkz3aMwpKUJPSz206jPSn+whOVz169MB3vvMdVFdXS75uNptx4sQJt7YTJ054XcMTHx+P+Ph4VftJREShE8gUk3MMZfn0rOBHVC6/HCMa3YPLo7kLsW7Y9zwuVWs6jIKj+whOV2fPnsWhQ4eQliZdtyAnJwdbtmxxa9u0aRNycnJC0T0iItJBIFNMZlMCVs0ZHlwdnNbWju3fXcJNRkGJZLgB1JkOo+DpPoLzP//zP5g+fToGDBiA48ePY/ny5YiNjcXs2bMBAHPnzkW/fv1QVFQEAHjwwQdx00034Xe/+x2mTp2K9evX4/PPP8fq1av1/BhERORDMJWGR2amIM2UAIu1RXIdjgCgT3I8fvfjYTh9tlWdSsZz5gCvv+7W9OH1U7DwlgclLxfQEarUmA6j4OkecI4dO4bZs2ejoaEBqampGDduHHbt2oXU1FQAwJEjRxDTqb7AmDFjsG7dOjz22GN49NFHceWVV+K9995Ddna2Xh+BiIh8CLbScGyMgOXTs5BfXA4BcAs5zgjz6+9fg7GDJQ62DIQgEYxaWhDzVSPgpQ+qTIeRKgRRjL7DMWw2G0wmE6xWK5KTk/XuDhGRoTm3Vnf9snHGAH+mkTQ/kmHVKuDnP3dvS00FTp4MXR9Ilj/f3ww4DDhERJqxO0SMW7lVdgeUc1pnR8EkxSMfmh2qKTVqU1cH9O0buj6QV/58f+s+RUVERMbla3t3IJWGY2ME1aoSAwA++QQYN06ic/L//q96H0h1DDhERKQZpVumddtaLTVqs2MHMHZs6PtCqmLAISIizSjdMh3yrdXHjwP9+nm2R9+qDcMKuzo4RERkHM7t3XKrU9SsNKyYIHiGmz//meHGYBhwiIhIM87t3QA8Qk7It1a3tEhPSYkikJ+v/ftTSDHgEBGRpvKy07BqznCYTe7TUKpUGlYqKwu49FL3th/9iKM2BsY1OEREpLm87DRMyTLrs7VaatSmuRlITNT+vUk3DDhERBQSId9avWwZ8PTTnu0ctYkKDDhERBQR/CquJzVq89//AldeqW0nKWww4BARUdhTfDzCBx8A06Z5PoCjNlGHi4yJiCisOc+y6loR2WJtQX5xOUor6zsaBMEz3PzjHww3UYoBh4iIwpbdIaJwY5XHQZ3AxdO8X1+9UX779w9/qGX3KIxxioqIiMKWr7OsalZKTEfdemvHVBVFNQYcIiIKW3JnVCW2fYuqP8zyfMHhkB7NoajDgENERGFL6oyqWqlRG4BrbcgN1+AQEVHY6nqWlVS4yX3sHdjtDkXPsztElB1qwIaKOpQdaoDdwVBkVBzBISKisOU8yypvaF/J1zMLSvDQTd9ByX+O+6yNo3irORmCIIrRN6Zns9lgMplgtVqRnJysd3eIiDTnV5G8cOuHxJqan/3gUewedhNEAE3n2l3tcoHFudW86xee88khOxOLguLP9zdHcIiIDE6PkQupILOpyuJfP+bPB157zaN5w95jyDrdjH9t/srjNWdtnM6BxddWcwFA4cYqTMky6xL6SBscweEIDhEZmB4jF1KBqkdiN7eRFp/9kNoJNWkSsGUL7A4R41Zu9bp9PM2UgB0FkxAbI6DsUANmv7zLZ7/fWDA6tGdlkd/8+f7mImMiIoNSUiSvcGOVqgtt5aoOS4UbyX5s3SpftG/LFgDArkMNXsMNANRbW7C7phGA/FbzrpReR5GBAYeIyKB8FckT4R4EguUtUHnj7EdsbAxw880SF1x8YmllPRauK1f03E1VFgDSW82lKL2OIgMDDhGRQYV65MJXoJKT1NosXdvG4fAIN/nF5Wj6Vno0qKsNFcdhd4geW827EtAxpTUyM8XvvlP4YsAhIjKoUI9cBBKUaldOw77nbvd8QRTdpqoCGR1qaG7D7ppG11ZzAB4hx/n78ulZXGBsMAw4REQGFeqRC3+DktSozb9Kd6Os+rTHuqBAR4ecoSsvOw2r5gyH2eTeR7MpgVvEDYrbxImIDMo5cpFfXA4BcBv90GLkwhmoLNYWryMtckctZBSUAB+dBD466bF9PNBptM6hKy87DVOyzGFRD4i0xxEcIiIDC+XIhZKpIKlwU5D3QEe46aTe2oL7i8vxx81fwe4Q/R4dkhudio0RkDPocswY1g85gy5nuDEw1sFhHRwiigKhrGQsVQfnn8WLcXXdfz2uHf3UZlhs3kdnzMnxeGJaFlZ88KXP0SGA1YmNzJ/vbwYcBhwiItV1DlQzru8veU1Z9WlFBfiAjtBy3/hMrN5eAwBeQw7PlzIuHtVARES6io0RkPPFx8Btt3m+eOHfq09W1Pn1zPe/qMeLdw7Hig88j3u448YrkNErketqyIUBh4iI1CdVjRhwq2vjz7oaZzHAnt3jsKNgEhcKk08MOEREpJ5vvgFSJLadOxweoUfprqvOTp5pcS0UJvKGu6iIiEgdgiAdbroU7XPqvOtKKR6nQEox4BARUfCkpqS+/NJtSkqKaxt7svfgwuMUyF8MOEREFDhBkD/9e8gQRY/Iy07DJ0sn4eHJV0q/xYX/5HEK5A8GHCKiKGF3iCg71IANFXUoO9TgcRyC36SCzS9/6XPURkpsjIAHJ38HL80ZjjQep0Aq4CJjIqIoIFV8L+B6MdnZwP79nu0qlFXjcQqkFhb6Y6E/IjK40sp65BeXe+xUCqjir4Lt30Ra8ef7W/cpqqKiItx4441ISkpC7969MXPmTBw8eNDrPWvXroUgCG4/CQlcWU9E1JXdIaJwY5XkNmxnW+HGKt/TVX//u/xaG4YbCkO6B5yPP/4YCxcuxK5du7Bp0ya0t7fjlltuQXNzs9f7kpOTUV9f7/o5fPhwiHpMRBQ5dtc0uk1LdeUsoLe7plH+IYIAzJolcTODDYUv3dfglJaWuv2+du1a9O7dG3v27MH48eNl7xMEAWazWevuERFFtJNnvB9k6fU6mw0wmTzb7XYgRvd/PybyKuz+CbVarQCAFKliUZ2cPXsWAwYMQHp6OmbMmIH9UgveLmhtbYXNZnP7ISKKBkoL43lcJwjS4UYUGW4oIoTVP6UOhwMPPfQQxo4di+zsbNnrrrrqKrz66qvYsGEDiouL4XA4MGbMGBw7dkzy+qKiIphMJtdPenq6Vh+BiChoam7ndh6HILcHSbKAntRam337OCVFESWsdlHl5+fjn//8J3bs2IH+/fsrvq+9vR1XX301Zs+ejRUrVni83traitbWVtfvNpsN6enp3EVFRLqxO0TJrdCqbue+wLmLCoDbYmOPXVTcIUVhzp9dVLqvwXFatGgRSkpKsH37dr/CDQB069YN119/PaqrqyVfj4+PR3x8vBrdJCIKmlyI+f51aVi9vcZjx5PF2oL84vKAi905j0Po+p7mzsFJKtw8+CDw3HN+vx9RONA94IiiiAceeADvvvsutm3bhszMTL+fYbfbsW/fPtx6660a9JCIooXcqIqa5GrSWKwt+Mv2Gsl7RHSMthRurMKULHNAfZItoDdqJPD55xJvylEbimy6B5yFCxdi3bp12LBhA5KSkmCxWAAAJpMJl156KQBg7ty56NevH4qKigAATz75JEaPHo3BgwejqakJzzzzDA4fPox7771Xt89BRJFNydRQsAFISU0aOZ23c+cMulzxe3YWGyO438spKTIw3QPOqlWrAAATJkxwa1+zZg3mz58PADhy5AhiOq3a/+abb7BgwQJYLBb07NkTI0aMwM6dO5GVlRWqbhORgXgbVXFODQEIem2Mr5o0Sijd9u3V++8DM2Z4tjPYkIGE1SLjUOFRDUTkZHeIGLdyq2zwEACYErvBeq496KMONlTU4cH1FcF0F28sGB3wCA4AjtpQRIuooxqIiPSkpNJvk0S4cb4GKDzqAMpr0kiR3M7tj7NnJcPN2P/9F0r3HQ+4X0ThigGHiKJasFM+io46uMBXTRpflk/PCmzRsyAASUkezRkFJTh+ph35xeUorawPsFdE4YkBh4iiWjCjKp0pCUqxMQKWT+9YK9g1pggXfn42PhOmSz2XR5oSuwXWMYlRm2nznkNGQQkA/0ehiCIFAw4RRbVgR1WclAYlZ00as8n9erMpwbWY2frteY/7ms614/7icjy5cb9HdWPJyseCIBluMgpKUGke7NbmzygUUaTQfRcVEZGenKMq+cXlEOBZ6VcE0ENmkbHzGrOfa2PkatL8q9IiWwvH6dVPavHqJ7WuHVyA5+6u2pXTPO57fVgefpW7yOuzVdmhRRQmGHCIKOr5qvQLQDYAAYGtjelak8buEPHYhkrF91usLbj/wvELTm+9/kuMPFblcW1Z9Wn86uVdPp+p1nQdUThgwCEigpdKvxeCi8+jDoK0u6YRjc1tiq/vOpokNWoDAHa7AyPRsQPLYm1RbRSKKNwx4BARXeBR6bcTXwEoWIFOD035ahdefud/Pdqdi4jfuFD52Ns0HBDEDi2iMMWAQ0SkkLcAFKxApofkRm2c4Qa4GJwUHbhJZCAMOEREYcC5m0vJUQ6Jbd+i6g+zPNoHPbIB9phYt7bOwUnrUSiicMKAQ0QUBjrv5vJWjUbJqA0gv65Gy1EoonDCOjhERJ1I1pQJEec0UppJerpKKtz8YM6zyJQINwDX1VB042GbPGyTiC4orawP+sRwNdgdots0Us7gXpLXOc+QCoc+E4WCP9/fDDgMOESEjnAjNT3k74nhqpOoRtz03UlI2rbZNTrTNRBxXQ0ZlT/f31yDQ0RRz+4QUbixSvbEcAEdoyRTssyhCw5paYDFItEhET26NHFdDZEnrsEhoqi3u6bR6+6lUJ3V5Fz/A0GQDTdGoedaJ4oOHMEhoqintMielmc1lVbW49Mn/4jlb6/0fNFAwQYIn7VOZGwcwSGiqKe0yJ5WZzWVVtYjb2hfyXCTWVCC0sp6Td5XD861Tl1HzCzWFuQXlxvqs5K+GHCIKOo5i+zJra4R0DHCoMVZTfYzZ5E3tK9H+5X/866rtk3hxipDTOH4WusEGOezkv4YcIgo6jmL7AGQDTn+1JRRvL5EEBCbnOTRnFFQgvbYbgBCt/4nFMJlrRNFB67BISLCxSJ7S9/Zh6Zz7W6vmRK7KX6O4vUlEtu/7/3h49h85SjJ535SfTrit3+Hw1onih4cwSEi6qRruAEA67l2RetDFK0vEQTJcJNRUCIbbgDghY+qMW7l1oheo6L3WieKLgw4RES4uD5EipL1IUrWl0ittREvuww5v9ksOzXWWaQvxNVzrRNFHwYcIiIEvz7E2/21K6ehRuqQTFGEcOaMz/U/nfsARO5CXG9rnXh+FqmNAYeICMGvD5Frlzv9u3NtG+f6H7PMIZtutyGyF+LKfVazKUG/4zDIkLjImIgIwa8P6dr+g8qt+MMHv/e4rqz6tOSxCnnZaZiSZcYfNh3ECx8d8tmPT6pPRezZU87PyvOzSEsMOEREuLg+xGJtkVxHI6BjlEFufUjn+yWnowDk/GYzdnhZXxIbI2Ds4FRFAafzNZFYBZjnZ5HWOEVFRITg14fExggonJwhGW6GLP4HMgtKFK0v8bUQV0qkLz4m0gIDDhHRBUGtDxEE3DJysEdzRkEJevYyKV5foqToYFeRvviYSAuCKBrsFDcFbDYbTCYTrFYrkpOT9e4OEYWA3SEqXvPhz7UAJOvafPX7l1A1cVrA60ukCgYq8caC0Zz6IcPy5/uba3CIyPD8Pb1a8foQiWADABBFXAngygD7C3guxP3qxFm88FG1z/tYBZioA6eoiMjQNDu92ku4UYszaM0Y1g9jB/dSdA+rABN1YMAhIsPS5PRqmaMWIIqqhpuuWAWYyD8MOERkWKqfXh2CURs5rAJM5B8GHCIyLNVOr/7733UZtemKVYCJlOMiYyIyLFVOr9Zx1EYKqwATKcOAQ0Rhwe+t2QoEVZ24pQW49FLP9rNnge7dg+pXsFgFmMg3Bhwi0p2/27iVcq5byS8uhwC4hRyv61bCbNSGiPzHNThEpCvNtnFf4Pe6Falws2YNww1RhAmLgPPiiy8iIyMDCQkJGDVqFHbv3u31+rfffhtDhgxBQkIChg4dig8//DBEPSUiNWmyjVtCXnYadhRMwhsLRuOPdwzDGwtGY0fBJPdw42379/z5Qb0/EYWe7gHnzTffxOLFi7F8+XKUl5fjuuuuQ25uLk6ePCl5/c6dOzF79mzcc8892Lt3L2bOnImZM2eisrIyxD0nomCpvo3bi85F80ZmpmB3TSM2VNSh7FBDQFNSdoeIskMNrmfwDCii8KL7WVSjRo3CjTfeiBdeeAEA4HA4kJ6ejgceeABLly71uP72229Hc3MzSkpKXG2jR4/GsGHD8NJLLyl6T55FRRQeNlTU4cH1FT6v++MdwzBjWD/Z1/1ZoNx5vc/eP85Gz5Yznhf5+L9FrdYMeaPFImyiSBMxZ1G1tbVhz549WLZsmastJiYGkydPRllZmeQ9ZWVlWLx4sVtbbm4u3nvvPdn3aW1tRWtrq+t3m80WXMeJSBVqbOP2Fja6bqf+prkNC9eVQwRQu3Ka9AMVhJv84nKPaTXnmiEt6tHoEaiIIp2uAef06dOw2+3o06ePW3ufPn1w4MAByXssFovk9RaLRfZ9ioqKUFhYGHyHiUhVQW3jhvewcX9xOXokdkPTuXZXe4wA3Prlv/Hi+ys9npVZUAKzKQE7HKLXU8a9rRkS0LFmaEqWWbXRFT0CFZER6L4GJxSWLVsGq9Xq+jl69KjeXSKKWp3XruyuacTjUwM7fkDJAuXO4QYAvn56mmS4ySgoUbTeJ5RrhoDQLcImMiJdR3B69eqF2NhYnDhxwq39xIkTMJvNkveYzWa/rgeA+Ph4xMfHB99hIgqK3FTLfeMz8f4X9W7tZh9TML7CRmfd7O346tkfeLQPfehNnIl3L9rn7dgG1Y5+UMifQMXCf0TudA04cXFxGDFiBLZs2YKZM2cC6FhkvGXLFixatEjynpycHGzZsgUPPfSQq23Tpk3IyckJQY+JKFDeplpWb6/Bi3dej57d4xUvolUaIuTW2mQUlEi2e1vvo8rRD34IdaAiMhLdKxkvXrwY8+bNww033ICRI0fiueeeQ3NzM+6++24AwNy5c9GvXz8UFRUBAB588EHcdNNN+N3vfoepU6di/fr1+Pzzz7F69Wo9PwYReaFk7cqKD77EjoJJiteuKAkRUuHmt+Pn4s85P/Zo97XeBwh+zZC/Qh2oiIxE94Bz++2349SpU3jiiSdgsVgwbNgwlJaWuhYSHzlyBDExF5cKjRkzBuvWrcNjjz2GRx99FFdeeSXee+89ZGdn6/URiMgHLaZavIUNf0dtfK33cQr46IcAhTpQERmJ7nVw9MA6OETu1K6x0vV5Fuu3ePitL3ze56veTVfOaS/gYthQEm5iBKDzulx/t1yHctu21GcELgYq7qKiaOLP9zcDDgMORTm1v6ylnpfSPQ6NzW0+731jwWi/F8s636/s0cmSr3cONs5Q4O96HymhLLzHOjhEHRhwfGDAIeogt/A30NEBb8/z9n80zqkWf9bguD9A+p6c32w2TChgJWOiCKpkTET6UbtonZKaLVKCWrvy5pvAHXdIvGHHO+4wUChwnqVFRMow4BBFKbUX/iqtS5PSvRsamy8W4PNV70aWggMyGQqIohcDDlGUUrvGitLrHp92DczJCYGPqrS3A3Fxnu2nTgG9eil/jp84RUQUWRhwiKKU2jVWlF5nTk4IfFRFwahNsKSCzKYqCxf5EkUYBhyiKKV2jRXNa7ZIhZulS4ELRUDVILVbqeuBnU487JIovEXFYZtE5MlZtA7w/6DLUDzv4s2CdLgRRdXDTX5xucc6IqlwA/CwS6Jwx4BDFMXystOwas5wmE3u00tmU0JAIxNqPy8UU1KA9x1g3qh9ejgRqYdTVERRLi87DVOyzKotoFXleTLBpqz6dMezAuqZPH9OJpfCwy6Jwg8DDhGpvp06qOfJhJuMghLg5V2aLO4NNqDwsEui8MMpKiLSnN0houxQAzZU1KHsUIP0mpU335QMNxkFJW7HLTgX95ZW1qvWv0ADioCO3VQ87JIo/HAEh4g0pegcJW+jNl0EUmXZF187wKRocXo4EamHIzhEpBm5nUnOUZh/VRyVDDef76yUDDdOwS7u7TqiBMDnDrAeid3c2gNeOE1EIcERHCLShK+zqWpXTgNWSr0ooq6iTtF7BLJ2xtuI0qo5wz1ecx4loeZCbCLSHgMOEWnC286k2pXTPBt/+lPglVcAqF892UnutPPORft2FEySDTI814oocjDgEJEmpEZXJIMN4FHXRouqyP6cns4gQxT5uAaHiDTRdXRFLtyUVZ/2aNOiKrI/p6cTUeTjCA4RacI5ClP26GTJ1zMKSpBmSsAOmVEYZ1VkuTUx/i7uVftUdCIKbww4RKSJ2BjBa7gBgDMt7fjNB1WYnGWWXLSrZpVlrdb1EFF4YsAhIll2hxhYuNiwAZg506O569bvs612vPJJLV75pFa2QrFaVZY1P+2ciMIKAw4RSVJUoE+KH0X7Ouu8k0mL2jLOdT35xeUQALeQw6J9RMbDRcZE5MFXgT7JYxIcDslwM/b+V32GG+Bi4CjcWCV9lIMKVD/tnIjCFkdwiMiNP9upXaMdAY7aSD3fuZNJq63aap+eTkThiQGHiNwo3U696+sGxAgCcgb38rhm45Dv4oEZBQH3QeudTGqfnk5E4YcBh4jcKA0XY69MlWz3d9RGCncyEVGwGHCIyI2ScCFXtC/YcMOdTESkFgYcoggX8FZuGd62UysJNjECEMgaYe5kIiI1MeAQRbCAt3J7IbedWumojVy4cUaWhyZ/B9Zv2/BexXE0Nre5Xg+0QjERkRRBFEVt9mOGMZvNBpPJBKvViuTkZL27QxQQuZOxnUEi2G3PzvA09LOPsPrdpzxe93c6qmdiNxT9cKirT2qPPBGR8fnz/c0RHKIQUfMLPaCt3H7Ky05D3tC+kq8FstYmLlZAUkI3bKioc31+7mQiIq0w4BCFgJpTSXaHiLWf1Cg+GTugECGKQIxnHdBZD6/FVz3SIJxrlwxX3pw404a7/vqp6/dgp9KIiLxhJWMijQVUFdjLs8at3IoVH3yp6PqA6skIgmS4ySgowedxvdB0IdwEO5kUyOcnIlKKAYdIQ76mkgDlRxPIBSVv/K4nI1GR+NP0bNeUlDPY9Ezshj7J8f49u4tQHM1ARNGLU1REGlJaFdjXVJK3oCTF73oyfhy1IAL45lw7Xr93FGIEASfPtKDXZfFY8lYFTtha/Zq6CsXRDEQUnTiCQ6QhpVNEvq7zFZQ687ueTIDnSJ0+24qcQZdjxrB+GDu4F379/Wvc3t8fWh/NQETRhyM4RBpSOkXk6zp/AoDiejI9ewJNTR7NSndI9eoej7JDDa5dYVOyzFg1Z7jHYmoleDQDEamNAYdIQ96qAgPKp5KUBoDHp16N+WMzfY/cBHH6twDAlNgNS97+Ahab566wHQWTXNvhe3WPx5K3v8AJW3Cfn4jIX5yiIlKR3SGi7FADNlTUoexQA4COqSLAc+rGn6kkZ1CSu0pAR8DwGW62bZMMNzm/2aw43IgAms61u4UboGNX1P3F5XjqgyoAwLRr+2Lslb3w6+8H//mJiPylW8Cpra3FPffcg8zMTFx66aUYNGgQli9fjra2Nq/3TZgwAYIguP3cf//9Ieo1kTznFu7ZL+/Cg+srMPvlXRi3ciuAjqrCZpP7KIzZlKC42rDz+ARAWVDoGrTsDrEj2Eyc6PHssurTiqeU+iTHo0diN8nXnCM0r35S6/rspZX1yMtOC/rzExH5S7cpqgMHDsDhcOAvf/kLBg8ejMrKSixYsADNzc149tlnvd67YMECPPnkk67fExMTte4ukVdyxyY4a72smjPcbeomkErGzqDQdY1L1zU3HkUFRRG1v53u+cCqKuDqq3Gyok7R+y+aOBg5gy53K9bnTefPnpedhilZZh7NQEQho1vAycvLQ15enuv3gQMH4uDBg1i1apXPgJOYmAiz2ax1F4kU8efYhGC3QvsKCl2DltwBmeh0BJ3S9T1jB/fya7Gz1JER3ApORKESVmtwrFYrUlJ8LzZ8/fXX0atXL2RnZ2PZsmU4d+6c1+tbW1ths9ncfojU4k+tGzU4g8KMYf2QM+hyt2mpzkFLKtzsybwOdrvDrU3J+p7Lu8fBYv0Wp8+0+tVXtT87EZFSYbOLqrq6Gs8//7zP0Zs777wTAwYMQN++ffGf//wHBQUFOHjwIN555x3Ze4qKilBYWKh2l4kAqFfrJljOoCU3auNcRPxGl6J6zvU9+cXlrkXEnYkAGprb8PBbXwAAYgTA38LDrHNDRKGm+gjO0qVLPRYBd/05cOCA2z11dXXIy8vDrFmzsGDBAq/Pv++++5Cbm4uhQ4firrvuwt/+9je8++67OHTokOw9y5Ytg9Vqdf0cPXpUlc9KBKhX6yZYJ8/4DjfO67qSWwgsJZBTFVjnhohCTfURnCVLlmD+/Plerxk4cKDrvx8/fhwTJ07EmDFjsHr1ar/fb9SoUQA6RoAGDRokeU18fDzi44M7N4dIjlq1boIyeDBmSIR8qa3fcmGj8/oei60FK0r2o7G5XfYtlYzksM4NEelF9YCTmpqK1NRURdfW1dVh4sSJGDFiBNasWYMYiROMfamoqAAApKVxqynpw9sUT0hqvSgs2qckbDjX95QdavAaboCOcPP41KtxvOlbvPJJrWe3Lvwn69wQkR50W2RcV1eHCRMm4IorrsCzzz6LU6dOwWKxwGKxuF0zZMgQ7N69GwBw6NAhrFixAnv27EFtbS3ef/99zJ07F+PHj8e1116r10ch0qfWS1mZZLjJLChBpkS4AZSHDaVrZnolxePx6dfgpTnDkcY6N0QURnRbZLxp0yZUV1ejuroa/fv3d3tNvLCFtb29HQcPHnTtkoqLi8PmzZvx3HPPobm5Genp6bjtttvw2GOPhbz/RF3lZadh0pA++H9ltTjceA4DUhLxk5wMxF2iwb9HyIzaQBSxqmsdHPhxPtUF/q4rYp0bIgo3giiKASwZjGw2mw0mkwlWqxXJycl6d4cMwqPAHi6ez6TaKIYoAlJTufv2AdnZrl/tDjGosGF3iBi3cqvPdUU7Cib5fG6wfSEicvLn+5sBhwGHVCBXydj5Na7KVI2XUZtgSYWQTVUW5BeXd7xF525c+E8lnykkoY+IogYDjg8MONFDzdEDuWc5Rzvkiv35M9ohSyrcjB0L7NgR2PM68RZCAAQcUEIS+ogoqvjz/R02hf6I1KZk9EBpAPL2LNOlcYorGft9VIGGozaAdmdo+XN8BaeriEgLDDhkSEq+uAFloxO+nnX32AxFffK7mq/G4UbLM7T8Ob6C51MRkRbC6iwqIjX4+uIGgKXv7EN+cbnHl7AztJRW1it+1oaK44r6pbia7/Dh0uFGFFULN4C2Z2iFy/EVRBS9GHDIcJR8cTeda/caWgo3Vrmmr3w9q6G5DSndu3k9rDJNaTVfQQD27pV4I/WXymkZQsLl+Aoiil4MOGQ4wY4KdB65UPqsHwzrBwAeIUdxgb2KioBHbewOEWWHGrChog5lhxpgV3hYlJYhRMkJ5YpDHxFRALgGhwxHrVEB56JaJSZnmXFjZkpgBfb8XGvTeWF07elzeGP3EVhs/u9y0vIMLd2PryCiqMeAQ4bj64tbKeeOIaUhIDZG8L+ar1S42bsXGDZM8nKp3VxddV5I7S3kaB1CnMdXBFtVmYgoEKyDwzo4huTc+QR4fnGLAHokdoNVZh1O17o13p4FBFjPJYAdUnK7uSQfD+W1d7QuxsdKxkSkFhb684EBJzr4KmDnT2hRNQRIhZucHGDnTtlbfBUTlPPGgtGKtmEzhBBRJGChPyL4PgDSn+kTVQ6TDKKuja/dXHKULpKOjRFYj4aIDIUBhwzN2xe3v6ElqBAQZNG+QHeGnT7TCrtD5GgMEUUdbhOnqOYMLTOG9UPOoMvVDwLTpqlStC/QnWErPvgS41ZudRUuJCKKFgw4FJYCre2it879hiAAH3zgeVEAy9581ZXxpmt1ZiKiaMApKgo7pZX1+PX7+2GxtbrazMnx+PX3rwnrrcXOhcgJNYfw0cs/87zA4ZCfqvLB25ZuX3i4JRFFI47gUFgprazH/cXlbuEGACy2VtwfxqMQzi3cZY9Olgw3pfuOBxxunJx1Zcwm9+kqc3I8fjS8n9d7gzlXiogoEnEEh8KG3SFi6Tv7vF6z9J19YTcK4TyQs2blNI/Xps7/I/b3GQTz+/tV6bfcwuiS/xzH38vrfN7Pwy2JKFow4FDY2HWoAU3n2r1e03SuHbsONWDslb1C1CvfYmNjUCbRnlFQ4vrvFlsrXthajQcnXxn8+0ns5uLhlkRE7jhFRWGj7OvTql4XEhLTTiVXjXMLN05/2PxfzabYeLglEZE7juBQGFE6fRPa6SnJKr+x0v9uIBVsOlum0RQbD7ckInLHERwKG0qL6IWy4m5pZT3GrdyK2S/vwoPrKzD75V0BhxsA+OZcO3Z93aB2NwF4WYRsSgjsvCwiogjGERwKG6MHXo4eid28rsPpmdgNoweGJuB0PdzyV1v/igWfved5oSh2TD1dONvKl7JDDRg7WJs1RKocKUFEZAAMOBQ2YmMEPP3DobjfS1Ao+uHQkHxZO3dGOcNNrcQOKQCw2x2IRUew+F52H/yz8oSCp2tbtJDnShERcYqKwkxedhpemjMc5mT3aZY0UwJeCuE0i/NwS7PttGS4yfjlRmQUlLjVlZkzKkPRs3MGhs8OMCIio+IIDoWdcJhmOXmmRXbUpvNam851ZUYP8j3Fdln8JTh5pgVlhxo4dUREpCEGHApLek+zzLi+v0db3t3P40DvTLe2znVllEyxnW09j4ff+gJAx6jU8ulZXPxLRKQBTlERdSYIkrVtMgpK3MKNXF2Zi1Ns8T7fiodgEhFphwGHyEki2Gwc8l1kdtn+7auuTF52Gj5ZejPeWDAaf7h9GFK6d5N8O+dS48KNVRFzWjoRUaRgwCFN2R0iyg41YENFHcoONYTnF7nMqA1EEd3efjOgujLOKTZzcgIam+XX5PAQTCIibXANDmmmtLIehRurUG+9uBA37NadyJ3wLXYEsWAXPCs93JKHYBIRqYsjOKQJZ5G8zuEGCKN1J7/9reyojTPcODlHY2YM64ecQZf7tfPJn0MwI2K0i4goQnAEh1TXtUheZyI61rAUbqzS5EwmRXyM2qjJeQimxdoiW94vpXs3NJxpxbiVW8N7tIuIKIJwBIdU5yySJ0e3dScnTkiHG4dDk3ADXDwEE5A/IrSxuR2L1u8N39EuIqIIxIBDqgvLdSeCAJjNHs1l1adh13gmSO4QTF+4y4qIKHAMOKQ6f9adhITEqE3e3c8jo6AEs1/ehXErt2o+SpKXnYaPH5mIlO5xft3HXVZERIFhwCHVOdedyE3JyBXJU53Con2hmgrac/gbNDa3BXQvd1kREfmHAYdU523dia8ieaqRCDb/HDbZ7Rwpp1BNBQUTUkI22kVEZBAMOKQJuXUnSorkBbVduls3yXBTVn0a+bkPyd4WiqmgQEJKyEa7iIgMRtdt4hkZGTh8+LBbW1FREZYuXSp7T0tLC5YsWYL169ejtbUVubm5+POf/4w+ffpo3d2oYXeIqpzkHUiRvKCKA3rZ/n2yok5Rn7WcClKyZbyzkI12EREZkO51cJ588kksWLDA9XtSUpLX6x9++GF88MEHePvtt2EymbBo0SL88Ic/xCeffKJ1V6OC2tWH/TkV3FkcsOuXv3ONjOzIz1//CnT6Z8il09bvcFj47Jy6yy8uhwD4DDlm1sEhIgqY7gEnKSkJZontu1KsViteeeUVrFu3DpMmTQIArFmzBldffTV27dqF0aNHa9lVwws4YKgg4OKACov2+Ro9EdARKLSeCnJO3UmFyMenZqFn97igR86IiCgMAs7TTz+NFStW4IorrsCdd96Jhx9+GJdcIt2tPXv2oL29HZMnT3a1DRkyBFdccQXKyspkA05raytaW1tdv9tsNnU/hAHoXX3Yn+KAOYMuBxobgcslRobsdiDGc2mZt9GTUE8FBXu+FRER+aZrwPnFL36B4cOHIyUlBTt37sSyZctQX1+P3//+95LXWywWxMXFoUePHm7tffr0gcVikX2foqIiFBYWqtl1w/E7YKjMr+KAAR61IDd6osdUkD9Td0RE5D/VA87SpUuxcuVKr9d8+eWXGDJkCBYvXuxqu/baaxEXF4ef/exnKCoqQnx8vGp9WrZsmdt72Ww2pKenq/Z8I9C7+rDStS8zru/v2bh3LzBsmKL7OXpCRBQdVA84S5Yswfz5871eM3DgQMn2UaNG4fz586itrcVVV13l8brZbEZbWxuamprcRnFOnDjhdR1PfHy8qoHJiPRehOtrjUztymnSNwZwhhRHT4iIjE/1gJOamorU1NSA7q2oqEBMTAx69+4t+fqIESPQrVs3bNmyBbfddhsA4ODBgzhy5AhycnIC7jPpvwjX2xoZyXAzdy7w2muqvb9aW+OJiCg86LYGp6ysDJ9++ikmTpyIpKQklJWV4eGHH8acOXPQs2dPAEBdXR1uvvlm/O1vf8PIkSNhMplwzz33YPHixUhJSUFycjIeeOAB5OTkGH4HldZfwKFchCv3WbqukSn/051I+VZiQbgKJ3937kPt6Wa8sfsILLaLC9GD2RpPRET6E0RRhW+LAJSXl+PnP/85Dhw4gNbWVmRmZuInP/kJFi9e7JpOqq2tRWZmJj766CNMmDABwMVCf2+88YZboT+lW82BjjU4JpMJVqsVycnJWnw8Valdm0bP91LyfLtDRGysTJFtFf5xlepDV84Yp+XWeCIi8o8/39+6BRw9RVLAkatNo+UXsFajRYo+y/F9QG6u580q/WMq1wcpzmm5HQWTOF1FRBQG/Pn+1r0ODsnTqzaNFotwlXyWvKF9pW9WKdx464Pk20LbrfFERKQdHrYZxvypTRPuvH2WxLZvUSO1kNhuVy3c+OqDN1qeT0VERNrgCE4Y07s2jZrk+qjm9u9A++CLludTERGRNjiCE8b0rk2jJqk+SoWbvZt2aRJu5PrgjYCOBdBan09FRETq4whOGNO7No2aOn8WyekoADm/2Ywdk0aGpA++IlSoz6ciIiJ1cQQnjDlr0wAXv3CdIu0L2PlZpMLN7747B5kFJZp/Fm9/z67MpgRuESciimDcJh7m28SB0NbB0cwjjwDPPuvRnFFQEvLPIvf3vOPGK5DRK5GVjImIwhTr4PgQaQEHiPCjBCRO/z43+CpsenuLbp8lov+eRERRinVwDCgiD4j84gvpU75FEYkAZoS6P51E5N+TiIgUY8AhbUiM2gDQbIcUERFRZ1xkTOpqbZUONyoX7SMiIvKGIzikHo7aEBFRmGDAIXVIhZvaWmDAAL8ew8W/RESkBgYcCk7//kBdnWd7AKM2htgOT0REYYFrcChwguAZbtauDTjc5BeXexyGabG2IL+4HKWV9UF0lIiIog0DDvlv5UrpKSlRBObN8/txdoeIwo1VkscnONsKN1bB7uBaHiIiUoZTVOQfqWAzbhzw738H/MjdNY0eIzediQDqrS3YXdPI2jVERKQIAw4ps28fcO21nu0q7JA6eUY+3ARyHREREQMO+abx9u/eSQmqXkdERMSAQ/La2oD4eM/28+eB2NiAH9t1K/iIAT2RZkqAxdoiuQ5HQMfp3iMzUwJ+TyIiii4MOCRNo1Ebua3g378uDau310AA3EKOsxfLp2exHg4RESnGXVTkSSrcfP21KuFGbiv46u01uG98Jswm92kosykBq+YMZx0cIiLyC0dw6KLx46V3Q6mw1sbXVnABwPtf1OPjRyZiz+FvWMmYiIiCwoATpbqug8kZ3MvzojfeAO64Q5X3U7oVfM/hb7gVnIiIgsaAE4U6r4P58Rf/h9+W/snzIpUPyORWcCIiCiUGnCjjXAcjAqhdOc3j9ZMTpqD3R/+n+vtyKzgREYUSA04Uca6D6Ws9iU9e+qnH6xkFJUgzJWCHQ1R93cvIzBRuBSciopDhLqoosrumEWWPTpYNN0DHOpgXtn6l+nvHxghYPj0LwMWt307cCk5ERGpjwIkW589LLiQe+MgGV7hx+sPmrzQ5vTsvOw2r5gznVnAiItKcIIoqryaNADabDSaTCVarFcnJyXp3R3syRfu6BpvO0kwJ2FEwSZMRla47uLgVnIiIlPDn+5trcIxOItyM+vlanEiS2BbeiZand8fGCNwKTkREmuIUlVEtWyYZbkr3HfcZbpy4ZZuIiCIVA44RCQLw9NPubZs3A6KIvOw0PDz5O4oewy3bREQUqRhwjOTLL6XX24gicPPNrl8XTRoMc7LEKeEXCOhYg8Mt20REFKkYcIxCEICsLPe2xx6TrEgcGyPg19+/BgK4ZZuIiIyJASfSWa3yozYrVsjexi3bRERkZNxFpaKQb38eOxbYudO97Ve/Av73fxXdnpedhilZZm7ZJiIiw2HAUUnnAyyd0kwJWD49S/3REIcDiI31bLfbgRj/BuW4ZZuIiIyIU1QqcB5g2TncAIDF2oL84nJ1qwI//rhnuMnJ6ZiS8jPcEBERGZVu34jbtm2DIAiSP5999pnsfRMmTPC4/v777w9hz905D7CUKgftbCvcWAW7Q4WC0YLgOf3U1OQ5TUVERBTldJuiGjNmDOrr3Uc2Hn/8cWzZsgU33HCD13sXLFiAJ5980vV7YmKiJn1UYndNo8fITWciVKgKfPw40K+fxMOj7pQNIiIiRXQLOHFxcTCbza7f29vbsWHDBjzwwAMQZM5OckpMTHS7V09Kq/0GXBW4qAh49FH3tv37PbeEExERkUvYLNp4//330dDQgLvvvtvnta+//jp69eqF7OxsLFu2DOfOnfN6fWtrK2w2m9uPWpRW+/W7KvCZMx1TUp3CzZGHl2HD3mMoi++jzpQXERGRQYXNLqpXXnkFubm56N+/v9fr7rzzTgwYMAB9+/bFf/7zHxQUFODgwYN45513ZO8pKipCYWGh2l0GAIzMTEGaKQEWa4vkOhwBHbVl/KoKvG4dcNddbk3fL1iP/+AyYH0FAA13aBERERmAIIrqLuRYunQpVq5c6fWaL7/8EkOGDHH9fuzYMQwYMABvvfUWbrvtNr/eb+vWrbj55ptRXV2NQYMGSV7T2tqK1tZW1+82mw3p6emKjltXwrmLCoBbyHFOtCkunNfeDvTvD5w86Wqq+/6PMe7quR7hye9nExERRTibzQaTyaTo+1v1gHPq1Ck0NDR4vWbgwIGIi4tz/b5ixQo8//zzqKurQ7du3fx6v+bmZlx22WUoLS1Fbm6uonv8+QMpFXQdnG3bgIkT3ZrsFV9g3IenZBcxO0eHdhRMki3OF/Lig0RERBrx5/tb9Smq1NRUpKamKr5eFEWsWbMGc+fO9TvcAEBFRQUAIC1N31GMgKsCiyIwYQKwffvFttGjgZ07sfvrRtRbj8rfCu87tEJafJCIiCiM6L7IeOvWraipqcG9997r8VpdXR2GDBmC3bt3AwAOHTqEFStWYM+ePaitrcX777+PuXPnYvz48bj22mtD3XUPzqrAM4b1Q86gy32Hm/37O4rzdQ43mzYBZWWAIAS1QyukxQeJiIjCjO4B55VXXsGYMWPc1uQ4tbe34+DBg65dUnFxcdi8eTNuueUWDBkyBEuWLMFtt92GjRs3hrrbwbvvPiA7++LvPXsCra3A5MmupkB3aIW0+CAREVEY0n0X1bp162Rfy8jIQOclQunp6fj4449D0S3tSBXtW7sWmDfP49JAd2iFpPggERFRGNN9BCeqrFzpGW6amiTDDdAx5bV8ekdBv66TXc7fl0/P8pgK07z4IBERUZhjwAkFZ9G+pUsvthUWdiwwNpm83pqXnYZVc4bDbHKfhjKbEmS3iGtWfJCIiChC6D5FZXjr1wOzZ7u3HT3aUe9GIX93aGlSfJCIiCiCMOBopb0duOIKwGK52PaTnwB/+1tAj3Pu0FJ67fLpWcgvLocA6eKDUlNbRERERsEpKi18/DEQF+cebioqAg43gQhkaouIiMgoOIKjtptvBrZuvfj7qFGuujahFnDxQSIiogjHgKOmbdvcw83//R8wZYpu3QH8m9oiIiIyCgYcNQ0cCIwbBxw7Bhw82DFNRURERCHHgKOmK64A/v1vvXtBREQU9bjImIiIiAyHAYeIiIgMhwGHiIiIDIcBh4iIiAyHAYeIiIgMhwGHiIiIDIcBh4iIiAyHdXAMxO4QeSwDERERGHAMo7SyHoUbq1BvbXG1pZkSsHx6Fg/WJCKiqMMpKgMoraxHfnG5W7gBAIu1BfnF5SitrNepZ0RERPpgwIlwdoeIwo1VECVec7YVbqyC3SF1BRERkTEx4ES43TWNHiM3nYkA6q0t2F3TGLpOERER6YwBJ8KdPCMfbgK5joiIyAgYcCJc76QEVa8jIiIyAgacCDcyMwVppgTIbQYX0LGbamRmSii7RUREpCsGnAgXGyNg+fQsAPAIOc7fl0/PYj0cIiKKKgw4BpCXnYZVc4bDbHKfhurZvRtevPN61sEhIqKow4BjEHnZaXh8ahZSuse52hqb27Higy9ZB4eIiKIOA45BlFbWY+G6cjQ2t7m1s9gfERFFIwYcA2CxPyIiIncMOAbAYn9ERETuGHAMgMX+iIiI3DHgGACL/REREbljwDEAFvsjIiJyx4BjACz2R0RE5I4BxyDkiv2ZTQlYNWc4i/0REVFUuUTvDpB68rLTMCXLjN01jTh5pgW9kzqmpThyQ0RE0YYBx2BiYwTkDLpc724QERHpilNUREREZDgMOERERGQ4mgWcp556CmPGjEFiYiJ69Oghec2RI0cwdepUJCYmonfv3njkkUdw/vx5r89tbGzEXXfdheTkZPTo0QP33HMPzp49q8EnICIiokilWcBpa2vDrFmzkJ+fL/m63W7H1KlT0dbWhp07d+K1117D2rVr8cQTT3h97l133YX9+/dj06ZNKCkpwfbt23Hfffdp8RGIiIgoQgmiKGp6AuPatWvx0EMPoampya39n//8J6ZNm4bjx4+jT58+AICXXnoJBQUFOHXqFOLi4jye9eWXXyIrKwufffYZbrjhBgBAaWkpbr31Vhw7dgx9+/ZV1CebzQaTyQSr1Yrk5OTgPiARERGFhD/f37qtwSkrK8PQoUNd4QYAcnNzYbPZsH//ftl7evTo4Qo3ADB58mTExMTg008/lX2v1tZW2Gw2tx8iIiIyLt0CjsVicQs3AFy/WywW2Xt69+7t1nbJJZcgJSVF9h4AKCoqgslkcv2kp6cH2XsiIiIKZ34FnKVLl0IQBK8/Bw4c0KqvAVu2bBmsVqvr5+jRo3p3iYiIiDTkV6G/JUuWYP78+V6vGThwoKJnmc1m7N69263txIkTrtfk7jl58qRb2/nz59HY2Ch7DwDEx8cjPj5eUb+IiIgo8vkVcFJTU5GamqrKG+fk5OCpp57CyZMnXdNOmzZtQnJyMrKysmTvaWpqwp49ezBixAgAwNatW+FwODBq1CjF7+1cV821OERERJHD+b2taH+UqJHDhw+Le/fuFQsLC8XLLrtM3Lt3r7h3717xzJkzoiiK4vnz58Xs7GzxlltuESsqKsTS0lIxNTVVXLZsmesZn376qXjVVVeJx44dc7Xl5eWJ119/vfjpp5+KO3bsEK+88kpx9uzZfvXt6NGjIgD+8Ic//OEPf/gTgT9Hjx71+V2v2Tbx+fPn47XXXvNo/+ijjzBhwgQAwOHDh5Gfn49t27ahe/fumDdvHp5++mlccknHwNK2bdswceJE1NTUICMjA0BHob9FixZh48aNiImJwW233YY//elPuOyyyxT3zeFw4Pjx40hKSoIgGPMgSpvNhvT0dBw9epRb4TXAv6+2+PfVFv++2uLfVzuiKOLMmTPo27cvYmK8LyPWvA4O6YO1frTFv6+2+PfVFv++2uLfNzzwLCoiIiIyHAYcIiIiMhwGHIOKj4/H8uXLuT1eI/z7aot/X23x76st/n3DA9fgEBERkeFwBIeIiIgMhwGHiIiIDIcBh4iIiAyHAYeIiIgMhwEnCtTW1uKee+5BZmYmLr30UgwaNAjLly9HW1ub3l0zjKeeegpjxoxBYmIievTooXd3It6LL76IjIwMJCQkYNSoUR4H81Jgtm/fjunTp6Nv374QBAHvvfee3l0ylKKiItx4441ISkpC7969MXPmTBw8eFDvbkUtBpwocODAATgcDvzlL3/B/v378Yc//AEvvfQSHn30Ub27ZhhtbW2YNWsW8vPz9e5KxHvzzTexePFiLF++HOXl5bjuuuuQm5uLkydP6t21iNfc3IzrrrsOL774ot5dMaSPP/4YCxcuxK5du7Bp0ya0t7fjlltuQXNzs95di0rcJh6lnnnmGaxatQpff/213l0xlLVr1+Khhx5CU1OT3l2JWKNGjcKNN96IF154AUDH2XHp6el44IEHsHTpUp17ZxyCIODdd9/FzJkz9e6KYZ06dQq9e/fGxx9/jPHjx+vdnajDEZwoZbVakZKSonc3iNy0tbVhz549mDx5sqstJiYGkydPRllZmY49I/Kf1WoFAP5/rU4YcKJQdXU1nn/+efzsZz/TuytEbk6fPg273Y4+ffq4tffp0wcWi0WnXhH5z+Fw4KGHHsLYsWORnZ2td3eiEgNOBFu6dCkEQfD6c+DAAbd76urqkJeXh1mzZmHBggU69TwyBPL3JSICgIULF6KyshLr16/XuytR6xK9O0CBW7JkCebPn+/1moEDB7r++/HjxzFx4kSMGTMGq1ev1rh3kc/fvy8Fr1evXoiNjcWJEyfc2k+cOAGz2axTr4j8s2jRIpSUlGD79u3o37+/3t2JWgw4ESw1NRWpqamKrq2rq8PEiRMxYsQIrFmzBjExHLzzxZ+/L6kjLi4OI0aMwJYtW1yLXx0OB7Zs2YJFixbp2zkiH0RRxAMPPIB3330X27ZtQ2Zmpt5dimoMOFGgrq4OEyZMwIABA/Dss8/i1KlTrtf4b8XqOHLkCBobG3HkyBHY7XZUVFQAAAYPHozLLrtM385FmMWLF2PevHm44YYbMHLkSDz33HNobm7G3XffrXfXIt7Zs2dRXV3t+r2mpgYVFRVISUnBFVdcoWPPjGHhwoVYt24dNmzYgKSkJNe6MZPJhEsvvVTn3kUfbhOPAmvXrpX9cuD//OqYP38+XnvtNY/2jz76CBMmTAh9hyLcCy+8gGeeeQYWiwXDhg3Dn/70J4waNUrvbkW8bdu2YeLEiR7t8+bNw9q1a0PfIYMRBEGyfc2aNT6nu0l9DDhERERkOFyIQURERIbDgENERESGw4BDREREhsOAQ0RERIbDgENERESGw4BDREREhsOAQ0RERIbDgENERESGw4BDREREhsOAQ0RERIbDgENERESGw4BDREREhvP/AUOyKpicOIvDAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "theta = jnp.array([1., 1.])\n",
    "init_lr = 1\n",
    "for i in range(1000):\n",
    "    if i % 100 == 0:\n",
    "        print(f\"loss: {loss_fn(theta, xs, ys):<.2f}\")\n",
    "    theta = update(theta, xs, ys,init_lr/(1+i))\n",
    "\n",
    "plt.scatter(xs, ys)\n",
    "plt.plot(xs, model(theta, xs), color='red')\n",
    "\n",
    "w, b = theta\n",
    "print(f\"w: {w:<.2f}, b: {b:<.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JIT Compilation with JAX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When to JIT:\n",
    "* It usually only saves time if the compiled function is complex and you will run it numerous times. \n",
    "* Generally, you want to jit the largest possible chunk of your computation; ideally, the entire **update** step. This gives the compiler maximum freedom to optimise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Caching behaviour:\n",
    "* JAX will only compile a function once (when first invoked), and then cache the compiled code for reuse.\n",
    "* If specifying `static_argnums`, for every unique set of static arguments, JAX will compile a new version of the function and cache it separately.\n",
    "* Avoid calling `jax.jit` inside loops, as this will result in a new compiled function being created and cached for every iteration of the loop. Because the cache relies on the hash of the function, it becomes problematic when equivalent functions are redefined. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jit called in a loop with partials:\n",
      "576 ms ± 91.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n",
      "jit called in a loop with lambdas:\n",
      "712 ms ± 61.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n",
      "jit called in a loop with caching:\n",
      "4.6 ms ± 986 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "import jax \n",
    "from functools import partial\n",
    "\n",
    "def unjitted_loop_body(prev_i):\n",
    "  return prev_i + 1\n",
    "\n",
    "def g_inner_jitted_partial(x, n):\n",
    "  i = 0\n",
    "  while i < n:\n",
    "    # Don't do this! each time the partial returns\n",
    "    # a function with different hash\n",
    "    i = jax.jit(partial(unjitted_loop_body))(i)\n",
    "  return x + i\n",
    "\n",
    "def g_inner_jitted_lambda(x, n):\n",
    "  i = 0\n",
    "  while i < n:\n",
    "    # Don't do this!, lambda will also return\n",
    "    # a function with a different hash\n",
    "    i = jax.jit(lambda x: unjitted_loop_body(x))(i)\n",
    "  return x + i\n",
    "\n",
    "def g_inner_jitted_normal(x, n):\n",
    "  i = 0\n",
    "  while i < n:\n",
    "    # this is OK, since JAX can find the\n",
    "    # cached, compiled function\n",
    "    i = jax.jit(unjitted_loop_body)(i)\n",
    "  return x + i\n",
    "\n",
    "print(\"jit called in a loop with partials:\")\n",
    "%timeit g_inner_jitted_partial(10, 20).block_until_ready()\n",
    "\n",
    "print(\"jit called in a loop with lambdas:\")\n",
    "%timeit g_inner_jitted_lambda(10, 20).block_until_ready()\n",
    "\n",
    "print(\"jit called in a loop with caching:\")\n",
    "%timeit g_inner_jitted_normal(10, 20).block_until_ready()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic Vectorization in JAX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`jax.vmap` generate such a **vectorized** implementation of a function automatically, by **tracing** the function similarly to jax.jit, and automatically **adding batch axes** at the beginning of each input.\n",
    "\n",
    "If the **batch dimension** is not the first, you may use the **in_axes** and **out_axes** arguments to specify the location of the batch dimension in inputs and outputs. These may be an integer if the batch axis is the same for all inputs and outputs, or lists, otherwise.\n",
    "\n",
    "`jax.vmap` also supports the case where `only one` of the arguments is `batched`: for example, if you would like to convolve to a single set of weights w with a batch of vectors x; in this case the in_axes argument can be set to `None`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([11., 20., 29.], dtype=float32)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import jax\n",
    "import jax.numpy as jnp\n",
    "\n",
    "x = jnp.arange(5)\n",
    "w = jnp.array([2., 3., 4.])\n",
    "\n",
    "def convolve(x, w):\n",
    "  output = []\n",
    "  for i in range(1, len(x)-1):\n",
    "    output.append(jnp.dot(x[i-1:i+2], w))\n",
    "  return jnp.array(output)\n",
    "\n",
    "convolve(x, w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Array([[11., 20., 29.],\n",
       "       [11., 20., 29.]], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs = jnp.stack([x, x])\n",
    "ws = jnp.stack([w, w])\n",
    "auto_batch_convolve = jax.vmap(convolve)\n",
    "print(\n",
    "auto_batch_convolve(xs, ws).shape)\n",
    "auto_batch_convolve(xs, ws)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 2) (3, 2)\n",
      "(3, 2)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Array([[11., 11.],\n",
       "       [20., 20.],\n",
       "       [29., 29.]], dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auto_batch_convolve_v2 = jax.vmap(convolve, in_axes=1, out_axes=1)\n",
    "\n",
    "xst = jnp.transpose(xs)\n",
    "wst = jnp.transpose(ws)\n",
    "\n",
    "print(xst.shape,wst.shape)\n",
    "print(\n",
    "auto_batch_convolve_v2(xst, wst).shape)\n",
    "\n",
    "auto_batch_convolve_v2(xst, wst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Array([[11., 20., 29.],\n",
       "       [11., 20., 29.]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_convolve_v3 = jax.vmap(convolve, in_axes=[0, None])\n",
    "print(\n",
    "batch_convolve_v3(xs, w).shape)\n",
    "batch_convolve_v3(xs, w)\n",
    "# Here w is not batched, the same w is used for all x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[11., 20., 29.],\n",
       "       [11., 20., 29.]], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jitted_batch_convolve = jax.jit(auto_batch_convolve)\n",
    "\n",
    "jitted_batch_convolve(xs, ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Automatic Differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-variable functions' derivatives: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{ \u001b[34m\u001b[22m\u001b[1mlambda \u001b[39m\u001b[22m\u001b[22m; a\u001b[35m:f32[]\u001b[39m. \u001b[34m\u001b[22m\u001b[1mlet\n",
      "    \u001b[39m\u001b[22m\u001b[22mb\u001b[35m:f32[]\u001b[39m = integer_pow[y=3] a\n",
      "    c\u001b[35m:f32[]\u001b[39m = integer_pow[y=2] a\n",
      "    d\u001b[35m:f32[]\u001b[39m = mul 2.0 c\n",
      "    e\u001b[35m:f32[]\u001b[39m = add b d\n",
      "    f\u001b[35m:f32[]\u001b[39m = mul 3.0 a\n",
      "    g\u001b[35m:f32[]\u001b[39m = sub e f\n",
      "    h\u001b[35m:f32[]\u001b[39m = add g 1.0\n",
      "  \u001b[34m\u001b[22m\u001b[1min \u001b[39m\u001b[22m\u001b[22m(h,) }\n",
      "{ \u001b[34m\u001b[22m\u001b[1mlambda \u001b[39m\u001b[22m\u001b[22m; a\u001b[35m:f32[]\u001b[39m. \u001b[34m\u001b[22m\u001b[1mlet\n",
      "    \u001b[39m\u001b[22m\u001b[22mb\u001b[35m:f32[]\u001b[39m = integer_pow[y=3] a\n",
      "    c\u001b[35m:f32[]\u001b[39m = integer_pow[y=2] a\n",
      "    d\u001b[35m:f32[]\u001b[39m = mul 3.0 c\n",
      "    e\u001b[35m:f32[]\u001b[39m = integer_pow[y=2] a\n",
      "    f\u001b[35m:f32[]\u001b[39m = integer_pow[y=1] a\n",
      "    g\u001b[35m:f32[]\u001b[39m = mul 2.0 f\n",
      "    h\u001b[35m:f32[]\u001b[39m = mul 2.0 e\n",
      "    i\u001b[35m:f32[]\u001b[39m = add b h\n",
      "    j\u001b[35m:f32[]\u001b[39m = mul 3.0 a\n",
      "    k\u001b[35m:f32[]\u001b[39m = sub i j\n",
      "    _\u001b[35m:f32[]\u001b[39m = add k 1.0\n",
      "    l\u001b[35m:f32[]\u001b[39m = neg 1.0\n",
      "    m\u001b[35m:f32[]\u001b[39m = mul 3.0 l\n",
      "    n\u001b[35m:f32[]\u001b[39m = mul 2.0 1.0\n",
      "    o\u001b[35m:f32[]\u001b[39m = mul n g\n",
      "    p\u001b[35m:f32[]\u001b[39m = add_any m o\n",
      "    q\u001b[35m:f32[]\u001b[39m = mul 1.0 d\n",
      "    r\u001b[35m:f32[]\u001b[39m = add_any p q\n",
      "  \u001b[34m\u001b[22m\u001b[1min \u001b[39m\u001b[22m\u001b[22m(r,) }\n"
     ]
    }
   ],
   "source": [
    "import jax\n",
    "\n",
    "f = lambda x: x**3 + 2*x**2 - 3*x + 1\n",
    "\n",
    "dfdx = jax.grad(f)\n",
    "print(jax.make_jaxpr(f)(1.))\n",
    "print(jax.make_jaxpr(dfdx)(1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0\n",
      "10.0\n",
      "6.0\n",
      "0.0\n"
     ]
    }
   ],
   "source": [
    "d2fdx = jax.grad(dfdx)\n",
    "d3fdx = jax.grad(d2fdx)\n",
    "d4fdx = jax.grad(d3fdx)\n",
    "print(dfdx(1.))\n",
    "print(d2fdx(1.))\n",
    "print(d3fdx(1.))\n",
    "print(d4fdx(1.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hessian matrix & Jacobian:\n",
    "\n",
    "Jacobian can be computed by `jax.jacfwd` and `jax.jacrev`: corresponding to forward- and reverse-mode autodiff.\n",
    "\n",
    "The choice depends on the relative sizes of the input and output spaces. If the input space is much larger than the output space, then reverse-mode autodiff is more efficient. If the output space is much larger than the input space, then forward-mode autodiff is more efficient.\n",
    "\n",
    "Some tricks can be used to avoid inefficiently computing full Jacobians."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[2., 0., 0.],\n",
       "       [0., 2., 0.],\n",
       "       [0., 0., 2.]], dtype=float32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def hessian(f):\n",
    "    return jax.jacfwd(jax.grad(f))\n",
    "\n",
    "import jax.numpy as jnp\n",
    "\n",
    "def f(x):\n",
    "    return jnp.dot(x, x)\n",
    "\n",
    "hessian(f)(jnp.array([1., 2., 3.]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Higher order optimization** is easy in JAX: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def meta_loss_fn(params, data):\n",
    "  \"\"\"Computes the loss after one step of SGD.\"\"\"\n",
    "  grads = jax.grad(loss_fn)(params, data)\n",
    "  return loss_fn(params - lr * grads, data)\n",
    "\n",
    "meta_grads = jax.grad(meta_loss_fn)(params, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stopping gradients\n",
    "\n",
    "Consider for instance the TD(0) (temporal difference) reinforcement learning update. Let’s assume the value estimate in a state is parameterised by a linear function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.2  2.4 -1.2]\n"
     ]
    }
   ],
   "source": [
    "# Value function and initial parameters\n",
    "value_fn = lambda theta, state: jnp.dot(theta, state)\n",
    "theta = jnp.array([0.1, -0.1, 0.])\n",
    "\n",
    "# An example transition.\n",
    "s_tm1 = jnp.array([1., 2., -1.])\n",
    "r_t = jnp.array(1.)\n",
    "s_t = jnp.array([2., 1., 0.])\n",
    "\n",
    "# Wrong way to compute the TD loss: the gradient include the dependency of target on theta.\n",
    "\n",
    "# def td_loss(theta, s_tm1, r_t, s_t):\n",
    "#   v_tm1 = value_fn(theta, s_tm1)\n",
    "#   target = r_t + value_fn(theta, s_t)\n",
    "#   return -0.5 * ((target - v_tm1) ** 2)\n",
    "\n",
    "# Right way: \n",
    "def td_loss(theta, s_tm1, r_t, s_t):\n",
    "    v_tm1 = value_fn(theta, s_tm1)\n",
    "    target = r_t + value_fn(theta, s_t)\n",
    "    return -0.5 * ((jax.lax.stop_gradient(target) - v_tm1) ** 2)\n",
    "\n",
    "\n",
    "td_update = jax.grad(td_loss)\n",
    "delta_theta = td_update(theta, s_tm1, r_t, s_t)\n",
    "\n",
    "print(delta_theta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([ 1.2,  2.4, -1.2], dtype=float32)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As a comparison, the original calculation of delta_theta is:\n",
    "s_grad = jax.grad(value_fn)(theta, s_tm1)\n",
    "delta_theta_original_calculation = (r_t + value_fn(theta, s_t) - value_fn(theta, s_tm1)) * s_grad\n",
    "\n",
    "delta_theta_original_calculation # [1.2, 2.4, -1.2], same as `delta_theta`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Array([ 1.,  2., -1.], dtype=float32), Array([2., 1., 0.], dtype=float32))"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "s_tm1 = jnp.array([1., 2., -1.])\n",
    "s_t = jnp.array([2., 1., 0.])\n",
    "\n",
    "f = lambda a,b: a.dot(b)\n",
    "jax.grad(f)(s_tm1, s_t)\n",
    "jax.grad(f,argnums=[0,1])(s_t, s_tm1)\n",
    "# Use argnums to specify which argument to differentiate with respect to\n",
    "# Default is argnums=0, means differentiate with respect to the first argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Straight-through estimator of gradient\n",
    "\n",
    "a trick for defining a gradient for a function that is not differentiable.\n",
    "\n",
    "Can be directly implemented using `jax.lax.stop_gradient`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f(x):  3.0\n",
      "straight_through_f(x): 3.0\n",
      "grad(f)(x): 0.0\n",
      "grad(straight_through_f)(x): 1.0\n"
     ]
    }
   ],
   "source": [
    "def f(x):\n",
    "    return jnp.round(x)  # non-differentiable\n",
    "\n",
    "def straight_through_f(x):\n",
    "    # Create an exactly-zero expression with Sterbenz lemma that has\n",
    "    # an exactly-one gradient.\n",
    "    zero = x - jax.lax.stop_gradient(x)\n",
    "    return zero + jax.lax.stop_gradient(f(x))\n",
    "\n",
    "print(\"f(x): \", f(3.2))\n",
    "print(\"straight_through_f(x):\", straight_through_f(3.2))\n",
    "\n",
    "print(\"grad(f)(x):\", jax.grad(f)(3.2))\n",
    "print(\"grad(straight_through_f)(x):\", jax.grad(straight_through_f)(3.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Per-example gradients\n",
    "\n",
    "Combine `jit`, `vmap`, `grad`, In JAX we can define it in an easy & efficient way.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[ 1.2,  2.4, -1.2],\n",
       "       [ 1.2,  2.4, -1.2]], dtype=float32)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perex_grads = jax.jit(jax.vmap(jax.grad(td_loss), in_axes=(None, 0, 0, 0)))\n",
    "                                            # theta, s_tm1, r_t, s_t\n",
    "                                            # theta is not batched, means\n",
    "                                            # We use the same theta for all samples\n",
    "                                            \n",
    "# Test it:\n",
    "batched_s_tm1 = jnp.stack([s_tm1, s_tm1])\n",
    "batched_r_t = jnp.stack([r_t, r_t])\n",
    "batched_s_t = jnp.stack([s_t, s_t])\n",
    "\n",
    "perex_grads(theta, batched_s_tm1, batched_r_t, batched_s_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.2  2.4 -1.2]\n",
      "[[ 1.2  2.4 -1.2]\n",
      " [ 1.2  2.4 -1.2]]\n",
      "[[ 1.2  2.4 -1.2]\n",
      " [ 1.2  2.4 -1.2]]\n",
      "9.39 ms ± 666 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n",
      "10.6 µs ± 1.32 µs per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "dtdloss_dtheta = jax.grad(td_loss)\n",
    "\n",
    "print(dtdloss_dtheta(theta, s_tm1, r_t, s_t))\n",
    "\n",
    "inefficient_perex_grads = jax.vmap(dtdloss_dtheta, in_axes=(None, 0, 0, 0))\n",
    "\n",
    "print(inefficient_perex_grads(theta, batched_s_tm1, batched_r_t, batched_s_t))\n",
    "\n",
    "perex_grads = jax.jit(inefficient_perex_grads)\n",
    "\n",
    "print(perex_grads(theta, batched_s_tm1, batched_r_t, batched_s_t))\n",
    "\n",
    "%timeit inefficient_perex_grads(theta, batched_s_tm1, batched_r_t, batched_s_t).block_until_ready()\n",
    "%timeit perex_grads(theta, batched_s_tm1, batched_r_t, batched_s_t).block_until_ready()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PRN in JAX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JAX’s random number generation differs from NumPy’s in important ways. The reason is that NumPy’s PRNG design makes it hard to simultaneously guarantee a number of desirable properties for JAX, specifically that code must be:\n",
    "\n",
    "* reproducible,\n",
    "\n",
    "* parallelizable,\n",
    "\n",
    "* vectorisable.\n",
    "\n",
    "Usage: generate a key, fork it, use part of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in NumPy, JAX’s random module also allows sampling of vectors of numbers. However, JAX does not provide a sequential equivalence guarantee, because doing so would interfere with the vectorization on SIMD hardware"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "individually: [-0.04838832  0.10796154 -1.2226542 ]\n",
      "all at once:  [ 0.18693547 -1.2806505  -1.5593132 ]\n"
     ]
    }
   ],
   "source": [
    "import jax \n",
    "from jax import random\n",
    "key = random.PRNGKey(42)\n",
    "subkeys = random.split(key, 3)\n",
    "sequence = np.stack([random.normal(subkey) for subkey in subkeys])\n",
    "print(\"individually:\", sequence)\n",
    "\n",
    "key = random.PRNGKey(42)\n",
    "print(\"all at once: \", random.normal(key, shape=(3,)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pytree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, we want to operate on objects that look like dicts of arrays, or lists of lists of dicts, or other nested structures. In JAX, we refer to these as **pytrees**, but you can sometimes see them called **nests**, or just trees.\n",
    "\n",
    "Related item: `jax.tree_utils`, `jax.tree_*`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Concepts**\n",
    "\n",
    "a pytree is a **container** of leaf elements and/or more **pytrees**. Containers include lists, tuples, and dicts. A leaf element is anything that’s not a pytree, e.g. an array. In other words, a pytree is just a possibly-nested standard or user-registered Python container. If nested, note that the container types do not need to match. A single “leaf”, i.e. a non-container object, is also considered a pytree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 'a', <object object at 0x000002CC54D3CBF0>] has 3 leaves: [1, 'a', <object object at 0x000002CC54D3CBF0>]\n",
      "(1, (2, 3), ())                               has 3 leaves: [1, 2, 3]\n",
      "[1, {'k1': 2, 'k2': (3, 4)}, 5]               has 5 leaves: [1, 2, 3, 4, 5]\n",
      "{'a': 2, 'b': (2, 3)}                         has 3 leaves: [2, 2, 3]\n",
      "Array([1, 2, 3], dtype=int32)                 has 1 leaves: [Array([1, 2, 3], dtype=int32)]\n"
     ]
    }
   ],
   "source": [
    "import jax\n",
    "import jax.numpy as jnp\n",
    "\n",
    "example_trees = [\n",
    "    [1, 'a', object()],\n",
    "    (1, (2, 3), ()),\n",
    "    [1, {'k1': 2, 'k2': (3, 4)}, 5],\n",
    "    {'a': 2, 'b': (2, 3)},\n",
    "    jnp.array([1, 2, 3]),\n",
    "]\n",
    "\n",
    "# Let's see how many leaves they have:\n",
    "for pytree in example_trees:\n",
    "    leaves = jax.tree_util.tree_leaves(pytree)\n",
    "    print(f\"{repr(pytree):<45} has {len(leaves)} leaves: {leaves}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Common operation: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[2, 4, 6], [2, 4], [2, 4, 6, 8]]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_of_lists = [\n",
    "    [1, 2, 3],\n",
    "    [1, 2],\n",
    "    [1, 2, 3, 4]\n",
    "]\n",
    "\n",
    "jax.tree_map(lambda x: x*2, list_of_lists)\n",
    "\n",
    "another_list_of_lists = list_of_lists\n",
    "jax.tree_map(lambda x, y: x+y, list_of_lists, another_list_of_lists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: MLP parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'biases': (128,), 'weights': (1, 128)},\n",
       " {'biases': (128,), 'weights': (128, 128)},\n",
       " {'biases': (1,), 'weights': (128, 1)}]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "def init_mlp_params(layer_widths):\n",
    "    params = []\n",
    "    for n_in, n_out in zip(layer_widths[:-1], layer_widths[1:]):\n",
    "        params.append(\n",
    "            dict(weights=np.random.normal(size=(n_in, n_out)) * np.sqrt(2/n_in),\n",
    "                biases=np.ones(shape=(n_out,))\n",
    "                )\n",
    "        )\n",
    "    return params\n",
    "\n",
    "params = init_mlp_params([1, 128, 128, 1])\n",
    "jax.tree_map(lambda x: x.shape, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward(params, x):\n",
    "    *hidden, last = params\n",
    "    for layer in hidden:\n",
    "        x = jax.nn.relu(x @ layer['weights'] + layer['biases'])\n",
    "    return x @ last['weights'] + last['biases']\n",
    "\n",
    "def loss_fn(params, x, y):\n",
    "    return jnp.mean((forward(params, x) - y) ** 2)\n",
    "\n",
    "LEARNING_RATE = 0.0001\n",
    "\n",
    "@jax.jit\n",
    "def update(params, x, y):\n",
    "\n",
    "    grads = jax.grad(loss_fn)(params, x, y)\n",
    "    # Note that `grads` is a pytree with the same structure as `params`.\n",
    "    # `jax.grad` is one of the many JAX functions that has\n",
    "    # built-in support for pytrees.\n",
    "\n",
    "    # This is handy, because we can apply the SGD update using tree utils:\n",
    "    return jax.tree_map(\n",
    "        lambda p, g: p - LEARNING_RATE * g, params, grads\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGdCAYAAABO2DpVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKYElEQVR4nO3de1xUZf4H8M+Z4aowKN6AJEXTFEkTU1PTrCxRw+xmW16yi5ummbmZ2k1Zt9R1t9zKsPyVVuaamzdsDStLy1tYrCmOlRe8pHhFGUC5zTm/P44zMjCXc+DM/fN+vXi5nHnOzBMLzIfn8n0ESZIkEBEREWlA5+0OEBERUeBgsCAiIiLNMFgQERGRZhgsiIiISDMMFkRERKQZBgsiIiLSDIMFERERaYbBgoiIiDQT4ukXFEURJ0+eRHR0NARB8PTLExERUR1IkoTi4mIkJCRAp3M8LuHxYHHy5EkkJiZ6+mWJiIhIA8ePH0fLli0dPu7xYBEdHQ1A7pjBYPD0yxMREVEdmEwmJCYmWt/HHfF4sLBMfxgMBgYLIiIiP+NqGQMXbxIREZFmGCyIiIhIMwwWREREpBkGCyIiItIMgwURERFphsGCiIiINKM6WJw4cQIjR45EkyZNEBkZiRtuuAE//fSTO/pGREREfkZVHYsLFy6gT58+uO222/Dll1+iWbNmOHDgABo3buyu/hEREZEfURUs5s2bh8TERCxZssR6LSkpSfNOqWUWJeTkF+JMcRmaR0egR1Is9DqeQ0JERORpqoJFVlYWBg4ciAcffBBbtmzBNddcg6effhpjx451eE95eTnKy8utn5tMprr31o7svAJkrDeioKjMei0+JgIz05ORlhKv6WsRERGRc6rWWBw+fBiZmZlo164dNm7ciPHjx2PSpEn46KOPHN4zZ84cxMTEWD+0PIAsO68A45fl2oQKADhVVIbxy3KRnVeg2WsRERGRa4IkSZLSxmFhYbjpppuwfft267VJkyZh165d2LFjh9177I1YJCYmoqioqF5nhZhFCbfM+7ZWqLAQAMTFRGDrtNs5LUJERFRPJpMJMTExLt+/VY1YxMfHIzk52eZax44dcezYMYf3hIeHWw8c0/LgsZz8QoehAgAkAAVFZcjJL9Tk9YiIiMg1VcGiT58++O2332yu/f7772jVqpWmnVLiTLHjUFGXdkRERFR/qoLFc889h507d+L111/HwYMHsXz5crz//vuYMGGCu/rnUPPoCE3bERERUf2pChbdu3fHmjVr8O9//xspKSmYPXs2FixYgBEjRrirfw71SIpFfEwEHK2eECDvDumRFOvJbhEREXmFWZSw49B5rNt9AjsOnYdZVLyEUlOqFm9qQeniDyUsu0IAeU2FhSVsZI5M5ZZTIiIKeJ4oveCWxZu+Ji0lHpkjUxEXYzvdERcTwVBBRERBwddKL/j1iIVFzcqb3Vo1xs9HL7ASJxERBTRPll5Q+v6tqvKmr9LrBPRq2wSAnNxunf8dK3ESEVHAU1N6wfI+6W5+PRVSk68NBxEREbmTL5ZeCJhgYRYlZKw3wt68juVaxnqj11bJEhERac0XSy8ETLBgJU4iIgo2vlh6IWCChS8OBxEREbmTXidgZrp81EbNcGH5fGZ6skc3MARMsPDF4SAiIiJ387XSCwGxKwS4Ohx0qqjM7joLy5YbVuIkIqJAk5YSjzuT42xKL3ir1ELABAvLcND4ZbkQYL8Sp6eHg4iIiDyleukFbwqYqRDA94aDiIiIgk3AjFhY+NJwEBERUbAJuGAB+M5wEBERUbAJqKkQIiIi8i4GCyIiItIMgwURERFphsGCiIiINMNgQURERJphsCAiIiLNMFgQERGRZhgsiIiISDMMFkRERKQZBgsiIiLSDIMFERERaYbBgoiIiDTDYEFERESaYbAgIiIizTBYEBERkWYYLIiIiEgzDBZERESkGQYLIiIi0gyDBREREWmGwYKIiIg0w2BBREREmmGwICIiIs0wWBAREZFmGCyIiIhIMwwWREREpBkGCyIiItIMgwURERFphsGCiIiINMNgQURERJphsCAiIiLNqAoWs2bNgiAINh8dOnRwV9+IiIjIz4SovaFTp0745ptvrj5BiOqnICIiogClOhWEhIQgLi7OHX0hIiIiP6d6jcWBAweQkJCANm3aYMSIETh27Jg7+kVERER+SNWIRc+ePbF06VJcf/31KCgoQEZGBvr27Yu8vDxER0fbvae8vBzl5eXWz00mU/16TERERD5LkCRJquvNFy9eRKtWrfDGG2/giSeesNtm1qxZyMjIqHW9qKgIBoOhri9NREREHmQymRATE+Py/bte200bNWqE9u3b4+DBgw7bzJgxA0VFRdaP48eP1+cliYiIyIfVK1iUlJTg0KFDiI+Pd9gmPDwcBoPB5oOIiIgCk6pg8fzzz2PLli04cuQItm/fjnvvvRd6vR4PP/ywu/pHREREfkTV4s0//vgDDz/8MM6fP49mzZrhlltuwc6dO9GsWTN39Y+IiIj8iKpgsWLFCnf1g4iIiAIAzwohIiIizTBYEBERkWYYLIiIiEgzDBZERESkGQYLIiIi0gyDBREREWmGwYKIiIg0w2BBREREmmGwICIiIs0wWBAREZFmGCyIiIhIMwwWREREpBkGCyIiItIMgwURERFphsGCiIiINMNgQURERJphsCAiIiLNhHi7A5oQzcDR7UDJaSCqBdCqN6DTe7tXREQU5MyihJz8QpwpLkPz6Aj0SIqFXid4u1tu5f/BwpgFZE8DTCevXjMkAGnzgOSh3usXEREFtey8AmSsN6KgqMx6LT4mAjPTk5GWEu/FnrmXf0+FGLOAlaNtQwUAmArk68Ys7/SLiIiCWnZeAcYvy7UJFQBwqqgM45flIjuvwEs9cz//DRaiWR6pgGTnwSvXsqfL7YiIiDzELErIWG909u6EjPVGmEV7Lfyf/waLo9trj1TYkADTCeDHRQwXRETkMTn5hbVGKqqTABQUlSEnv9BznfIg/w0WJaeVtdv4IrAghdMiRETkEWeKHYeKurTzN/4bLKJaKG/LNRdEROQhzaMjNG3nb/w3WLTqLe/+gJJtO1xzQUREntEjKRbxMREO350EyLtDeiTFerJbHuO/wUKnl7eUAlAcLkwn5LUZREREbqLXCZiZngyg9ruT5fOZ6ckBW8/Cf4MFINepGP4xYFCxH1jp2gwiIqI6SkuJR+bIVMTF2E53xMVEIHNkakDXsfD/AlnJQ4EOQ+TdHxtfdN1ezdoMIiKiOkpLicedyXGsvOmXdHqg5zhgxzvyQk27u4cFeU1Gq96e7h0REQUpvU5Ar7ZNvN0Nj/LvqZDqnK65uPJ52lyeIUJERORGgRMsAMdrLgwJ8nWeHUJERORWgTEVUp1lzQVPOyUiIvK4wAsWgBwikvp6uxdERERBJ7CmQoiIiMirGCyIiIhIMwwWREREpBkGCyIiItIMgwURERFphsGCiIiINMNgQURERJphsCAiIiLNMFgQERGRZhgsiIiISDMMFkRERKSZegWLuXPnQhAETJ48WaPuEBERkT+rc7DYtWsX3nvvPXTu3FnL/hAREZEfq1OwKCkpwYgRI7B48WI0btxY6z4RERGRn6pTsJgwYQKGDBmCAQMGuGxbXl4Ok8lk80FERESBKUTtDStWrEBubi527dqlqP2cOXOQkZGhumNERETkf1SNWBw/fhzPPvssPv30U0RERCi6Z8aMGSgqKrJ+HD9+vE4dJSIiIt8nSJIkKW28du1a3HvvvdDr9dZrZrMZgiBAp9OhvLzc5jF7TCYTYmJiUFRUBIPBUPeeExERkccoff9WNRVyxx13YO/evTbXHnvsMXTo0AHTpk1zGSqIiIjITUQzcHQ7UHIaiGoBtOoN6Dz/vqwqWERHRyMlJcXmWsOGDdGkSZNa14mIiMhDjFlA9jTAdPLqNUMCkDYPSB7q0a6w8iYREZE/M2YBK0fbhgoAMBXI141ZHu2O6l0hNW3evFmDbhAREZFqolkeqYC95ZISAAHIng50GOKxaRGOWBAREfmro9trj1TYkADTCbmdhzBYEBER+auS09q20wCDBRERkb+KaqFtOw0wWBAREfmrVr3l3R8QHDQQAMM1cjsPYbAgIiLyVzq9vKUUQO1wceXztLkerWfBYOEDzKKEHYfOY93uE9hx6DzMouJiqEREFOyShwLDPwYM8bbXDQnydQ/Xsaj3dlOqn+y8AmSsN6KgqMx6LT4mAjPTk5GWEu/kTiIioiuSh8pbSn2g8qaqs0K0wLNCrsrOK8D4Zbm1dh9bBrMyR6YyXBARkU9Q+v7NqRAvMYsSMtYbHZY0AYCM9UZOixARkV9hsPCSnPxCm+mPmiQABUVlyMkv9FyniIiI6onBwkvOFDsOFXVpR0RE5AsYLLykeXSEpu2IiIh8AYOFl/RIikV8TITDkiYA0KRhGLq1auyxPhEREdUXg4WX6HUCZqYnA3BcL+18aQVunf8dsvMKPNcxIiKiemCw8KK0lHhkjkxFXIzj6Y5TRWUYvyyX4YKIiPwCg4WXpaXEY8vU2xDbMNTu49x6SkRE/oTBwgf8fPQCCksrHT7OradEROQvGCx8ALeeEhFRoGCw8AHcekpERIGCh5D5gB6tYjA4+iBCSs/gDBohR+wAsVrmEwDExUSgR1Ks9zpJRESkAIOFtxmzoM+ehncrTwJh8qWTUiwyKkdjo9jDuhV1Znoy9DpnVS+IiMjniGafOHHUkxgsvMmYBawcDdQ4iiwOhcgMXYDxlZOxJ7ofj1AnIvJHxiwgexpgOnn1miEBSJsnH3MeoHhsureIZmBBiu03XDUSBFQ0iEPIlDzoQ5j/iIj8ioM/HK0lEYd/7Hfhgsem+7qj2x2GCgAQICH8UgH0x7Z5sFNERFRvolkeqagVKnD1WvZ0uV0AYrDwlpLTytr9Z7ScfImIyD+4+MMRkADTCbldAGKw8JaoFsraXb4oD6cxXBAR+QelfzgqbednGCy8pVVveRGP0/NNqwngYTMiooCi9A9Hpe38DIOFt+j08spgRQJ72IyIKKC4/MNRAAzXyO0CEIOFNyUPlVcGRzZW1j5Ah82IiAKKzR+ONcPFlc/T5gZsPQsGC29LHgo8sFRZ2wAdNiMiCjiWPxwNNWoQGRL8cqupGiyQ4AuS+srfbKYC2N+eJMiPB+iwGRFRQEoeCnQYwsqb5AWWYbOVoyEPk1UPF4E/bEZEFLB0evmPxyDCqRBfEcTDZkREFDg4YuFLgnTYjIiIAgeDha8JwmEzIiIKHJwKISIiIs0wWBAREZFmOBXi58yihJz8QpwpLkPz6Aj0SIqFXqewTDgREZHGGCz8WHZeAWZn7UViyS9ojos4g0Y4HtUFrwy9AWkp8a6fgIiISGMMFn4qO68Aa5cvwn9CP0ZCWKH1+snyWPx1+WjgkXEMF0RE5HFcY+GHzKKEzWs/xLuhCxCHQpvH4lCId0MXYPPaD2EW7VXxJCIich8GCz+Uc+gsJlX+HwCg5nIKy+eTKj9AzqGzHu4ZEREFOwYLP2Q+sg0JQmGtUGGhE4AE4TzMR7Z5tmNERF5mFiXsOHQe63afwI5D5zly6wWq1lhkZmYiMzMTR44cAQB06tQJr776KgYNGuSOvpEDzYWLmrYjIgoEG/YU4OV1eSgsrbBei4+JwMz0ZK458yBVIxYtW7bE3Llz8fPPP+Onn37C7bffjnvuuQf79u1zV//IjrZt2mrajojI383ZYMTTy3NtQgUAFBSVYfyyXGTnFXipZ8FHVbBIT0/H4MGD0a5dO7Rv3x6vvfYaoqKisHPnTnf1j+zQt+6Dy5FxcDTCJ0rA5cg46Fv38WzHiIi8YMOek3jv+3yHj0sAMtYbOS3iIXVeY2E2m7FixQqUlpaiV69eWvaJXNHpEZk+H4IgQKzxkAhAEAREps/n4WVEFPDMooSX1+W5bFdQVIac/EKX7aj+VNex2Lt3L3r16oWysjJERUVhzZo1SE5Odti+vLwc5eXl1s9NJlPdekq2kodCGP4xkD0NMJ20XhYM10BIm8tj1okoKOTkF6KwtFJR2zPFZW7uDQF1CBbXX389du/ejaKiInz++ed49NFHsWXLFofhYs6cOcjIyKh3R8mO5KEQahyzLvCYdSIKImrCQvPoCDf2hCwESZLqNek0YMAAtG3bFu+9957dx+2NWCQmJqKoqAgGg6E+L01EREFux6HzeHix63V+TRqGIeelATxLqR5MJhNiYmJcvn/Xu6S3KIo2waGm8PBwhIeH1/dliIiIaumRFIv4mAgUFDkfuZh9TwpDhYeoChYzZszAoEGDcO2116K4uBjLly/H5s2bsXHjRnf1j4iIyC7L6c6DU+LwwbYjDts91S8JgzuzjoWnqAoWZ86cwejRo1FQUICYmBh07twZGzduxJ133umu/hEREdWSnVeAjPVGm5EKnQCbbfixDUPxt3tSMLhzghd6GLxUBYsPPvjAXf0gIiJSJDuvAOOX5aLmAkHLisHH+7TGnclx6JEUy+kPL+BZIURE5DfMooSM9cZaoQKQC2HpIeLUnm/Qs/Rb6I9uBUSzp7sY9Oq9eJOIiMhTcvILHS7UHKT7EX8L/RBNKouB1VcuGhKAtHms7eNBHLEgIiK/8bXxlN3r0/XL8W7ov9BEKLZ9wHQSWDkaMGZ5oHcEMFgEN9EM5P8A7P1c/pdDhkTkw8yihJU//VHreppuJ54K+cLJnRKQPZ2/4zyEUyHByphVqxw4hwyJyJe98+0BlJRX2VzTQcTfQpdAcLVG03RCrlKc1Nd9HSQAHLEITsYseWiweqgAAFMBhwyJyCeZRQlLatSq0EHEGH02mtac/nCk5LT2HaNaOGIRbESzPFLhcE21IA8ZdhjCM0eIyGfk5Bfi4uWrh40N1OVgZujHSBBUnFga1cINPaOaOGIRbI5urz1SYUO6OmRIROQjqh82NlCXg8zQBYiDilDRoCnQqrcbekY1ccQi2CgdCuSQIRH5EMvJpDqImBO6GALgel1FdUP+yVFYD+GIRbBROhTIIUMi8iGWw8beDFmIWKFUXajoPQnoNMxdXaMaGCyCTave8u4POPqpFADDNRwyJCKfotcJeLfrMQzV71B+U4OmwAMfAXfNdl/HqBYGi2Cj08tbSgHUDhdXPk+byyFDIvItohld98xWPlIx8HXg+d+BlGHu7BXZwWARjJKHAsM/hmSwPUZYMiQAwz9mHQsi8j1HtwOXzitrG9kY6DmOfyB5CRdvBqlssTtml/0LiRW/oDku4gwa4XhZF7wi3oA0b3eOiKgmNQvKe45nqPAiBosgVP3I4RNItl4XTJUYvywXmSNTkZYS7/gJiIg8TemC8nAD0O959/aFnOJUSJBxdeQwAGSsN8Is2mtBROQl1oXnLgx9m6MVXsZgEWScHTkMyOGioKgMOfkqCs8QEWnJ3gGJ1oXnTlZvclupT+BUSJCpXr1Oi3ZERJoyZkHKngahWoVgyZAAwXJA4vCPax+g2KApMPif3AHiIxgsgoylep1W7YiINGPMgrRyNCRINuMSkukksHI0BMuutQ5D5F0iJafltRetenP6w4cwWAQZS/W6U0VldtdZCADiYiLQIym29oOimT/MROQeohmX109FuCRBV2O2QwdAlCSUrZ+KSMsBiTz+3GcxWAQZvU7AzPRkjF+WCwG2Z5xafpZnpidDX/Mn25hVe/jRkCDPebLuBRHVk/nINkRePuVwCYVOACIvn4L5yDbo2/TzbOdIFS7eDEJpKfHIHJmKuBjb6Y64mAj7W02NWcDK0bVPRTUVyNeNWW7uMREFukOHD2najryHIxZBKi0lHncmxyEnvxBnisvQPFqe/qg1UiGa5ZEKhxtUBSB7ujznyWkRIqqjM1IjtNewHXkPg0UQ0+sE9GrbxHmjo9trj1TYkADTCbkd5zyJqI70rfvg5NZYxKGw1hoLABAl4BSaQN+6j+c7R6pwKoScU1pGV025XSKiGnq0bYa3Qp8EIIeI6iyfvxX6BHq0bebhnpFaDBbknNIyukrbERHZodcJ6D/scTxdORmnYLsr7RSa4OnKyeg/7PHa07XkczgVQs5ZyuiaCmB/nYUgP96qt6d7RkQBJi0lHnhkHB7M6oPEkmoHJEZ1wSsP3sAzjPwEgwU5Zymju3I04GiDatpcLtwkIk1cXVjezfnCcvJZDBbkmqMyuoYEOVSwjgURaUjRwnLyWQwWpAzL6BIRkQIMFqQcy+gSEZEL3BVCREREmmGwICIiIs0wWBAREZFmuMaCFDOLkuuzRYiIKKgxWJAi2XkFyFhvREFRmfVafEwEZqYns2gNERFZcSqEXMrOK8D4Zbk2oQIAThWVYfyyXGTnFXipZ0RE5GsYLMgpsyghY73R4aHpAJCx3ghzzVODiIgoKDFYkFM5+YW1RiqqkwAUFJUhJ7/Qc50iIiKfxTUW5NSZYsehoi7tiCjAiGZW5CUbDBbkVPPoCE3bEVEAMWY5OENoHs8QCmKcCiGneiTFIj4mAo42lQqQd4f0SIr1ZLeIyNuMWfKpx9VDBQCYCuTrxizv9Iu8jsGCnNLrBMxMTwaAWuHC8vnM9GTl9SxEM5D/A7D3c/lf0axZX4nIQ0SzPFLhbFl39nT+fAcpToWQS2kp8cgcmVqrjkWc2joWHDYlCgxHt9ceqbAhAaYTcjseXBh0VAWLOXPmYPXq1fj1118RGRmJ3r17Y968ebj++uvd1T/yEWkp8bgzOa7ulTctw6Y1/8KxDJsO/5jhgsjXWRZq7lc4zVFy2r39IZ+kKlhs2bIFEyZMQPfu3VFVVYUXX3wRd911F4xGIxo2bOiuPpKP0OsE9GrbRP2NLodNBXnYtMMQriYn8lXGLODLF4BiFQXxolq4rz/ks1QFi+zsbJvPly5diubNm+Pnn39Gv379NO0YBRAOmxL5N2MWsHKUihsEeZqzVW+3dYl8V73WWBQVFQEAYmMd7wgoLy9HeXm59XOTyVSflyR/pHQ4lMOmRL5HNAPrJ6m44cr0aNpcjkAGqTrvChFFEZMnT0afPn2QkpLisN2cOXMQExNj/UhMTKzrS5K/UjocWnKaq8iJfE3+D8DlC8rbGxK4ZirICZIk1emQh/Hjx+PLL7/E1q1b0bJlS4ft7I1YJCYmoqioCAaDoS4vTf5GNAMLUuSFmnbXWVTDXSJEvuWz0cD+da7bdbgb6DmOlTcDmMlkQkxMjMv37zqNWEycOBFffPEFvvvuO6ehAgDCw8NhMBhsPijI6PRyWABQuxpGDaaT8lxu3lp394qIXDFmKQsVANCso7xGiqEi6KkKFpIkYeLEiVizZg2+/fZbJCUluatfFGiSh8rDowaFNS9WPQbsW+vWLhGRE9bdXApx4TVdoSpYTJgwAcuWLcPy5csRHR2NU6dO4dSpU7h8+bK7+keBJHkoMDkPGPi667aSCPznUZYFJvIWl7u5ZBKAyvDGWFeUhB2HzsMs1ml2nQKIql0hmZmZAID+/fvbXF+yZAnGjBmjVZ8okOn06va2s74FkXeo2KU1sfhRbPxsLwD57CBVFXkp4KieCrH3wVBBqqgJFpb6FkTkWQp/Tt+ofAAbxR7Wz08VlWH8slxk56kopEUBhYeQkee16i3v/lCK9S2IPM/6c2p/wbUI4KQUi4XmYTbXLRMhGeuNnBYJUgwW5Hk2u0QUYFlgIs9zsptLggBIQEblaIh23kYkAAVFZcjJL3R/P8nnMFiQdyQPBR74CBCcfQsKgOEalgUm8hYHu7kuR7bA+MrJNlMg9pwpLnP6OAUmHptO3pMyTP5D6D+P2nlQACABqY8C+9bIoxYsvEOkPcuJpSWn7f+cJQ+VF1BXa/NL1fXY+MEul0/dPDrCjR0nX8VgQd7VaRggfCLvl6++tS2ysfzv5mpbU1mVk0hb+9YC//0LcOnc1Wv2fs50eps6FT1ECfExEThVVGa3lq4AIC4mAj2SHJ8jRYGrziW960ppSVAKMtX/ajp/CNg8B7XLf1+Z5+U5BET199UrwPa3HDwo2P05M4sScvILcaa4DEfOXcKCb34HYPuTalmNkTkylVtOA4zS92+OWJBHVf/F1Dxa/otGrxOu/kVkOVfE7t9BEgCBtS2I6itvrZNQAQBSrZ+z7LwCzMrah1Omq2c/xUSGQBAEXLxUab0WxzoWQY/BgjwmO68AGeuNKCi6uqCrVjEdl9X+pKu1LVhCmEi9qgpg/TOu21X7OcvOK8C4Zbm1mhRdrgIAPDegHVo3bWj7xwIFLe4KIY/IzivA+GW5NqECsFNMR2nNCta2IFLPmAW80REoNylrX3IaZlHC9NV7nTZbsv0I7u6cgF5tmzBUEIMFuZ9ZlJCx3uhwcgOoVkxHac0K1rYgUseYBawcbbtQ05WoFth56LzNVIc9Fy9VYueh8/XsIAUKBgtyu5z8wlojFdXZFNNxUe2PtS2I6kA0A19Og/21Sw40aAq06o1PfzyiqPmOwyoCCwU0BgtyO6VFcs4Ulzmt9mf9PG0uF24SqbHqSaDY9UmlNob8E9nGM9iQp3TakVMgJGOwILdTWiTH2s5BtT8YErjVlEitr14B9q1Wd0/vSTB3vAcZ642Kb+nVtonKjlGg4q4QcrseSbFOi+kAQKMGoRBFCWZRkhd/2an2x8qbRCpVVQA73lHePtwApL8NpAxDzqHzTqcwq2vcIBQ3t2GwIBmDBbmdXidgZnoyxi/LtRTqruXipUqM+OBH2+2nNar91eKqFDFRsFs/GZBEZW0bNAWm7AdCwgCoO+djzn03cDcIWXEqhDwiLSUemSNTERfjfFqkoKgM45blYsMeF/PBxiy5kNZHdwOrnpD/XZAiXyci+Wfhl0+Vt7/7TWuoAJRPYT43oD2LYZENBgvymLSUeGyddjs+fbInGkWGOm078d//w4Y9BfYftGybq1lIy1QgX2e4oGAnmuXzd5S68ZFaa5csU5jOxiHiDOGYePt1desjBSwGC/IovU6AThBw8bLzffGiBDy9vFrhLOsDll+YjqpiXClFLJq16jKRfxHNwM5MFxVsq9MBd/+r1lXLFCZgf3+WAGDW0E6cAqFaGCzI49TM3VoLZ1m4LPkNuRTxlr/XsXdEfsyYBcy/DvjqJeX39J5oMwVSnaMpzLiYCB4yRg5x8SZ5nNK5W+Bq4SzrVjalpby3zAUqLwF3za5DD4n8kDELWDlK3T2d7nP5M5KWEo87k+PsHx5IZAeDBXmcZe5W6VY2mxEONaW8t78FJHQDUoap6yCRv7FW1lQhOgG4//8UNdXrBNapIMU4FUIeV33uVgmbEQ5ryW+F/juF6y0o8H3/D5WVNQVg0Dxuzya3YLAgr0hLice7j3SFs9FUAfKx6j2SYq9etCn5rcDl8/IvXaJAZcwCNr+uvH1krE0FW7MoYceh81i3+wR2HDpvu6aJqA4ESZI8+l1kMpkQExODoqIiGAwGT740+aANewrw9PLcWtcthbSeG9AOrZs2rD2vu3meul+mwz9hKXAKPKIZmN8WuHxB+T2j1gFt+wOQf/5eXpeHwtIK68M2ReqIqlH6/s1gQV6XnVeAjPVGmzUXjRrIdS6qH9ds8wtPNAPz28kjEkoYrgEm7+XQLwWW7+bIC5WVik4AnssDdHrM2WDEe9/n220mANz1QbUoff/mVAh5naVw1r/H3ox//elGPDegPYouVdqECgA4VVSG8cuu1LbQ6YG731D+IqYT8lZVokCxby2wRcW0IGBdV7Fhz0mHoQKQRwtrbfUmUojBgnyCZdX53Z0TsGLXMYflr4Bqv/A6DQN6T1L+Ikq3qhL5OmMW8J9HYb9QnB1h0dbpwIoqEdNW73F5i2WrN5FaDBbkU3LyC51uQ5VQ4xfeXbOBftOVPbmarapEvko0A+tVBOrQKOCFw0DyUGTnFeDmOZtQXKZsp5SaYnZEFgwW5FOU/iKzadf/BRdbUAV5jUWr3vXrHJEvWPWkusWafSYBIWHIzivA+GW5Ngs1XVFTzI7IgsGCfIrSX2Q27axbUC0nGFR35fO0uVy4Sf4vby2wb7Xy9pGxQL/nYRYlzMoyKp04AQA0aRhmu9WbSCEGC/Iprk5UtFvbApC3kg7/GDDUWMVuSLDZs0/kt0QzsGGKunvS/wXo9Ji8IhenTOqmNWbfk8Ky3VQnLOlNPsVSlXP8slxrLQsLy6+4menJ9n/hJQ8FOgyRd3+UnJbXVLTqbTtSIZqdP07kq45uBy4p3F4NAXhgKZA8FHM2GLF+zylVL/VUvyQM7sytplQ3DBbkcywnKtasbRGnpHCPTg8k9bX/mDFLPnK9+umohgR5GoUjGuTr1OxqunUakDIMFVUiFv/geFtpTdEResy7rzMGd1ZRNp+oBgYL8klKTlQ0i5LyExeNWcDK0ai1Pc9UIF/ndAn5OqW7msKigVtfAAB8suMIlJaiaNIwDDtm3IGwEM6QU/0wWJDPcnaior1qnQ5LEYtmeaTCYXUMAcieLk+jcFqEfJXlAD6Ti8PG7nnH+n38/YGzip/+tXtTGCpIE/wuIr9j2TZXs96FTWXO6o5ud/HLWGJlTvJ9NrufHOg9SS4cB/nnZMvv5xQ99QOp17B8N2mGwYL8ilmUkLHe/rY56crHi2vyUFElXn1A6dw0K3OSr7PufqqxBqJBU+CBj+SCcbj6c6KEAOD1+zpr3FEKZpwKIb/iqjInABSWVqDr7K/wzwe7yH+FKZ2bZmVO8gcudj+ZRQkfbs13+XNi8ed+SZwCIU0xWJBfUVqZs7TcjHHLcrFoZCrSki1z0wWwv85CkB9nZU7yFw52P2XnFWD66r21DvBz5I4OzTBjcLLWvaMgx5hKfkVtieGZ6/Jghu7K3DTgtDKnaAZ2LAQ2TJX/rVJe+pjI2zbsKcC4ZbmKQwUAPNm3rRt7RMGKwYL8iqUyp1KniyvwzrcHXVfm/GMX8FoLYOOLQM778r+vtQC+ekXj/wIi7W3YcxIT/52ruL3DCrZEGuBUCPkVS2XOccuU/xJ985vfcX1cFNJSHMxNfzML2P5W7Rsl8er1K4viiHxNdl4Bnl7+P9X3OaxgS1RPHLEgv5OWEo/nBrRXdU/GeiPMonR1bvqGB+R/RTOw4x3nN3NahHyUWZQwffVeVfc0ahCKzJGp3F5KbqM6WHz//fdIT09HQkICBEHA2rVr3dAtIucm3n4dWkSHK25fUFSGnPzC2g/sWiyPTDgjmeV2RD7mrU0HVK2pAICFDzNUkHupDhalpaXo0qULFi5c6I7+ECmi1wnIuKeTqnvs7ii5cETZzQe/AfJ/kEc4iHzAhj0FeGvTAVX3xBnCcbODarZEWlG9xmLQoEEYNGiQO/pCpEpaSjwWjUzFX/7zC0rLXb/h291R0ri1shc79K380aAJMPgNIGWYqr4SaUleV6F8nZHFrKGduK6C3M7tayzKy8thMplsPoi0kpYSj/+9chdiG4Y5bON0BXz3sYCg4sfg0nng80eB7BfVd5ZIA3VaVxEZItd04RQIeYDbg8WcOXMQExNj/UhMTHT3S1KQCQvR4fV7UyDAYZUKxyvgQ8KAXhPVv+jOhcCnD6m/jwiQp9TyfwD2fq56im3yCnW1Ku5PvQY/v3IXQwV5jCBJksJDde3cLAhYs2YNhg0b5rBNeXk5ysvLrZ+bTCYkJiaiqKgIBoOhri9NVIuSE08dHrX+1Svy7hBXCzlraj8IeGSFlv8ZFOiMWfJpu9UPxjMkyEXckoc6vXXDnpOqtpY2bhCKn16+k9MfpAmTyYSYmBiX799ur2MRHh6O8HDlq/eJ6iotJR53JsfZDw5wETzumg3c/oq8++PgN/J6CiV+/xLIWw2k3OeO/yQKNMYsYOVo1CotbyqQrw//2GG4MIsSXl6Xp+rl5tx3A0MFeRzrWFBA0esE9GrbBPfceA16tW1iEypcHrUeEgb0mgDcMkXdi2ZNZJ0Lck00yyMVDs/mBZA93eG0SE5+IQpLlU2BCADefaQrpz/IK1QHi5KSEuzevRu7d+8GAOTn52P37t04duyY1n0j0oSro9aBagW0ALkaZwMVW/IqSoE3Osh/jRI5cnS77fRHLRJgOiG3s0PpAXwA8Owd7TC4c4LrhkRuoDpY/PTTT+jatSu6du0KAJgyZQq6du2KV199VfPOEWnB1VHrEmoU0NLp5S2lalw6Lw9lM1yQPaIZOLxFWduS03YvKz2ALyo8BM/c0U5pz4g0p3qNRf/+/VGP9Z5EHqf0Lz2bdinDgD8myLs/1MieLp9HotOru48Cl73Fms5EtbB72XIAn7OQDAB/v78z11WQV3GNBQU8pX/p1WqX9jrQLk3FKzkfyqYgZMwCVo5SGCoEwHCNPBVnh+UAPmeR4al+SRjcmesqyLsYLCjgWf7Sc/QL2WkBrRGfyVtK1Ti8haW/Sf4eWD9JYeMr351pc52OdqWlxCNzZCriY2xDcGzDULz7SFfMGJxcx84SaadedSzqQuk+WCItWXaFALZr8i1hI3NkKm7v0AKf7DiCo4WX0Cq2AUb1ao2wkCvZO281sG4iUFmq7AUV1iWgALZ5HrD5dWVtDdfIoULh94vDeixEbqT0/ZvBgoKGszoW/zt2AYt/yIdY7adBJwBj+yZd/SuwqkLe/XHpvIJXu/JL3kldAgpgohmY3xa4fMF1235Tgf4zuC6HfJ7PFMgi8hWOCmj9PXs/3vs+v1Z7UYL1+ozByXKdi7sXXClwBNivRwDbx9Y+DYRFA2368Y0jmBzZqixUAEDSrfzeoIDCNRYUVGoW0DKLEhb/UDtUVLf4h3xUVF0p9Z08VB6FMChcIFdRDCwbBsy/jltRg0XeWmDFI8raRjZ2uFiTyF8xWFBQ+2THEZvpD3tESW5nlTwUmJwnD2ErdblQ3h3AcBHYvnpFPv22okRZ+57jOVpBAYfBgoLa0cJLitpt/v0sdhw6f7U6p04vD2Gr5aRkM/m5fWuB7W8pbx8ZC/R73m3dIfIWrrGgoNYqtoGidj8cOIcfDpxDwzA9nuybhEl3tIe+VW9594epAM7XW1RjqXOR1LfunSbfI5qBNeMUN5cg4Pcef8Ove05xVwcFHI5YUFAb1as11Pw+L60w41+bDuKGWRuRbTwjbylVy0HJZvJjq54Eqi4raloR1ggzQqZi4MZGeHbFbjy8eCdumfetfBAeUQBgsKCgFhaiw9i+Sarvu1RhxrhlucgWu8uLOdUcWuagZDP5IdEMHPwW2Lda8S2PljyNFSU32lyzOWWXyM8xWFDQmzE4GU/1S1I1cmExZeVumDukA1N+BSIVhIsGTYHiAiD/B6618HfGLGBBCrDsXsW3nIcBP4q1q2PaPWWXyE8xWBBBDhe/zh6EV4Z0RN92TRXfd6lCxAuf/wKzLhRIX6DghnPA6rHAR3fLb0rcJeKfjFlyPROlB4sBkCTgpYoxEB382q11yi6Rn2KwILoiLESHJ/q2wdP9r1N136rcE+gzd9OVaZFP5NoESpgKeNS6P6qqAL54DooX7F6xTUxGtnizy3ZKT+Ml8lUMFkQ19EiKRWzDUFX3nDKVy3PkYndg6iFg1Dq5zsUtf3Gy/uLKGxO3oPoPY9aVsu7nFN8iSYBZAsZUTlfUXulpvES+isGCqAa9TsDf7kmp070Z640wQwe07Q/c/jLQ9jYXZ4vwqHW/YZn+UHRWjMxyEtPiqrtR5WJ3v9NTdon8CIMFkR2DOyeo3i1imSNfui3/6gI8pVtLuQXVt4lmIHsa1E5/mKHDe1V3Y67ZeYlvy7rhmenJrGdBfo/BgsiBl4Yk44lbWqu+b/Z/91+tS6B0a+nZX7lTxFdVVQBfvqB4oaYoASVSOP5aOQIdype6DBUAEBcTgcyRqUhLUXgGDZEP47HpRC689l+jy4PKarL8zZk5ogvSvr5TeXVOQ4JcdItHrfuGr14BdrwDSKKi5pIk/788vnIyNoo9XLZvGK7H+6Nuws1tmnCkgnye0vdvjlgQufDSkGS8+0gqYhuGKb5HuvLxl8/zYOzy0pVIoeCNw3SSh5X5io0vyWd/KAwVgFynQmmoAID593dGn+uaMlRQQGGwIFJgcOd47HppAJ4b0E7VfaXlZgz+ujH+IvwFlyNVVNxcO57TIt6Ut1oeqVBIlIBzkgE3l7+jOFQ81S8Jgzsn1LWHRD6LwYJIIb1OwLMD2mPRyFTEx6jbErj6cio6XfgHcvp9BPT9i+sbKkqAz5+oY0+pXoxZwOePKW5uWaf7UuXjLnd+AIAhIgTvPpKKGYNrV+AkCgQMFkQqpaXEY+u02/HKkI6q7hOhw7M/RkNUMiUCAMY1QN5a9R2kurPu/lDuFJoonv6IjtDjp5fvxODOXKRJgYvBgqgO9DoBY/okIT4mQmlMACBvR/3xsPI6CPjvFE6JeNLR7arKdH9UNQC3lP9L8fTHvPs6IyyEv3YpsPE7nKiO9DoBM9PVD2e/la/ir9XL54Hv/6H6NagORDNweIuippIEVEk6zK4a7fDsj5qe7MM1FRQcGCyI6iEtJR6ZI1NVlQD/UUyGSVKxRmPz69wl4m6Wk0p/mK/4lv+rGqxoTQUADOjYHC/XIYQS+SMGC6J6SkuJx84ZA9C4gbJwIUKHFyr/DFUVZHieiPsYs+QtvgqnQCQJeL9qiKLCVwKAsX1b4/8e7V7PThL5DwYLIg2Ehegw574bFLfPFm/G+1VDlIcLnifiHqIZWD9JUVNJkj8mVD6DOeYRTts2CNPjpcEd8dvfBuGlIZ206CmR32CwINJIWko8Fo1MRSOFIxdzzCPwXtVg5eEifwtHLbT2/T+AyxcUNS1AE4yrnIwNYi+n7QQAbwzvgrH92nChJgUllvQm0phZlLDz0HnsOHwOJy5cxprdzofYJ+k/x5TQ1cqenCW/tSOagfltFQWLtyqHYYH5AZcLNeNjIjAzPZlnflBAUvr+rWzlEREpptcJ6NOuKfq0ayqHjPxCnCoqc3hSyDvm+/BwyGa0QCFcVnY2nZSP7h7+McNFfR3drni0YruU4jJUPDegHSbe3o7luSnocZyOyI2qb0l19HYjQodZlaPl/61g/FCCBKx/ltMi9bX9bUXNLkhRyBE7OHw8PiYCi0am4tkB7RkqiMBgQeR2li2pcU7KgG8Ue2B85WScQqzL5xMA4HIh61vUgVmUsO3gOaz79F1IBzYquufDqoF2RysEAfjksR7YOu12Tn0QVcM1FkQeYhYl5OQXouDiZUz5zy922+ggYnLI55gUstbl85UJkQh96Q/oQzijqUR2XgGmr96LkkuX8XP4U4gRLru8xyRF4sbyxXaDxVP9knjeBwUVHptO5GP0OgG92jbBfd1aYmzfJLttROiwXUxR9HwR0mUs+tt4ZOcVaNnNgGMWJfzrmwMYtywXN5f9oDhUAMDKqv61QoUAhgoiZ/inDpEXvDQkGUfOl+Jr45laj+WIHXBBikJjocTl8zwi/Rfdlg3FwJQEjLy5FW5u04Tz/NVk5xVgVtY+nDKVY7p+OZ4K+QKCii/PN1I36/9uEKbH4JQ4vM7zPoicYrAg8pLFo7vji90n8fyqX1BWKVqvi9Dhw6o0/CX0c5fP0VgoxQT9Wryddx++zDuFiBAdbuvQDCN7tsbNbYMvZFimm04VXca2g+fxee4fAIBBuh/xVMgXqp7rnGRAjtgBMZEheHdEN4Y2IoW4xoLIy8yihLc3HcCCTQes13QQsSf8SUQJZS7vlyRgnJ1juxs1CMXc+24I+IWFZlHC9oPn8Pa3B/HL8YsoN4s2j+sg4pfwsYhWOP1h+Y34dOWz+FLsiUUjUwP+a0ikBNdYEPkJvU7A5DvbY9HIVMRf2TkiQof3qu5WdL8EYGboJ9DB9g314qVKjFuWi7+u34cdh87DrGQvq5/JzitA51kbMerDHOQcKawVKgCgl26f4lBh8V7V3dgS0puhgqgOOGJB5EMsQ/lfG09h6bbD+Cl8HBqjRNG6gIcrXsQOJws/GzcIQa82TdCmWTR6tW3il0P7ZlHCzsPnsePQeRw6W4wv8047bT9Ql4N/hC5CtIKRHwAokhpgWuWT0HcahrceTvW7rw+ROyl9/2awIPJR2XkF+HLl+3hTeMN1RU4AxVIEplb+GdnizYqev1GDUIzp1RpVogRAQq82TX1uXYalPPr2w+ewK/8Cfjl+AeVmZb+yBupykBm6AAKgKJhdksLQX7cUs+7tisGdE+rXcaIAxGBBFADMooQfl7yA3sffV9RekoAs8814rmqiyxLU9oTpBQztkuC1nQ+WEZszxWU4cq4US7bl4+LlKtXPE4Iq7AyfgFgUKwplAHDshkm45t6/+lSwIvIlDBZEgUI0Q3ozBSg+6bAseE3FUgReqHwKX4o96/yy1zVriEYNwhBvCENURBjOlZajYVgI7u/aEr3bNdXkDbiiSsQnO47gaOElXCo344cDZ3G6uLxezzlQl4PXQz9AE6FYUXsJgBAWBUw/Buj09XptokDGYEEUSIxZwMrRkCApDheSJC9CnGt+RPPuhOkFpHeJR3NDBE5ekBdGtmzcAL2va4rurWOx48A5vLf1EA6fLUWIXkDXljF4IPVa6PQCfswvBCAh/9wlfJlXoOh8FKUG6X7Eu6H/AqBs+kPClRLpwz/hoW5ELrg1WCxcuBDz58/HqVOn0KVLF7z99tvo0aOH6xsVdkwURVRUVKjtFpHmQkNDodf7yF+xxixg/STFJ3JafrLHV05SvO7Cn6XpdmJh6NvQCyp+pUU2AdIXMFQQKeC2Y9M/++wzTJkyBYsWLULPnj2xYMECDBw4EL/99huaN29er04DQEVFBfLz8yGKtbeNEXlDo0aNEBcXB0FNyUZ3SB4KhBuAT+5R1NzS3fmh76OoMgo/isl1WnfhDwbrdmBh6NuqqmqiQVNgyn4gJMxt/SIKRqpHLHr27Inu3bvjnXfeASCPLiQmJuKZZ57B9OnTXd7vLPFIkoRjx46hsrISCQkJ0OkC85cg+QdJknDp0iWcOXMGjRo1Qny8D9QzEM3AP9oBl86rvvWkFIuMytG1Cmn5uxf1H2NsSLa6UAEBGP4xRyqIVHDLiEVFRQV+/vlnzJgxw3pNp9NhwIAB2LFjh917ysvLUV5+dTGWyWRy+PxVVVW4dOkSEhIS0KBBAzVdI3KLyMhIAMCZM2fQvHlz70+L6PTA4DeAzx9VfWscCpEZugBfmntgmTjA6QhGCKowWr8RPXS/oVQKxSVEoJlgQqkUDqPUCuelRogVTGgslECSBOyQkr0yIrIm9GXcqDusfqTi7jcZKojcRFWwOHfuHMxmM1q0aGFzvUWLFvj111/t3jNnzhxkZGQoen6z2QwACAvj0CT5DkvIrays9H6wAICUYcDJScD2t1TdZtnEMSQkB0OQg0IpCkur7kKIIEIniTAIpZCgQ1vhD/TS7YfewZv1/dhe69okrMVlKRSZVUPxjvlejwSMF/Wf4EbdYShezQpw+oPIA9x+CNmMGTMwZcoU6+cmkwmJiYlO7/H6XDZRNT75/XjXbOCabsC6iUCFsm2VNTUWSjAldLVmXYoUKjEldBX+HPJfLK4ahGuFM2golCNHvB4fmweiSsNfN1EhIp4MyVaVKQDIIxUMFURupeonvWnTptDr9Th92raM7unTpxEXF2f3nvDwcISHh9e9h0RkX6dhQMd0YNWTwD71AcFdcakhyvBc6Brr52n6n/ByyKf4v6pBeN08StVzxcdEYPhNiagSRZy8cBkJjSLRu21T3Hz2M+i+UrE8TNADD3zI6Q8iD1A1XhkWFoZu3bph06ZN1muiKGLTpk3o1auX5p2jqzZv3gxBEHDx4kXF97Ru3RoLFixwW5/U6t+/PyZPnmz9XIv++dp/o8fp9MCDS4AHPgIaNPF2bwDYrx+hE4CxIV9ibehLTu/VCcDdnePxrz/diH+PvRlbp92O5+5sj6kDO+DNP3XF1LvaoY9+H/R7/6OuU/d/KAcxInI71ROhU6ZMweLFi/HRRx9h//79GD9+PEpLS/HYY4+5o39+YcyYMRAEAePGjav12IQJEyAIAsaMGeP5jvm4Xbt24c9//rOitkuXLkWjRo3q9RwBLWUY8PwBYHQWENnI272xSxCALrp8vB8y33otJkKPB1KvweherfDKkI74dfYgvPNIKu658Rr0qn5uiWgGNs8D5iYCHw8FCv6n/IUfWCJ/fYjII1RPej700EM4e/YsXn31VZw6dQo33ngjsrOzay3o9CrRDBzdDpScBqJaAK16u71Ub2JiIlasWIE333zTupOgrKwMy5cvx7XXXuvW1/akiooKzRbXNmvWzCeeI2Do9ECbW4H0t4GVoyHXlfQtggDcGfI/bGy2CBWpY5HcayD0IU5+DYlmYMvfga0LALOyE0pt9BwPpNxX5/4SkXp1Wro9ceJEHD16FOXl5fjxxx/Rs2fdzyPQnDELWJACfHQ3sOoJ+d8FKfJ1N0pNTUViYiJWr74617169Wpce+216Nq1q03b8vJyTJo0Cc2bN0dERARuueUW7Nq1y6bNhg0b0L59e0RGRuK2227DkSNHar3m1q1b0bdvX0RGRiIxMRGTJk1CaWmp4j6PGTMGw4YNQ0ZGBpo1awaDwYBx48bZVD3t378/Jk6ciMmTJ6Np06YYOHAgACAvLw+DBg1CVFQUWrRogVGjRuHcuXPW+0pLSzF69GhERUUhPj4e//znP2u9fs1pjIsXL+Kpp55CixYtEBERgZSUFHzxxRfYvHkzHnvsMRQVFUEQBAiCgFmzZtl9jmPHjuGee+5BVFQUDAYDhg8fbrMmaNasWbjxxhvxySefoHXr1oiJicGf/vQnFBfXbQGkT0oeKtdoiPaBuht2CACuv/A9btg0Cvp/Xuf4Z9OYBcy9Ftgyt26hIiEVGDS3Xn0lIvUCqwLVlfMUYDppe91UIF93c7h4/PHHsWTJEuvnH374od0pohdeeAGrVq3CRx99hNzcXFx33XUYOHAgCgsLAQDHjx/Hfffdh/T0dOzevRtPPvlkreJjhw4dQlpaGu6//37s2bMHn332GbZu3YqJEyeq6vOmTZuwf/9+bN68Gf/+97+xevXqWtuDP/roI4SFhWHbtm1YtGgRLl68iNtvvx1du3bFTz/9hOzsbJw+fRrDhw+33jN16lRs2bIF69atw1dffYXNmzcjNzfXYT9EUcSgQYOwbds2LFu2DEajEXPnzoVer0fv3r2xYMECGAwGFBQUoKCgAM8//7zd57jnnntQWFiILVu24Ouvv8bhw4fx0EMP1frarV27Fl988QW++OILbNmyBXPnBtgbUPJQ4Ll9QP8Xvd0T5y5fAFaOApY9AOxYCFRcBvJ/ALJnyNcrSurwpALQawLw5+807y4Rueb27aYeI5qB7GmwP/x75aih7OlAhyFumxYZOXIkZsyYgaNHjwIAtm3bhhUrVmDz5s3WNqWlpcjMzMTSpUsxaNAgAMDixYvx9ddf44MPPsDUqVORmZmJtm3bWv/Kv/7667F3717MmzfP+jxz5szBiBEjrIsh27Vrh7feegu33norMjMzERERoajPYWFh+PDDD9GgQQN06tQJf/3rXzF16lTMnj3bWvm0Xbt2+Pvf/269529/+xu6du2K119/3Xrtww8/RGJiIn7//XckJCTggw8+wLJly3DHHXcAkMNJy5YtHfbjm2++QU5ODvbv34/27dsDANq0aWN9PCYmBoIgONx9BMghae/evcjPz7duaf7444/RqVMn7Nq1C927dwcgB5ClS5ciOjoaADBq1Chs2rQJr732mqKvmd/Q6YH+04DmHVWdMeIVB7+WPzbWMwh1fggY+g63lBJ5UeAEi6Pba49U2JAA0wm5XVJft3ShWbNmGDJkCJYuXQpJkjBkyBA0bdrUps2hQ4dQWVmJPn36WK+FhoaiR48e2L9/PwBg//79taaXau66+eWXX7Bnzx58+umn1muSJEEUReTn56Njx46K+tylSxebKqe9evVCSUkJjh8/jlatWgEAunXrVuu1v/vuO0RFRdV6vkOHDuHy5cuoqKiw+W+IjY3F9ddf77Afu3fvRsuWLa2hoi7279+PxMREmzopycnJaNSoEfbv328NFq1bt7aGCgCIj4/HmTNn6vy6Pi95qByo838Ajm6Vc7ZOD/z4HlCmIGyERgItewJhDYCwhkCLG+S1S4c3y9tczfU75lxTXUcxVBB5WeAEi5LTrtuoaVdHjz/+uHU6YuHChW57nZKSEjz11FOYNGlSrce0XizasGHDWq+dnp5uM4JiER8fj4MHD6p+DcuCV08IDQ21+VwQhMA/9E6nB9r2lz8sbn0BOLJVDhySGSi7CAg6oHFroHkKcPm888XPN/4JGLbQNrCcPwgY19Ru6wnRCXJficirAidYRCnclaK0XR2lpaWhoqICgiBYFzpW17ZtW+t6BcuIQGVlJXbt2mWd1ujYsSOysmzXg+zcudPm89TUVBiNRlx33XX16u8vv/yCy5cvW9/Yd+7ciaioKKfVUVNTU7Fq1Sq0bt0aIXZW9Ldt2xahoaH48ccfrSHnwoUL+P3333Hrrbfafc7OnTvjjz/+wO+//2531CIsLMxa8t2Rjh074vjx4zh+/Li1/0ajERcvXkRycrLTe4OSZRdJG/v/nyh+jpqBZd8wuWiXWFnPDqo0aJ7bd38RkWuBs3izVW/AkADH9QQFwHCN2/+i0ev12L9/P4xGo91zJRo2bIjx48dj6tSpyM7OhtFoxNixY3Hp0iU88cQTAIBx48bhwIEDmDp1Kn777TcsX74cS5cutXmeadOmYfv27Zg4cSJ2796NAwcOYN26daoXb1ZUVOCJJ56A0WjEhg0bMHPmTEycONHpybITJkxAYWEhHn74YezatQuHDh3Cxo0b8dhjj8FsNiMqKgpPPPEEpk6dim+//RZ5eXkYM2aM0+e89dZb0a9fP9x///34+uuvkZ+fjy+//BLZ2dkA5OmLkpISbNq0CefOncOlS5dqPceAAQNwww03YMSIEcjNzUVOTg5Gjx6NW2+9FTfddJOqrwvVQ6dhwMungX7TAL2ytT71EhkLDP+EVTWJfETgBAudHkizDM3XDBdXPk+b65G/aAwGg9MjZefOnYv7778fo0aNQmpqKg4ePIiNGzeicePGAOSpjFWrVmHt2rXo0qULFi1aZLNQEpD/wt+yZQt+//139O3bF127dsWrr76KhIQEVX2944470K5dO/Tr1w8PPfQQhg4dat3K6UhCQgK2bdsGs9mMu+66CzfccAMmT56MRo0aWcPD/Pnz0bdvX6Snp2PAgAG45ZZbaq3VqGnVqlXo3r07Hn74YSQnJ+OFF16wjlL07t0b48aNw0MPPYRmzZrZLCa1EAQB69atQ+PGjdGvXz8MGDAAbdq0wWeffabqa0Ia0OmB218EXjrpvp0p13QHRq0Dph5kqCDyIYIkSR6touPsPPeysjLk5+cjKSlJ8a6GWoxZ8u6Q6gs5DdfIoYK/fGyMGTMGFy9exNq1a73dFZ+myfdlsDNmAeufBS4X1v+5wg3A0LdZopvIw5y9f1cXOGssLCwr4D1ceZOInLD8XFoWix7eDJzY5fI2ebRRAjreAzRtL+/oan0Lf56JfFjgBQtA/qXjpi2lRFRH1ReL3vEy8NUrwI53AMnJjhxDAkcbifxMYAYLUqTmglAij7prNnD7K8CuxcCFI/I2126PAyd+4mgjkR9jsCAi7wkJk8tvV8fRRiK/Fji7QoiIiMjrfDJYeHijCpFTAV+Vk4hIQz41FRIaGgpBEHD27Fk0a9YMguCo2BWR+0mShIqKCpw9exY6nQ5hYTyDgojIFZ8KFnq9Hi1btsQff/yBI0eOeLs7RACABg0a4Nprr3VaOZSIiGQ+FSwAICoqCu3atUNlpYfPGSCyQ6/XIyQkhKNnREQK+VywAORf5vbO2SAiIiLfxrFdIiIi0gyDBREREWmGwYKIiIg04/E1FpYaFSaTydMvTURERHVked92VWvK48GiuLgYAJCYmOjplyYiIqJ6Ki4uRkxMjMPHBcnDZS5FUcTJkycRHR0ddFv4TCYTEhMTcfz4cadn2ZN2+DX3PH7NPY9fc88Lxq+5JEkoLi5GQkKC07o+Hh+x0Ol0aNmypadf1qcYDIag+Ub0Ffyaex6/5p7Hr7nnBdvX3NlIhQUXbxIREZFmGCyIiIhIMwwWHhQeHo6ZM2ciPDzc210JGvyaex6/5p7Hr7nn8WvumMcXbxIREVHg4ogFERERaYbBgoiIiDTDYEFERESaYbAgIiIizTBYeMGRI0fwxBNPICkpCZGRkWjbti1mzpyJiooKb3ctoL322mvo3bs3GjRogEaNGnm7OwFp4cKFaN26NSIiItCzZ0/k5OR4u0sB7fvvv0d6ejoSEhIgCALWrl3r7S4FtDlz5qB79+6Ijo5G8+bNMWzYMPz222/e7pbPYbDwgl9//RWiKOK9997Dvn378Oabb2LRokV48cUXvd21gFZRUYEHH3wQ48eP93ZXAtJnn32GKVOmYObMmcjNzUWXLl0wcOBAnDlzxttdC1ilpaXo0qULFi5c6O2uBIUtW7ZgwoQJ2LlzJ77++mtUVlbirrvuQmlpqbe75lO43dRHzJ8/H5mZmTh8+LC3uxLwli5dismTJ+PixYve7kpA6dmzJ7p374533nkHgHwuUGJiIp555hlMnz7dy70LfIIgYM2aNRg2bJi3uxI0zp49i+bNm2PLli3o16+ft7vjMzhi4SOKiooQGxvr7W4Q1UlFRQV+/vlnDBgwwHpNp9NhwIAB2LFjhxd7RuQ+RUVFAMDf3TUwWPiAgwcP4u2338ZTTz3l7a4Q1cm5c+dgNpvRokULm+stWrTAqVOnvNQrIvcRRRGTJ09Gnz59kJKS4u3u+BQGCw1Nnz4dgiA4/fj1119t7jlx4gTS0tLw4IMPYuzYsV7quf+qy9eciKi+JkyYgLy8PKxYscLbXfE5Hj82PZD95S9/wZgxY5y2adOmjfV/nzx5Erfddht69+6N999/3829C0xqv+bkHk2bNoVer8fp06dtrp8+fRpxcXFe6hWRe0ycOBFffPEFvv/+e7Rs2dLb3fE5DBYaatasGZo1a6ao7YkTJ3DbbbehW7duWLJkCXQ6Dh7VhZqvOblPWFgYunXrhk2bNlkXD4qiiE2bNmHixIne7RyRRiRJwjPPPIM1a9Zg8+bNSEpK8naXfBKDhRecOHEC/fv3R6tWrfCPf/wDZ8+etT7Gv+7c59ixYygsLMSxY8dgNpuxe/duAMB1112HqKgo73YuAEyZMgWPPvoobrrpJvTo0QMLFixAaWkpHnvsMW93LWCVlJTg4MGD1s/z8/Oxe/duxMbG4tprr/VizwLThAkTsHz5cqxbtw7R0dHW9UMxMTGIjIz0cu98iEQet2TJEgmA3Q9yn0cffdTu1/y7777zdtcCxttvvy1de+21UlhYmNSjRw9p586d3u5SQPvuu+/sfk8/+uij3u5aQHL0e3vJkiXe7ppPYR0LIiIi0gwn9omIiEgzDBZERESkGQYLIiIi0gyDBREREWmGwYKIiIg0w2BBREREmmGwICIiIs0wWBAREZFmGCyIiIhIMwwWREREpBkGCyIiItIMgwURERFp5v8B7C/W56BLdgwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "xs = np.random.normal(size=(128, 1))\n",
    "ys = xs ** 2\n",
    "\n",
    "for _ in range(1000):\n",
    "    params = update(params, xs, ys)  \n",
    "\n",
    "plt.scatter(xs, ys)\n",
    "plt.scatter(xs, forward(params, xs), label='Model prediction')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key paths\n",
    "\n",
    "In a pytree each leaf has a key path. A key path for a leaf is **a list of keys**, where the length of the list is equal to the depth of the leaf in the pytree . Each key is a **hashable** object that represents an index into the corresponding pytree node type. The type of the key depends on the pytree node type; for example, the type of keys for dicts is different from the type of keys for tuples.\n",
    "\n",
    "![Alt text](image-4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value of tree[0]: 1\n",
      "Value of tree[1]['k1']: 2\n",
      "Value of tree[1]['k2'][0]: 3\n",
      "Value of tree[1]['k2'][1]: 4\n",
      "Value of tree[2].name: foo\n",
      "Key path of tree[0]: (SequenceKey(idx=0),)\n",
      "Key path of tree[1]['k1']: (SequenceKey(idx=1), DictKey(key='k1'))\n",
      "Key path of tree[1]['k2'][0]: (SequenceKey(idx=1), DictKey(key='k2'), SequenceKey(idx=0))\n",
      "Key path of tree[1]['k2'][1]: (SequenceKey(idx=1), DictKey(key='k2'), SequenceKey(idx=1))\n",
      "Key path of tree[2].name: (SequenceKey(idx=2), GetAttrKey(name='name'))\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "ATuple = collections.namedtuple(\"ATuple\", ('name'))\n",
    "\n",
    "tree = [1, {'k1': 2, 'k2': (3, 4)}, ATuple('foo')]\n",
    "flattened, _ = jax.tree_util.tree_flatten_with_path(tree)\n",
    "for key_path, value in flattened:\n",
    "    print(f'Value of tree{jax.tree_util.keystr(key_path)}: {value}')\n",
    "for key_path, _ in flattened:\n",
    "    print(f'Key path of tree{jax.tree_util.keystr(key_path)}: {repr(key_path)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Pytree Nodes\n",
    "i.e. define container class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<__main__.MyContainer at 0x2cc54d2a730>,\n",
       " <__main__.MyContainer at 0x2cc59f279d0>]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Old way to define a class:\n",
    "class MyContainer:\n",
    "    \"\"\"A named container.\"\"\"\n",
    "\n",
    "    def __init__(self, name: str, a: int, b: int, c: int):\n",
    "        self.name = name\n",
    "        self.a = a\n",
    "        self.b = b\n",
    "        self.c = c\n",
    "        \n",
    "jax.tree_util.tree_leaves([\n",
    "    MyContainer('Alice', 1, 2, 3),\n",
    "    MyContainer('Bob', 4, 5, 6)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To register our container as pytree\n",
    "# We need to define a `tree_flatten` and `tree_unflatten` method\n",
    "\n",
    "from typing import Iterable\n",
    "\n",
    "def flatten_MyContainer(container) -> tuple[Iterable[int], str]:\n",
    "    \"\"\"Returns an iterable over container contents, and aux data.\"\"\"\n",
    "    flat_contents = [container.a, container.b, container.c]\n",
    "\n",
    "    # we don't want the name to appear as a child, so it is auxiliary data.\n",
    "    # auxiliary data is usually a description of the structure of a node,\n",
    "    # e.g., the keys of a dict -- anything that isn't a node's children.\n",
    "    aux_data = container.name\n",
    "    return flat_contents, aux_data\n",
    "\n",
    "def unflatten_MyContainer(\n",
    "        aux_data: str, flat_contents: Iterable[int]) -> MyContainer:\n",
    "    \"\"\"Converts aux data and the flat contents into a MyContainer.\"\"\"\n",
    "    return MyContainer(aux_data, *flat_contents)\n",
    "\n",
    "jax.tree_util.register_pytree_node(\n",
    "    MyContainer, flatten_MyContainer, unflatten_MyContainer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value of tree[0][<flat index 0>]: 1\n",
      "Value of tree[0][<flat index 1>]: 2\n",
      "Value of tree[0][<flat index 2>]: 3\n",
      "Value of tree[1][<flat index 0>]: 4\n",
      "Value of tree[1][<flat index 1>]: 5\n",
      "Value of tree[1][<flat index 2>]: 6\n"
     ]
    }
   ],
   "source": [
    "\n",
    "jax.tree_util.tree_leaves([\n",
    "    MyContainer('Alice', 1, 2, 3),\n",
    "    MyContainer('Bob', 4, 5, 6)\n",
    "])\n",
    "\n",
    "flattened, _ = jax.tree_util.tree_flatten_with_path([\n",
    "    MyContainer('Alice', 1, 2, 3),\n",
    "    MyContainer('Bob', 4, 5, 6)\n",
    "])\n",
    "for key_path, value in flattened:\n",
    "    print(f'Value of tree{jax.tree_util.keystr(key_path)}: {value}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also define the key path version of the above container\n",
    "class MyKeyPathContainer(MyContainer):\n",
    "    pass\n",
    "\n",
    "def flatten_with_keys_MyKeyPathContainer(container) -> tuple[Iterable[int], str]:\n",
    "    \"\"\"Returns an iterable over container contents, and aux data.\"\"\"\n",
    "\n",
    "    # GetAttrKey is a common way to express an attribute key. Users are free\n",
    "    # to pick any other expression that fits their use cases the best.\n",
    "    flat_contents = [(jax.tree_util.GetAttrKey('a'), container.a),\n",
    "                    (jax.tree_util.GetAttrKey('b'), container.b),\n",
    "                    (jax.tree_util.GetAttrKey('c'), container.c)]\n",
    "\n",
    "    # we don't want the name to appear as a child, so it is auxiliary data.\n",
    "    # auxiliary data is usually a description of the structure of a node,\n",
    "    # e.g., the keys of a dict -- anything that isn't a node's children.\n",
    "    aux_data = container.name\n",
    "    return flat_contents, aux_data\n",
    "\n",
    "def unflatten_MyKeyPathContainer(\n",
    "        aux_data: str, flat_contents: Iterable[int]) -> MyKeyPathContainer:\n",
    "    \"\"\"Converts aux data and the flat contents into a MyContainer.\"\"\"\n",
    "    return MyKeyPathContainer(aux_data, *flat_contents)\n",
    "\n",
    "jax.tree_util.register_pytree_with_keys(\n",
    "    MyKeyPathContainer, flatten_with_keys_MyKeyPathContainer, unflatten_MyKeyPathContainer)\n",
    "\n",
    "jax.tree_util.tree_leaves([\n",
    "    MyKeyPathContainer('Alice', 1, 2, 3),\n",
    "    MyKeyPathContainer('Bob', 4, 5, 6)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value of tree[0].a: 1\n",
      "Value of tree[0].b: 2\n",
      "Value of tree[0].c: 3\n",
      "Value of tree[1].a: 4\n",
      "Value of tree[1].b: 5\n",
      "Value of tree[1].c: 6\n"
     ]
    }
   ],
   "source": [
    "\n",
    "flattened, _ = jax.tree_util.tree_flatten_with_path([\n",
    "    MyKeyPathContainer('Alice', 1, 2, 3),\n",
    "    MyKeyPathContainer('Bob', 4, 5, 6)\n",
    "])\n",
    "for key_path, value in flattened:\n",
    "    print(f'Value of tree{jax.tree_util.keystr(key_path)}: {value}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Alice', 1, 2, 3, 'Bob', 4, 5, 6]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import NamedTuple, Any\n",
    "\n",
    "class MyOtherContainer(NamedTuple):\n",
    "  name: str\n",
    "  a: Any\n",
    "  b: Any\n",
    "  c: Any\n",
    "\n",
    "# NamedTuple subclasses are handled as pytree nodes, so\n",
    "# this will work out-of-the-box:\n",
    "jax.tree_util.tree_leaves([\n",
    "    MyOtherContainer('Alice', 1, 2, 3),\n",
    "    MyOtherContainer('Bob', 4, 5, 6)\n",
    "])\n",
    "\n",
    "# Note that name is now also a child."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import NamedTuple, Any\n",
    "\n",
    "@jax.tree_util.register_static\n",
    "class StaticStr(str):\n",
    "  pass\n",
    "\n",
    "\n",
    "class YetAnotherContainer(NamedTuple):\n",
    "  name: StaticStr\n",
    "  a: Any\n",
    "  b: Any\n",
    "  c: Any\n",
    "\n",
    "\n",
    "# NamedTuple subclasses are handled as pytree nodes, so\n",
    "# this will work out-of-the-box:\n",
    "jax.tree_util.tree_leaves([\n",
    "    YetAnotherContainer(StaticStr('Alice'), 1, 2, 3),\n",
    "    YetAnotherContainer(StaticStr('Bob'), 4, 5, 6)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common gotchas and patterns\n",
    "\n",
    "https://jax.readthedocs.io/en/latest/jax-101/05.1-pytrees.html#common-pytree-gotchas-and-patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'obs': [3, 4], 't': [1, 2]}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\test\\AppData\\Local\\Temp\\ipykernel_6664\\468251687.py:10: DeprecationWarning: jax.tree_transpose is deprecated: use jax.tree_util.tree_transpose.\n",
      "  jax.tree_transpose(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'obs': [3, 4], 't': [1, 2]}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def tree_transpose(list_of_trees):\n",
    "    \"\"\"Convert a list of trees of identical structure into a single tree of lists.\"\"\"\n",
    "    return jax.tree_map(lambda *xs: list(xs), *list_of_trees)\n",
    "\n",
    "\n",
    "# Convert a dataset from row-major to column-major:\n",
    "episode_steps = [dict(t=1, obs=3), dict(t=2, obs=4)]\n",
    "print(tree_transpose(episode_steps))\n",
    "\n",
    "jax.tree_transpose(\n",
    "  outer_treedef = jax.tree_util.tree_structure([0 for e in episode_steps]),\n",
    "  inner_treedef = jax.tree_util.tree_structure(episode_steps[0]),\n",
    "  pytree_to_transpose = episode_steps\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`jax.pmap` to transform a function written for one device into a function that runs in parallel on multiple devices.\n",
    "\n",
    "Keep in mind that when calling the transformed function, the size of the specified axis in arguments must not exceed the number of devices available to the host.\n",
    "\n",
    "`jax.pmap` JIT-compiles the function given to it as part of its operation, so there is no need to additionally `jax.jit` it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[CpuDevice(id=0)]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import jax\n",
    "jax.devices()\n",
    "# Should have multiple devices for this part of code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hosts and devices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[CpuDevice(id=0), CpuDevice(id=1)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import jax\n",
    "os.environ['XLA_FLAGS'] = '--xla_force_host_platform_device_count=2'\n",
    "# We can emulate a multi-device environment by setting this flag.\n",
    "# Should set it before initialization.\n",
    "jax.devices()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2, 3, 4]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import jax.numpy as jnp\n",
    "\n",
    "x = np.arange(5)\n",
    "w = np.array([2., 3., 4.])\n",
    "\n",
    "def convolve(x, w):\n",
    "    output = []\n",
    "    for i in range(1, len(x)-1):\n",
    "        output.append(jnp.dot(x[i-1:i+2], w))\n",
    "    return jnp.array(output)\n",
    "\n",
    "convolve(x, w)\n",
    "n_devices = jax.local_device_count() \n",
    "xs = np.arange(5 * n_devices).reshape(-1, 5)\n",
    "ws = np.stack([w] * n_devices)\n",
    "\n",
    "xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[11., 20., 29.]], dtype=float32)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jax.vmap(convolve)(xs, ws)\n",
    "jax.pmap(convolve)(xs, ws)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[ 78., 138., 198.]], dtype=float32)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If we were to run another parallel computation, the elements would stay on their respective devices, without incurring cross-device communication costs.\n",
    "jax.pmap(convolve)(xs, jax.pmap(convolve)(xs, ws))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[11., 20., 29.]], dtype=float32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# similar to `vmap`, `pmap` also has `in_axes` and `out_axes` arguments:\n",
    "jax.pmap(convolve, in_axes=(0, None))(xs, w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Communication between devices\n",
    "\n",
    "collective ops are needed. `jax.lax.p*`.\n",
    "\n",
    "In order to use the collective ops we must specify the** name of the pmap-ed axis** through axis_name argument, and then refer to it when calling the op."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array([[1., 1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def normalized_convolution(x, w):\n",
    "  output = []\n",
    "  for i in range(1, len(x)-1):\n",
    "    output.append(jnp.dot(x[i-1:i+2], w))\n",
    "  output = jnp.array(output)\n",
    "  return output / jax.lax.psum(output, axis_name='p')\n",
    "\n",
    "jax.pmap(normalized_convolution, axis_name='p')(xs, ws)\n",
    "\n",
    "# jax.vmap also supports axis_name, which allows jax.lax.p* operations to be used in the vectorisation context in the same way they would be used in a jax.pmap:\n",
    "jax.vmap(normalized_convolution, axis_name='p')(xs, ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nesting pmap and vmap\n",
    "\n",
    "We need to specify different axis name for vmap and pmap.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jax.vmap(jax.pmap(f, axis_name='i'), axis_name='j')\n",
    "# jax.lax.psum(..., axis_name='i')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: Data parallelism"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import NamedTuple\n",
    "import functools\n",
    "\n",
    "class Params(NamedTuple):\n",
    "  weight: jnp.ndarray\n",
    "  bias: jnp.ndarray\n",
    "\n",
    "\n",
    "def init(rng) -> Params:\n",
    "  \"\"\"Returns the initial model params.\"\"\"\n",
    "  weights_key, bias_key = jax.random.split(rng)\n",
    "  weight = jax.random.normal(weights_key, ())\n",
    "  bias = jax.random.normal(bias_key, ())\n",
    "  return Params(weight, bias)\n",
    "\n",
    "\n",
    "def loss_fn(params: Params, xs: jnp.ndarray, ys: jnp.ndarray) -> jnp.ndarray:\n",
    "  \"\"\"Computes the least squares error of the model's predictions on x against y.\"\"\"\n",
    "  pred = params.weight * xs + params.bias\n",
    "  return jnp.mean((pred - ys) ** 2)\n",
    "\n",
    "LEARNING_RATE = 0.005\n",
    "\n",
    "# So far, the code is identical to the single-device case. Here's what's new:\n",
    "\n",
    "\n",
    "# Remember that the `axis_name` is just an arbitrary string label used\n",
    "# to later tell `jax.lax.pmean` which axis to reduce over. Here, we call it\n",
    "# 'num_devices', but could have used anything, so long as `pmean` used the same.\n",
    "@functools.partial(jax.pmap, axis_name='num_devices')\n",
    "def update(params: Params, xs: jnp.ndarray, ys: jnp.ndarray) -> tuple[Params, jnp.ndarray]:\n",
    "  \"\"\"Performs one SGD update step on params using the given data.\"\"\"\n",
    "\n",
    "  # Compute the gradients on the given minibatch (individually on each device).\n",
    "  loss, grads = jax.value_and_grad(loss_fn)(params, xs, ys)\n",
    "\n",
    "  # Combine the gradient across all devices (by taking their mean).\n",
    "  grads = jax.lax.pmean(grads, axis_name='num_devices')\n",
    "\n",
    "  # Also combine the loss. Unnecessary for the update, but useful for logging.\n",
    "  loss = jax.lax.pmean(loss, axis_name='num_devices')\n",
    "\n",
    "  # Each device performs its own update, but since we start with the same params\n",
    "  # and synchronise gradients, the params stay in sync.\n",
    "  new_params = jax.tree_map(\n",
    "      lambda param, g: param - g * LEARNING_RATE, params, grads)\n",
    "\n",
    "  return new_params, loss\n",
    "\n",
    "# Generate true data from y = w*x + b + noise\n",
    "true_w, true_b = 2, -1\n",
    "xs = np.random.normal(size=(128, 1))\n",
    "noise = 0.5 * np.random.normal(size=(128, 1))\n",
    "ys = xs * true_w + true_b + noise\n",
    "\n",
    "# Initialise parameters and replicate across devices.\n",
    "params = init(jax.random.PRNGKey(123))\n",
    "n_devices = jax.local_device_count()\n",
    "replicated_params = jax.tree_map(lambda x: jnp.array([x] * n_devices), params)\n",
    "\n",
    "def split(arr):\n",
    "  \"\"\"Splits the first axis of `arr` evenly across the number of devices.\"\"\"\n",
    "  return arr.reshape(n_devices, arr.shape[0] // n_devices, *arr.shape[1:])\n",
    "\n",
    "# Reshape xs and ys for the pmapped `update()`.\n",
    "x_split = split(xs)\n",
    "y_split = split(ys)\n",
    "\n",
    "type(x_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after first `update()`, `replicated_params.weight` is a <class 'jaxlib.xla_extension.ArrayImpl'>\n",
      "after first `update()`, `loss` is a <class 'jaxlib.xla_extension.ArrayImpl'>\n",
      "after first `update()`, `x_split` is a <class 'numpy.ndarray'>\n",
      "Step   0, loss: 6.821\n",
      "Step 100, loss: 0.654\n",
      "Step 200, loss: 0.236\n",
      "Step 300, loss: 0.204\n",
      "Step 400, loss: 0.201\n",
      "Step 500, loss: 0.200\n",
      "Step 600, loss: 0.200\n",
      "Step 700, loss: 0.200\n",
      "Step 800, loss: 0.200\n",
      "Step 900, loss: 0.200\n"
     ]
    }
   ],
   "source": [
    "def type_after_update(name, obj):\n",
    "    print(f\"after first `update()`, `{name}` is a\", type(obj))\n",
    "\n",
    "# Actual training loop.\n",
    "for i in range(1000):\n",
    "\n",
    "    # This is where the params and data gets communicated to devices:\n",
    "    replicated_params, loss = update(replicated_params, x_split, y_split)\n",
    "\n",
    "    # The returned `replicated_params` and `loss` are now both jax.Arrays,\n",
    "    # indicating that they're on the devices.\n",
    "    # `x_split`, of course, remains a NumPy array on the host.\n",
    "    if i == 0:\n",
    "        type_after_update('replicated_params.weight', replicated_params.weight)\n",
    "        type_after_update('loss', loss)\n",
    "        type_after_update('x_split', x_split)\n",
    "\n",
    "    if i % 100 == 0:\n",
    "        # Note that loss is actually an array of shape [num_devices], with identical\n",
    "        # entries, because each device returns its copy of the loss.\n",
    "        # So, we take the first element to print it.\n",
    "        print(f\"Step {i:3d}, loss: {loss[0]:.3f}\")\n",
    "\n",
    "\n",
    "# Plot results.\n",
    "\n",
    "# Like the loss, the leaves of params have an extra leading dimension,\n",
    "# so we take the params from the first device.\n",
    "params = jax.device_get(jax.tree_map(lambda x: x[0], replicated_params))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABT1UlEQVR4nO3deVxUVf8H8M8wsqkwioJgoiBaSuSuiJq5FqWmrbZYamZpWm6PqW1mPaWm/rTUzLK0UivLzK0otzJ9NExDQ1wJ0hRcUAdEGJa5vz+mGebOeu8wO5/368Xr6Z65y2FePt6v53zP9ygEQRBARERE5AEBnu4AERER1VwMRIiIiMhjGIgQERGRxzAQISIiIo9hIEJEREQew0CEiIiIPIaBCBEREXkMAxEiIiLymFqe7oAtWq0W58+fR1hYGBQKhae7Q0RERBIIgoCioiI0btwYAQG2xzy8OhA5f/48YmNjPd0NIiIicsDZs2fRpEkTm+d4dSASFhYGQPeLhIeHe7g3REREJEVhYSFiY2MN73FbvDoQ0U/HhIeHMxAhIiLyMVLSKpisSkRERB7DQISIiIg8hoEIEREReYxX54hIIQgCKioqUFlZ6emuUA0XGBgIpVLp6W4QEfkUnw5EysrKkJeXhxs3bni6K0RQKBRo0qQJ6tat6+muEBH5DJ8NRLRaLXJycqBUKtG4cWMEBQWx6Bl5jCAIuHTpEv755x+0bNmSIyNERBL5bCBSVlYGrVaL2NhY1K5d29PdIUJkZCRyc3NRXl7OQISISCKfT1a1VzqWyF04IkdEJB/f4kREROQxbgtE5syZA4VCgYkTJ7rrkUREROTl3BKIHDhwAMuXL0ebNm3c8TgC8PPPP0OhUODatWuSr4mLi8OiRYtc1ic5Xn/9dbRr185wPGLECAwZMqRa93TGPYiIyLlcHohcv34djz/+OD766CPUr1/f1Y/zCSNGjIBCocCYMWPMPhs3bhwUCgVGjBjh/o7Z8frrr0OhUEChUKBWrVqIi4vDpEmTcP36dZc/+91338WqVasknZubmwuFQoGMjAyH70FERO7h8kBk3LhxGDBgAPr162f3XI1Gg8LCQtGPv4qNjcWXX36JkpISQ1tpaSnWrl2Lpk2berBntt16663Iy8tDbm4u5s6diw8//BBTpkyxeG5ZWZnTnqtSqVCvXj2P34OIyG+cPg3MmAEcP+7Rbrg0EPnyyy9x6NAhzJ49W9L5s2fPhkqlMvzExsbKe6AgAMXFnvkRBFld7dChA2JjY/Htt98a2r799ls0bdoU7du3F52r0WjwwgsvICoqCiEhIejRowcOHDggOuf777/HzTffjNDQUPTu3Ru5ublmz9yzZw9uv/12hIaGIjY2Fi+88AKKi4tl9btWrVqIjo5GkyZNMHToUDz++OPYtGkTgKrplBUrViA+Ph4hISEAgGvXruHpp59GZGQkwsPD0adPHxw+fFh03zlz5qBRo0YICwvDqFGjUFpaKvrcdFpFq9XinXfeQYsWLRAcHIymTZvirbfeAgDEx8cDANq3bw+FQoFevXpZvIe971U/vbVjxw506tQJtWvXRrdu3XDixAlZ3xkRkddZsABo2RKYMweYO9ejXXFZIHL27FlMmDABa9asMbyQ7JkxYwbUarXh5+zZs/IeeuMGULeuZ34cqO761FNPYeXKlYbjTz75BCNHjjQ778UXX8T69evx6aef4tChQ2jRogXuuusuXLlyBYDuu77//vsxaNAgZGRk4Omnn8b06dNF98jOzkZqaioeeOABHDlyBF999RX27NmD8ePHy+63sdDQUNHIx+nTp7F+/Xp8++23hqmRhx56CBcvXsQPP/yAgwcPokOHDujbt6+h/+vWrcPrr7+Ot99+G7///jtiYmLw/vvv23zujBkzMGfOHLz66qvIysrC2rVr0ahRIwBAeno6AGD79u3Iy8sTBXvG7H2vei+//DIWLFiA33//HbVq1cJTTz3l0HdFRORxlZVAZCTwn/9UtQ0f7rn+AIDgIhs2bBAACEql0vADQFAoFIJSqRQqKirs3kOtVgsABLVabfZZSUmJkJWVJZSUlFQ1Xr8uCLqxCff/XL8u+bsZPny4MHjwYOHixYtCcHCwkJubK+Tm5gohISHCpUuXhMGDBwvDhw//91e6LgQGBgpr1qwxXF9WViY0btxYeOeddwRBEIQZM2YIiYmJomdMmzZNACBcvXpVEARBGDVqlPDMM8+Izvn111+FgIAAw3fYrFkzYeHChVb7PXPmTKFt27aG499//11o2LCh8OCDDxo+DwwMFC5evCh6Rnh4uFBaWiq6V0JCgrB8+XJBEAQhJSVFeO6550SfJycni56l/84EQRAKCwuF4OBg4aOPPrLYz5ycHAGA8Mcff4jaje8h5XvdtWuXAEDYvn274ZytW7cKAMR/7v5l8c8kEZG3yMoyf3f9849LHmXr/W3KZZVV+/btiz///FPUNnLkSLRq1QrTpk1zTeXJ2rUBNyROWn22TJGRkRgwYABWrVoFQRAwYMAANGzYUHROdnY2ysvL0b17d0NbYGAgunTpgmPHjgEAjh07huTkZNF1KSkpouPDhw/jyJEjWLNmjaFNEARDqfzWrVtL6vOff/6JunXrorKyEmVlZRgwYACWLFli+LxZs2aIjIwUPff69eto0KCB6D4lJSXIzs429N80cTclJQW7du2y2Idjx45Bo9Ggb9++kvpsiZTvVc94tVdMTAwA4OLFi16dy0NE/qdSKyA95wouFpUiKiwEXeIjoAyQWEjx7beBl1+uOu7UCUhPB7ygEKPLApGwsDAkJSWJ2urUqYMGDRqYtTuNQgHUqeOae7vIU089ZZgeWbp0qcuec/36dTz77LN44YUXzD6T80K95ZZbsGnTJtSqVcuwx4+xOibf//Xr1xETE4Off/7Z7F6OJo6GhoY6dJ2jAgMDDf+tr56q1Wrd2gciqtnSMvMwa3MW8tRV+XMxqhDMHJSI1KQY6xdWVAAqlTh94PPPgWHDXNhbeVhZ1cNSU1NRVlaG8vJy3HXXXWafJyQkICgoCHv37jW0lZeX48CBA0hMTAQAtG7d2pAXobd//37RcYcOHZCVlYUWLVqY/ZgGE7YEBQWhRYsWiIuLk3Rdhw4dkJ+fj1q1apk9Vz/607p1a/z22282+2+sZcuWCA0NxY4dO6z2EQAqKyut3kPK90pE5A3SMvMwdvUhURACAPnqUoxdfQhpmXmWL/zzTyAwUByE5OV5VRACuHnTO0v/Kq7plEqlYSrA0nRVnTp1MHbsWEydOhURERFo2rQp3nnnHdy4cQOjRo0CAIwZMwYLFizA1KlT8fTTT+PgwYNm9TKmTZuGrl27Yvz48Xj66adRp04dZGVlYdu2baKpFWfr168fUlJSMGTIELzzzju4+eabcf78eWzduhX33XcfOnXqhAkTJmDEiBHo1KkTunfvjjVr1uDo0aNo3ry5xXuGhIRg2rRpePHFFxEUFITu3bvj0qVLOHr0KEaNGoWoqCiEhoYiLS0NTZo0QUhICFQqlegeUr5XIiJ7qjVdIvH+szZnwdK6TAGAAsCszVnonxgtfu6rrwL//W/Vcc+ewM8/e8VUjCmf3X3Xn4SHh9v8fM6cOdBqtXjiiSdQVFSETp064ccffzQUiGvatCnWr1+PSZMmYfHixejSpQvefvtt0eqONm3a4JdffsHLL7+M22+/HYIgICEhAUOHDnXp76ZQKPD999/j5ZdfxsiRI3Hp0iVER0ejZ8+ehlUuQ4cORXZ2Nl588UWUlpbigQcewNixY/Hjjz9ave+rr76KWrVq4bXXXsP58+cRExNjyDOpVasW3nvvPbzxxht47bXXcPvtt1sMgu19r0REtjg8XSJDes4Vs5EQYwKAPHUp0nOuICWhAVBWBgQHi09atw546CGn9McVFIIgswCGGxUWFkKlUkGtVpu9rEtLS5GTkyOqV0HkSfwzSVRz6KdLTF+g+vGGZcM6OCUY2ZhxDhO+zLB73ruPtMNg7QWgY0fxB5cuASaLINzB1vvbFHNEiIiIZLA3XQLopksqtdX/d35UmLR/1HR6f444CLnrLt0CXQ8EIXJxaoaIiEgG2dMl1dAlPgIxqhDkq0stBj7BFWU4seB+ceN33wGDB1frue7EEREiIqJ/VWoF7MsuwMaMc9iXXWBxVONikfUgxJHzbFEGKDBzkG4ln2maafvzJ8yDkCtXfCoIATgiQkREBEB68qnU6RKp59mTmhSDZcM6iPr2+rYPMOLQlqqThgwBNmxwyvPczecDES/OtaUahn8WiXyXteRTfa0O4+RTe9MlCgDRKt1SXmdJTYpB/8Ro/J71D5JvMylC+cMPQGqq057lbj47NaOvdnnDgc3miFxBv/mfS7YvICKXkZt8amu6RH88c1CiU+uJAIDyf3vNg5Br13w6CAF8eEREqVSiXr16uHjxIgCgdu3ahvLbRO6m1Wpx6dIl1K5dG7Vq+ez/rYhqJEeSTy1NlwC6kRBn1hExePpp4OOPq44fewww2jvMl/n035jR0dEAYAhGiDwpICAATZs2ZUBM5GMcTT7VT5e4srIqrl8HwsLEbTt2AH36VPvWrq4KK5VPByIKhQIxMTGIiopCeXm5p7tDNVxQUBACAnx2tpOoxqpO8qkyQFHtJbpW7dwJmO4yXlQE1K1b7Vu7oyqsVD4diOgplUrOyxMRkUM8kXxq17Bh4qmXUaOAFSuccms5ibnuwH++ERFRjeap5FOL1GrdxnTGQcju3U4LQtxZFVYqBiJERFTj6ZNPo1Xi6ZdoVYj7RgjS0oB69cRtN24At9/utEfIScx1F7+YmiEiIqoutySfWnP//eKCZOPHA4sXO/0x7qwKKxUDESIion+5NPnUkqtXgQiT3JP9+4HkZJc8zt1VYaXg1AwREZEnbNpkHoSUlLgsCAGqEnOtjfEooFs9487EXAYiRERE7nb33eLN6aZOBQQBCHHtSIRXJeb+i4EIERGRu1y+rFsVk5ZW1XbwIPDOO27rglck5hphjggREZE7fP018PDD4jaNBggKcntXPJqYa4KBCBERkSsJAtCrl64eiN6rrwJvvOGxLgEeSMy1goEIERGRq+TnAzEmUx1HjgC33eaZ/ngh5ogQERG5wpo14iCkbl2gvJxBiAkGIkRE5PcqtQL2ZRdgY8Y57MsucG0Jc0EAOnfW7Rej99Zbug3ranEiwhS/ESIi8mtu3Wn23DmgSRNx27FjQKtWzn2OH+GICBER+S39TrOm+6vod5pNy8xz3sM++UQchERGAhUVLg1C3DrS4yIcESEiIr9kb6dZBXQ7zfZPjK7eslVBAG69VTfyobdgATB5suP3lMCtIz0uxBERIiLyS27Zafbvv4GAAHEQcuqUW4IQt430uBgDESIi8ksu32n2/feBuLiq46ZNgcpKoEULx+4nkb2RHkA30uMr0zQMRIiIyC/lXi6WdJ7snWa1WiA+Hhg3rqptyZKq0REXc8tIjxsxR4SIiPxOWmYeFm4/ZfMcBYCIOkHIV5dgX3aBtBLn2dnmIx45OeKRERdz+UiPmzEQISIiv6KfurBHAFBQXIZJ6w4DkJDouXChOPejdWvg6FHdJnZuJHUER/ZIj4dwaoaIiPyKvakLa/LVpRiz+hDe2HxUvBS2shJo1EgchHz0EZCV5fYgBAC6xEcgRhUCa09WQBdUdYmPcGe3HMYRESIi8iuOTknoUzs/2ZuLT/bmIkYVgnduDcTtg+8Qn3j2rHnRMjdSBigwc1Aixq4+BAUgSlrVByczByV6ZCddR3BEhIiI/IqzpiTuS/tMHIR07KhLVPVgEKKXmhSDZcM6IFol/l2jVSFYNqyDT9UR4YgIERH5Ff3URb661OISV3uU2kocWTQUdcqrRla0n36GgCefcF4nnSA1KQb9E6ORnnMFF4tKERUWIi3h1stwRISIiPyKfuoCgNU8CmtuvpSL7HmDRUFI53Gf47fu9zixh86jDFAgJaEBBre7CSkJDXwuCAEYiBARkR+yNnURUScItYOUFq+Z9Osa/PTJeMNxepNExL24GZfq1veZpbC+iFMzRETkl4ynLrZn5WNDxjlcKS4zO69WZQVOLLgfSkFraBt37zRsbX274dhXlsL6IgYiRETkt5QBCqhLyvDJ3lyL+SK3XsjG1lUTRG0dnl+DK7VVAHRTO9E+tBTWFzEQISIiv2VrX5bpP6/EmN/WG453x7XHk0PfNBz74lJYX8RAhIiI/Jal4mZBFeU4ueA+UdvmWe/j7cCbAaNzo+1VWiWnYCBCREReo1IrOHU5qmmSadvzJ7Dx8ynithe+wBv33o49bRr7/FJYX8RAhIiIvEJaZh5mbc4SjWDY3f/FDuMk05nbl2Pkwc2G459adsUz979iOE+/FJbci4EIERF5XFpmHsauPmSWy5GvLsXY1YccrhbaJT4CcbUV+HnmAFH7iAdfx88JnZiM6gVYR4SIiDzKVkKpvm3W5qyqTehk3PfYNz+YBSG3TfwKPyd0MhwzGdWzXBqIzJ49G507d0ZYWBiioqIwZMgQnDhxwpWPJCIiH2Nvt1wBQJ66FOk5VyTfMy0zD5u7DkTS0KogZGPrOxA3bQuKgusAAAIUwDM945mM6mEuDUR++eUXjBs3Dvv378e2bdtQXl6OO++8E8XFxa58LBER+RCpVUulnrctPRuptzXGkAPfG9oeG/pfTLh3qug8QQA+3J2DtMw86Z0lp3NpjkhaWproeNWqVYiKisLBgwfRs2dPVz6aiIh8hNSqpVLOq9yxE/379RW1JU76GjeCQs3OFaCrFTJrcxb6J0ZzesZD3JojolarAQAREZaTgjQaDQoLC0U/RETk3/S75VoLAxTQrZ6xm1D6xBNQGgUh627rh7hpWywGIXqOTPuQc7lt1YxWq8XEiRPRvXt3JCUlWTxn9uzZmDVrlru6RERELiKnHoh+t9yxqw9BAYiSViVVNy0sBFQqUdNDj83BgVjL7xpLuKmd5ygEQZCXhuygsWPH4ocffsCePXvQpEkTi+doNBpoNBrDcWFhIWJjY6FWqxEeHu6ObhIRUTU5Wg/Eoet+/BFITRU1tZr8DUoD5W1S98Xorqwh4kSFhYVQqVSS3t9uCUTGjx+PjRs3Yvfu3YiPj5d8nZxfhIiIPM9aPRD9WIa9eiCyKqs++CCwvmqvGDz3HCoXL0GPuTuRry61uBzYlL6OyJ5pfZgj4kRy3t8unZoRBAHPP/88NmzYgJ9//llWEEJERL7FXj0QKYmhkqqbXr0KmOYa7t8PJCdDCVid5jHFTe28g0uTVceNG4fVq1dj7dq1CAsLQ35+PvLz81FSUuLKxxIRkQe4oh6Imc2bzYOQkhIgOdlwmJoUg2XDOiBaJZ6eMY01olUhDldsJedx6YjIsmXLAAC9evUSta9cuRIjRoxw5aOJiMjNnF0PxFilVkBhnztR/5ftVY1TpgDz51s8PzUpBv0To0XTPB2b1cfBv69yUzsv4/KpGSIiqhmcWQ/E2I5fM9G3522ob9Q2Ytz7eGTEEKRavcryNI/+2Nm7/JLjuOkdERE5xdXiMgQoAGtbwjiywdwfC1eg7+TRoraW/9mACmUgfnFwMzxX7PJLjuOmd0REVG1pmXkYt/aQ1SBET05iqNCrF9obBSGLU4YibtoWlCsDHd4MT7+qxzSXRb/LL8u9ux9HRIiIqFpsrZbRC1AASx5tL23E4cIFIDpaVGk1deRiHI8Sr7w0Tn6VUgPEGat6yPk4IkJERNVib7UMoJuuqV8n2P7N1q4FoqMNhyW1gtHiP9+ZBSHGpCa/umVVD8nGEREiIqoWp6yWEQTdEtwDBwxNZya/hJ6B3ezeV2ryqytX9ZDjGIgQEVG1VHu1zLlzgOnWH1lZuOmWVoixUSVVbvKrq1b1UPVwaoaIiKqlWrvnrlwpDkIaNAAqKoDWrQ2b4envYXpPQF7yq9N2+SWnYiBCRETV4lDAIAhAUhLw1FNVbfPmAZcvA0qloclalVRHqqI6O7Ah53Db7ruO4KZ3RES+Q3J9jjNngGbNxBefOgW0aGH13s4sQMY6Iq7ndbvvOoqBCBGRb7EbMHzwATB2bNVx06ZATg4Q4N4BelZWdS2v2X2XiIj8k/GLvGGdYEABXL6usf5S12qBli2Bv/6qalu8GBg/3r0d/5ekXX7JLRiIEBGRLJamNoyZTXP89ReQkCA659DPh3BWFYWo7AKORtRwnJohIvIzrpx20JdIt/Xi0D9p2bAOSN3+FTBpkuGz63EJ6D/6A+QVagxtzM/wP5yaISLyI3ICC1cmYkop5Q7oKpQqtZXonHIrcP2qoT3z9XkYVNIaglEQAlTt8+LIBnbk+xiIEBF5MTmBhbXRCme96KWUcgeA5gX/YOeKMaK2yr/PYPSakxBKzK/nPi81G+uIEBF5KTk7xdrb0A3QvejLKrTYl12AjRnnsC+7QNbOtVJKn4/Z/40oCLnWKgnQapFeXpv7vJBFHBEhIvJCcneKlbqhW9fZO3CluMzQLmfaxlbpc6W2EhnvPoKwshJD26QBk/Hwuy8hRaHgPi9kFUdEiIi8kNydYqW+wI2DEMDy6Io11kqkt7z0N7LnDRYFIV3GfYb93e8xlEvnPi9kDQMRIiIvJHcEwdEXuPG0jb1pGksl0ifsWYttn4wznHPgpkTEv7gZl+pGiMqlc58XsoaBCBGRF5I7gtAlPgLR4Y4HI1LzM/R7v9xUtxZOzRuMSXvXGj4bf++LeGjYO4iuF2qWGMt9Xsga5ogQEXkh/QhCvrrUYp6IArqN3/QjCNuy8lFaUVmtZ0odhUmtuIDUV1NFbb/tP4b+wWF43MbyYn0QY7oKKJp1RGo0BiJERF5IP4IwdvUhKABRMGI6giClyJgUkkZhZswA5swxHAp9+2H/8i8lF09LTYpB/8Ro7vNCBgxEiIi8lJQRBKlFxuyxm5+h0QAh4kDl0KKPMe5GM+R9tF90H3ujG9znhYyxxDsRkZezVVl1X3YBHjUKBBz1QIfGiA4PhUIBpDRviK4JDapGKQ4cALp0EZ2/Y89RPL05xywAEpV351RLjcUS70REfsTWCIKz6m6sP3Te8N9LdmWjXu1AzLn/NqSumAu8+27Viffei8oN3+GVuTsl1zghsoWBCBGRD3NV3Y0S9XWk3tZY3Lh1K3DPPUjPLpBc44RTMGQPAxEiIh9mb3WNnmnCqy0d/jmGb9dMFTdeuwaoVADk1zghsoV1RIiIfJi9+hwKAM/2jEcjiTVG3k5bIgpCNre6HftOXzYEIYD0UZjLRRpZe9lQzcRAhIjIx+lX10SrxAFCtCoEy4Z1QPum9WFvPCS0rBS5cwfiscNphrbHh/4Xzw+eZjayYa9Kqt6bW4+hx9ydksrHU83FVTNERH7C0uqabVn5dmuMdD1zBF9+8ZKo7daJ61AcXBsA8MXorma5HvraJYDtEIeraGomOe9vjogQEfkJ/eqawe1uMgQO9mqMLNj6f6Ig5OukfoibtsUQhESHB1usL2JtFMaUnL1sqGZisioRkZ+ytYNvHc0NHF30sKjt4cfmID02SdT22kDr+7+kJsWgT6tGeHPLUXy+/4zVfnAVDdnCQISIqBpsFRvz9DOtrVrpkfMHVq97VdTWavI3KA00H92oXyfY6jO3ZeWbVX21hatoyBIGIkREDkrLzDN7EUspce6uZ1pa3bL0u9kYcGKv4fiz9gPw2p1jrT7vYlGpxWfWqx2IazfKZfXdVTVPyLcxR4SIyAH6ZE3T0YB8dSnGrj7kkpUicp/ZJT4CEXUCAQDhpdeRO3egKAi5b9h8m0EIAOReLrb4TDlBiAIS9rKhGouBCBGRTLY2mnNVcqajz3yyazP0OZ2OI+8+Imq/Zcq3+OOmVlafpw8evkg/U60N9Ux3CiYyxUCEiEgmW0mggDg501PPTMvMQ4+5O9FmzDB8sv4Nw3kfdR6CuGlboKkVZPVe+nDhkc5NkV+oqVa/9bVMuHSXrGGOCBGRTJ4ocS7nmWmZeZjx0c/4473HRJ8NHL4ImdEt7N4j+t+cE02F1qG+6r06oDVGdI/nSAjZxECEiEgmqUmX1U3ONF6p8uvJS5KuaVg3GFtmLsYfa98Qtbf8zwaUKwPtXh9RJxC/TO2NoFoB2Jdd4FC/FdAFMwxCSAoGIkREMtnbaE7/Iq5OcqallSr2xKhCcOsT96H7/j2GtiUpD2N+zycl3+NKcTkO/n0VKQkNJG+oZ4w5ISQXc0SIiGSyt9EcUL0X8fdH8jDGwkoVWxoUX8O+l/qhnlEQkjpysawgRE8/DSTl96xXWzzKwpwQkosjIkREDtCXODcdtYiuZh2R74+cx/gv/pB1zb1Zv+C9zfMMx6W1gpA0cR0qlI79FW88pWTv9+yfGO32gm7kX7jpHRFRNTizsmpapm4kRDJBwIbP/4P2eScMTdo33kAHbTKulcgrNgZUTSntmdbH7HfwRAVZ8l1y3t8cESEiqgb9RnPVpa8TIlVUUQHS3x8uvkfmUShvTcTI7aewcPtJWc+3N6XkrN+TyBRzRIiI3KRSK2BfdgE2ZpzDvuwCUfExe3VCjD3w5w5REHI1JAxpGWehvFWXzzG+Twuz3A1TprEGczvIUzgiQkTkBvb2iJFUJ0QQ8MPK59H6Uq6haUnqaLSYN0sUQCgDFJhz/20Wp3n08ceSR9ujfp1gTrWQx7llRGTp0qWIi4tDSEgIkpOTkZ6e7o7HEhG5nK1RDj0pe8TYqzkSU3gJue8MEgUhf2z/DWO3Lrc4ipGaFIMPhnVAjEp8X/3Ixz1tGiMloQEGt7sJKQkNGISQx7h8ROSrr77C5MmT8cEHHyA5ORmLFi3CXXfdhRMnTiAqKsrVjycichkpO+Ha2yNGAd0eMb9M7W21ZsdjGT/g7R+XVjXExgK5uWgfYPnfkvrEUk2FFvMfbAsogMvXNRz5IK/k8lUzycnJ6Ny5M5YsWQIA0Gq1iI2NxfPPP4/p06fbvJarZojIW+lHOUz/AtW/4vX5FvuyC/DoR/vt3u+L0V2hLinD2H+nUwQACkGLXR8+i7hrVbvqat99FwEvvGCzX/aCIyJXk/P+dunUTFlZGQ4ePIh+/fpVPTAgAP369cO+fftc+WgiIpeRsxOunD1i9DU7olUhiL2Wj5x37hUFIT2eXYFu1xPx7vaTFqeCpEwBEXkbl07NXL58GZWVlWjUqJGovVGjRjh+/LjZ+RqNBhpN1U6PhYWFruweEZFD5OyEK3dfmtSkGNy5Yx0Clk80fJYdcRP6Pv0BoFAAhRos3H7K8FmMUWExKVNA/ROjOTVDXsWrlu/Onj0bKpXK8BMbG+vpLhERmZEzyqHfr8Xaq18BXTDRJT4C0GqBxo0RMHGi4fPpd41H39HLdUGIBfrRjiU7T0sOjoi8iUsDkYYNG0KpVOLChQui9gsXLiA6Otrs/BkzZkCtVht+zp4968ruERE5RM4oh+R9aU6fApRKIK9q+iRl7Ep82S7V5jP0IyAr/5cjqU9Sgygid3FpIBIUFISOHTtix44dhjatVosdO3YgJSXF7Pzg4GCEh4eLfoiIvI29UQ5AtxmcViugUiuIcj+MGYqIff85cMsthvZrt9yKuBc3Iy88UlJ/BADXbkgr6S41iCJyF5cv3508eTKGDx+OTp06oUuXLli0aBGKi4sxcuRIVz+aiMgl9KMcY1cfggKwmJdx7UY5Hv/4N9GKFbMN4pqqoIyKBK5dq7rw009xrPsAQMJKG1P1QgOhLim32B/9PjJd4iNk35fIlVyeIzJ06FDMnz8fr732Gtq1a4eMjAykpaWZJbASEfkSa6McpoxXrOj3axnc7iakaC5AGRQoDkLOnweefFLSiIslI7vHAbAzBcREVfIy3H2XiGoEV+0eW6kVsP+vAoxbc8jqjrdmu9q++Sbw2mtVJ3TrBuzZI0pI1S/FBSyPuFi7/7asfNYRIY/j7rtEREZcWeRLGaBAgEJhNQgBqlasHDh5AV3bNAXKjc794gvgkUfMrtGPuJj225TpaIfFKSBWUyUvxkCEiPxWpVbAkp2nRHU39PRTJvZ2nJUykiJlJUrri3+ha+uB4sYLFwAbW12YBhW5l2/gi/QzyC+sel60hYBKPwVE5AsYiBCRX0rLzMPrm7JEL21jUop8SR1JsbcSZeovn2Lc/q+rGvr2BbZvl/R7mAYV4/u04GgH+RUGIkTkd6ztA2PKuMiX6QiCtXtYGknRJ5eablgXVFGOkwvuE9/g22+B+0zaZOBoB/kbr6qsSkRUXbb2gbHGdGpFzl4yACwWLbst75R5EHL5crWCECJ/xECEiPyKvX1gLDGdWpGzl4xealIMlj7WHvXrBOKVHR9h82eTDJ9d7NUfEASgAUcyiExxaoaI/IqcEubWinzJ2UtGLy0zD3O/O4xDM+8RnXNg8WfoPP4JyX0iqmk4IkJEfkVuCXNLRb7k7piblpmHD+euwS8mQUibCV/i4X8ikJaZZ+lyIgIDESLyM1KrkkaHB1tduqu/hy0RdQKRX1iKvacu48aoZ/Dt6qmGz7a0uh1x07agMKQuAHE+iRSVWgH7sguwMeMc9mUXyLqWyNdwaoaI/IrxPjDWDGwTg3cfaW82EmJcM6Rjs/rYcsT6SMaV4nK89Pl+HFv4oKh92MNvYk98e8OxrZU5ljhafM1VlWOJXI2BCBH5ndSkGDzTMx7Ld+dY/HzrkTwMbBMjerFbCgBsST7zJ776Yoao7daJ61AcXNvi+dbqmRiTs2TY9DqWdSdfxakZIvI7lVoBmw7bzsswni7RBwBSg5D5WxeKgpBvkvoibtoWq0EIALy55ajNXBG5S4b1rPXdeLM9Im/GQISI/I6c5bdy6o7U0dxA7tyBeDBzh6Ft6KOz8Z8Bk2xcpXOluNxmYODIkmFHgxcib8JAhIj8jpzlt1LrjvTI+QNHFz0sams96Rv81vQ2WX2zFhg4smTYkeCFyNswR4SI/I7U5beXizTQShgtWLJxLgYe/9VwvLrd3XjlrnGy+2UrcVXukmHAseCFyNswECEiv2Nt7xdTb249hog6QVY/D9MU489FQ0Vt9z8+D4eatDY7VwEgPLQW1CUVdvtnKTCw12dLxdccCV6IvA2nZojI71ja+8Waq8VlFtt7Zx8wC0JumfKtxSBE76nuzSX1z1JgYKvP+mPT4mv2aqYooFs9Y1o5lsibMBAhIr+UmhSDZcM6INpOYTJLow8frX8DK7+ZZThe0Wkw4qZtgaaW5dGTAAWw9LEOGN+nRbUCA2t9jlaFWFy660jwQuRtFIIgeG06dWFhIVQqFdRqNcLDwz3dHSLyQZVaAav25uDNrcfsnhtSXor/LXsKESWFhrZBTy7EnzEt7V77xeiuSEloYFhOC4iDHH0oYK0WiGmf5RQnYx0R8jZy3t/MESEiv6YMUKBhWLDd8xIun8XSjXNEQUjL/2xAuTJQ0nO2ZeUjJaGBYVTDNDCIlhEYKAMUkqqw6qUmxaB/YjQrq5JPYiBCRH7PXrLmA3/uwJvb3kftcg0u1amHKfdMwu7mHWU947uM83h5gG4axBOBgdzghchbMBAhIr9nbUVKaFkp3ty2zFCgbE+ztpg08D+4VLe+7GdcKS7Dkp2nMKHfzQAYGBBJxWRVIvJ7lpI6b76Ui82fTsSDmTtQqQjA0t5PYvjDbzgUhOgt3H6KJdWJZGIgQkQ1gmFFSngwhh7+EZs+m4wWV/7BxbAIbJj3GeIXzYE2QGl3ua89LKlOJA+nZoioxkhtVhd3Zq5EQNpaAMAv8R0wecBkFFyqh5itWXimZzw2Hc4TJZkGKAA5cYW1yqlEZBkDESLyOnKXr0qSkQEMHYqAkydRoQjAgp5P4IPkByAodAPD+epSfLg7B0sfa4/6dYJxsagUl4s0kpb9mmJJdSLpGIgQkVfQBx/bsvLxXcZ5XDGqeFqtmhiCAHzwATBpEqDR4IIqEs8NnIqDTRLFp0GXP/Lm1mPYM60PlAEKbMw459DvwpLqRNIxECEij7NUkMtYvroUY1cfklQMTEStBp55Bli3DgBwpfeduOu2EbgWarnAkn5TulV7c9AwLBiXizSyfg/j/WBcMqpD5IcYiBCRR+krkdpKw9B/NmtzFvonRkt7of/+OzB0KPDXX0CtWsCcOfi1z1Bc++qw3UsdmY4xLqm+LSuflU6JJOKqGSLymEqtgFmbs2wGIcby1KVYsvOU7ZMEAXjvPaBbN10Q0qwZ8OuvwJQpiAoPrXafrdHvBwMAY1cfMhvd0Y/qcHkvkRgDESLymPScK1anY6yxWavj6lXggQeACROA8nJgyBDgjz+Arl0B2N+t1lHjeydgz7Q+6J8YbTWwMh7V4fJeoioMRIjIYxxdXWLxZf7bb0D79sCGDUBgIPDuu8C33wL1qwqU2dqttjq6t4iEMkBhN7DS56Ck51xx4tOJfBsDESLyGEdXl4he5oIALFgA9OgB/P030Lw58L//oXL889j31xVszDiHfdkFhsDFUNhMVf2VLQrocj+6xEcAkB5YcXkvURUmqxKRx1jbA0aKi0WlQEEBMGIEsGWLrvHBB4EVK5B29gZmzd1pNVnUdFM6R+qFGCen6pNnpQZWXN5LVIUjIkTkMfqpEkcyJhJOHtZNxWzZAgQHA++/D6xbh7SzNyQli+o3pRvc7iaM6B5vN3fEdKGOKjQQE/u1RP/EaEObvRwU0xEUImIgQkReoG6w9MHZAEGLFzM24NbH7gXOngVatgT27wfGjkWlAIeSRW3ljij+/VnyaAdM6ncz6oUGAgCulZRj4fZT6DF3pyi4sXUfQDyCQkQMRIjIg74/ch5jVh/CdU2FpPMjbqjxydez8NyPH0NRWQk89hhw8CDQrh0A+6twbCWLWssd0S/LDQgAFm0/iWsl5aLPTUda7N2HdUSIxJgjQkQe8f2RPIz/4g/J5yef+ROLt8xHVFEBEBICLF4MjBoFKKpGF6qbLGqaO6KviAoAPebutDrSooC42Jq1+3AkhMgcAxEicru0zDw8t/aQpHMDtJUYt28dJu79AkpBC6FVKxxZ8CFyGzdH1F9XRC94ZySL6nNHjO3LLpA80qK/1tJ9iMgcAxEicit9NVUpIq9fxcIt89Hjb11Z9nP3PoxhHZ5Azu5CABkAxKth7K3CMd4LRg4uyyVyHeaIEJFbSa2m2i03A9+veh49/j6MitBQHPnvIvRo/SRySsTTG8Y5Gq5KFuWyXCLXYSBCRG5lb9QgsLIcm1dNwOqvXkVk8TVob70VivQDeDYgSdJqGFcki3JZLpHrcGqGiNzK1qhBUv5pbPl0ouE48+6H0HrdSqRfKEWeOtfqdaY5Gs5OFtWPtIxdfQgKQBQQcVkuUfUwECEit7KWx/HyzhUYfeA7w/FPLbvimTbDEbN4P+5Jija7jyXGoy3OThbVj7TM2pwlmlqKNspRISL5GIgQkSSVWsEpIwymowtBFWU4seB+0TlPPfAadrboAkCXA/Lx3lxJ93Z1jgaX5RI5HwMRIrIrLTPPbCQgphojAfrRhW+WfoMVH7wg+qzNhC9RGFLXcKwfNQlQAKYb7hpzV44Gl+USOReTVYnIprTMPEl7t8iVuuy/oiBk6y3dETdtiygIMWYrCAGAe9vGcGSCyAe5LBDJzc3FqFGjEB8fj9DQUCQkJGDmzJkoKytz1SOJyMn0NT/k7t1i040bumqo779vaPrf+2swbsiMavV10+E8ef0gIq/gsqmZ48ePQ6vVYvny5WjRogUyMzMxevRoFBcXY/78+a56LBE5kdS9W1btzUHDsGD7ORO7dwN33CFuKyyE4mIZ8NH+avXVtLIpEfkGlwUiqampSE1NNRw3b94cJ06cwLJlyxiIEPmI7Vn5ks57c+sxw39bzR0ZORJYtarq+MkngU8/BQB0qSPYrIgqFSubEvket+aIqNVqRERYTybTaDQoLCwU/RCRZ6Rl5klerWLMLHekqEg3FWMchOzaZQhCAEiqiCoFK5sS+R63BSKnT5/G4sWL8eyzz1o9Z/bs2VCpVIaf2NhYd3WPiIzI2Q/GlCh35KdtQHi4+ITr14Fevcyus1UR9f3H2rOyKZGfUgiCIGskdPr06Zg7d67Nc44dO4ZWrVoZjs+dO4c77rgDvXr1wooVK6xep9FooNFoDMeFhYWIjY2FWq1GuOlfZkTkMvuyC/BoNXM23tv0Du49truq4ZlngOXL7V5nrV6JfvUOYLmyqaPl24nI+QoLC6FSqSS9v2UHIpcuXUJBQYHNc5o3b46goCAAwPnz59GrVy907doVq1atQkCA9EEYOb8IETnPxoxzmPBlhkPXhmmK8eeioeLGPXuA7t2r3S9n1zMhIteQ8/6WnawaGRmJyMhISeeeO3cOvXv3RseOHbFy5UpZQQgReY6juRa9sg9g1TezRG2V14uRnl+Cixnnql2JlJVNifyPy1bNnDt3Dr169UKzZs0wf/58XLp0yfBZdLS0fSOIyDOs7Qdjy4ff/hd3nqqazvmq2/0IW7YEb763z6kjGKxsSuRfZE/NSLVq1SqMHDnS4mdSH8mpGSLP0edk2Pt/q6qkCIffe1TUNvjJ/0PXoXfhw905Ztczp4PI/8l5f7tsrmTEiBEQBMHiDxF5v9SkGLzQt6XNc+48uc8sCOn5xvd4dspQbDqc59yKrETkl5i0QUQWpWXm4dP/5Vr9/PMvX8GHG94yHJ8cOQ77Tl/GrpdTUb9OsKSKrOk5V5zYYyLyRdx9l4jM2JqWibihxqHFj4vaDm/ahbaDehmOpVY4ZSVUIuKICBGJ2NrobuCx3aIgpCygFnq8mYakAeL9Y6SuumElVCLiiAgRiVjc6E4Q8M2aF9HpXNWeMgu7P4b3ejyGZUPamC2flbLqpkGdIHRsVt/JvSciX8MRESISMd3oLvL6FeS+M0gUhPR/aik+7T/c6soXZYACrw5obXPFTUFxGe6Yt6tqTxoiqpEYiBD5gUqtgH3ZBdiYcQ77sgscXo1SqRWwIeOc4fj+zB04sPRJw3FhUG0kTN2IU5HNsPRx68tv0zLzRDvyWmO2QR4R1TicmiHycc4se56ecwVXissBQcCWTyci6UK24bO5dwzHsq4PAdBNq3RtbrmomNT6I4Bu9YwCuqW8/ROjWSGVqAZiIELkw6y99PUjDXKLhl0sKkV04WXsXzZC1N7n6Q/wV4MmhuPB7RpbDBpsJbpaY7yUt0t8BMu3E9UwDESIfJStl76jIw1JW9dh/7LJhuOLdeqj63OroA1Qis7rn2h5mwaLia4Sbc/Kx+R1GdzQjqiGYY4IkY/R54Ms3HbCeUXDBAG45RYkvFIVhLzRZzS6jP/cLAgJUMDqapfq1AX5eG+u2e/DHBIi/8cRESIvVqkVRFMVV4s1eHPrMVmjDnaDg5wcoHlzUdPtz67A2XqWRz20AnDw76sWN55zpC6IAoBCobuvKeaQEPk/BiJEXspSEqojbAYHS5YAzz9vOCxu0gxJjy2GoLA9WGotuJG7a68CumDD1hZUxiM73HWXyP9waobIC+mTUKsbhMSodAmfZrRaIDZWFIRg2TIc+fmg3SAEsB7cKAMUmDkoEUDVLru2RKtC8FT3OAlnshw8kb/iiAiRl3Fk5Yk1rw5obT6dcfo00NJkV92//waaNkUXrWBzREMBXfBgMbj5V2pSDJYN62BxSfGrAxJRv06QaFVMes4VfLI31+7vwnLwRP6JgQiRl6nOyhNT9esEixvmzwemTq06TkoCjhzRJWmgakRj7OpDhmkTYwKAVwck2s3VSE2KQf/EaElLce1N50gJfojId3FqhsjLOHMKwnCvykqgQQNxEPLJJ8CffxqCED39iEa0yvIIxJtbsyStYlEGKJCS0ACD292ElIQGVoMXW9M5+uOZg+wHP0TkmxiIEHkZZ05BXC7SoPJoFlCrFnDFaBnvuXPAyJFWr0tNisGrA1pb/MwVS2qtBT/RqhDZRdmIyLdwaobIy8hdeWLLlZdfh/LXz41u3gXYv98wCmK6PFg/fVKpFazuFeOqJbVypnOIyH8wECHyMvbyNCTdQ1uJzIUPI7RCY2g7PGcJ2k4bZzi2tUeNKjRIcrE0Zy6p1U/nEFHNwakZIi9kbaqifu1Au9e2upiD7HmDRUFI5/GfY4y2FcoqtNiXXYA3Nh/FGAvLg/XTLtuy8iX1k0tqiai6OCJC5KVMpyoa1g3GlHUZNq+ZvPtzvLDvK8Pxvqa34dFHZ+sO1KXoOnu7bnddK/TTLhszzkvqI5fUElF1MRAh8mLGUxX7sguQX6ixeF5gZTlOzb9P1DZmyAyk3dJd1GYrCNETABQUlyGiTiCuFpdzSS0RuRSnZoh8hLVpkFvzT5sFIe2fX2MWhMh1X7ubAHBJLRG5FgMRIh9haRrkpZ0fY+unEw3Hu5p3RNy0LbhaW1Xt5/VLjOaSWiJyOU7NELmYtSWychkv6w2qKMOJBfeLPh/1wKvY0SLZ4ZU2esbTLsoABZfUEpFLMRAhciFbS2Tljijol/Uun7sWG1b/R/RZ2wlfojCkLp7tGY9Nh/NEz2tQJwgFxWWSnmFp2oVLaonIlRSCYGsDbs8qLCyESqWCWq1GeHi4p7tDJIt+B13T/4PpX/YOTW+MHw8sXWo4/OHmbhh730ui4MZ0BKZjs/q4Y94uSQXSHA2SiIiMyXl/c0SEyAVs7aArpTKp2XROoxAow+qKzsn6ZB3K2nfDFybTJZZGMOwVSBvVPQ79EqM57UJEbsdAhMgF7O2gq69MumpvDkZ0jxe9/E2nczqfzcTXa6eLb6BWIzE8HIkS+6MvkOasaSIiImdhIELkAlIrjr659RhW7MkxBAOm0zlzv38XQ//cZjj/3MAHcNPmbxzqE/dyISJvxECEyAXkVBzVl1Vf+lh7vLn1GAQAtctKkLXwIdF5jz7yNnLbdMEereBw8MDEUyLyNqwjQuQC+qW2UsIF/ejHKxszkacuRbfcDLMgJHHS19jXrI1hozkiIn/BQITIBfRLbQHzyqSWCNCVX3930zys/eoVQ/vatnchbtoW3AgKNbRxozki8iecmiFyEWsJopaEaYrx56KhorYHHn8HB5uYp6Nyozki8icMRIhcSJ8gumpvDt7cesziOb2yf8eqb14Xtd0yeT00gcGiNlsbzTmreisRkbsxECFyMWWAAiO6x2PFnhyzomIffvtf3Hlqv+F4ZcdBmNXvWbN72NpozpnVW4mI3I2VVYncxHhprqqkCIffe1T0+eAnFuBw41sAAAEKQGv0/0xrgYUj1Vs5ekJErsbKqkTV5IqXtT5n5Me3l2PhF7NEn908ZQPKagUajrUC8OqA1mgYFmz1+Y5Ub+XoCRF5GwYiVGNIDS6c9bK29LzUKSOQ+tNPhnM+SH4Ac3qNtHh9w7BgDG53k9X7S63emp5zBSkJDayOnujrmDi09w0RUTUxEKEaQWpw4ayXtenzIm6ocWjx46JzBox4F0cbJVi9R8O6wVY/A6Qv471YVFrtvW+IiFyFdUTI7+mDC9PRA31wkZaZB8D+VAege1lXam2nVZk+757je0RBiDZAiZb/2WAzCBE91Aqpy3ijwkJkjZ4QEbkTAxHya3KCC2e8rEXPEwSsW/Mi3t84x/D5ou6PosPM71GuDLR6D73LxRqbn9ur3qqAbtSnS3yErNETIiJ34tQM+TU5wYUzXtb650Vev4oDS58QfXbnU0twMjIOuFEu6Tn2pmb01VvHrj4EBcQDKKbLfeWMnhARuRNHRMivyQkunPGyvlhUivsyd4qCkOtBoUiYulEXhPyrXmig3dLvU9ZlGKaNrNGvxIlWifsUrQoR5bPIGT0hInInjoiQX5MTXOhf1qZFx/RsVTYFAAgC+g67B4OPHjE0vdPzSbyf8rDZqSO7x2HR9lNmIxnGLhRqJCXI6qu32loRJGf0hIjInTgiQn5NzkiArY3q7L6s//kHCAhAXaMgpM/TH5gFIfrnje/TEsuGdUCjcOuBkpwEWWWAAikJDTC43U1ISWhgsY9SR0+IiNzJLYGIRqNBu3btoFAokJGR4Y5HEgGwvQuupeDCoZf1xx8DsbGGQ02DSCRM3YicBk1sPi81KQYLHmprs//OXs2SmhSDPdP64IvRXfHuI+3wxeiu2DOtD4MQIvIYt0zNvPjii2jcuDEOHz7sjscRiVjbBTfaSpEyKVMdAABBABITgePHq9r+7/8QPGkSllqoW2LpefZWxug5czWLfvSEiMgbuDwQ+eGHH/DTTz9h/fr1+OGHH1z9OCKLJAcX/7L3sq7MyYWyeby48fRpICFB1vO4moWIajqXBiIXLlzA6NGj8d1336F27dp2z9doNNBoqv6FWFhY6MruUQ3jrJGArJffRuLbLxuOz6ga4ZGpn+O1ktpIlfm8aifIEhH5OJfliAiCgBEjRmDMmDHo1KmTpGtmz54NlUpl+Ik1mncn8jitFiWNm4iCkFfufA49x3yMvKIyUZVWqaqVIEtE5AdkByLTp0+HQqGw+XP8+HEsXrwYRUVFmDFjhuR7z5gxA2q12vBz9uxZud0jco3TpwGlEqF55wxN3cd8gtXt7wEgb4WLKa5mIaKaTCEIgqy/NS9duoSCggKb5zRv3hwPP/wwNm/eDIWi6l9ylZWVUCqVePzxx/Hpp5/afVZhYSFUKhXUajXCw8PldJPIabTz5yNg6lTD8ckGTXHnqKWAwvIoxRejuzo0BSR1d2AiIm8n5/0tOxCR6syZM6Icj/Pnz+Ouu+7CN998g+TkZDRp0sTG1ToMRMijKitRFhmFoKtVS2en3v0Cvm5zp83L3n2kHQa2acyggohqLDnvb5clqzZt2lR0XLduXQBAQkKCpCCEyKOOHwdat0aQUVPyc6twIayh3UtzLxejx9ydoqW7MVaWChMR1XSsrEpk6u23gdatDYeHo1si7sXNdoMQBYD6tQOxcPsps4328tWlDiWzEhH5O7ftNRMXFwcXzQIRSWYzD6OiAlCpgBs3DOdPGDgFG2/tbfe++kkXa3/ChX/PmbU5C/0TozlNQ0T0L256RzVGmoVqp4YpE+Ey0KaN6PxO4z/H5Tr1Jd07WhWCRzo3xcLtJ62eY1yunZVNiYh0ODVDNUJaZh7Grj5kccrk5LOTxUFIz57Yd+qSpCBkfO8Ew34tcQ3tF+0DnFuunYjI13FEhPxepVbArM1ZZtMmgZXlODX/PnHj118DDz6ILlpBUsXTSf1vMUyzsFw7EZF8HBEhv5eec8VsJOTW/NNmQciB9BPAgw8CcKziqb5cu7XsDwV0U0Es105EVIWBCPk906mQGbs+wdZPJxqOf4nvgLhpW3A+sI7oPLkVT1munYhIPk7NkN/TT4UEV5ThxIL7RZ+Nvv8VbGvZVXSeMbm79uqDF9Ok2GjWESEisoiBCPm9LvER6FuYg4+XPS9qbzPhSxSG1LW6w63pUt+BbRpLGs2QG7wQEdVkDETI7yknvICPly0xHKfdnIIx9+l20LU2ZWJzqa+EUQ1lgIJLdImIJGAgQv7rxg2gjjjvY9KIt7GhUdVSXUtTJvqlvqarZfTVUbkjLhGR8zAQIf+0Zw9w++3iNrUa8+uG4WEbUybWlvoCrI5KROQKXDVD/mf0aHEQ8thjgCAA4eGGKZPB7W5CSkIDs2DC0lJfY8bVUYmIqPo4IkL+4/p1ICxM3LZjB9Cnj+RbSK16yuqoRETOwRER8g87d5oHIUVFsoIQAGhYN1jSeayOSkTkHAxEyPc9/jjQt2/V8ahRuqmYunVl3SYtMw9T1mXYPIfVUYmInItTM+S71GqgXj1x2+7d5kmqElhbKWOM1VGJiJyPgQj5prQ04O67xW03bgChoaIm06JklgqL2VopY4zVUYmInI+BCPme++8HNmyoOh4/Hli82Ow0qUXJ7K2U0Zv/YFt0b9mwen0nIiIRBiLkO65eBSJMcjP27weSk81OlVOUTOoKmMvFGkd6TURENjBZlXzDpk3mQUhJicUgxF5RMkBXlKxSqzuSugKGK2WIiJyPgQh5v7vvBgYPrjqeOlW3KibEcmAgtyhZl/gIxKhCYC39lCtliIhch4EIea/LlwGFQpeY+q8j3+3AxscmYF92gWFEw5TcomTKAAVmDkoEALNghCtliIhcizki5J2+/hp4+GHDoaBQoOes73F2XwmwLwOA9d1wHZlqSU2KwbJhHcySW7lShojItRiIkFvZXU4rCMAddwC//mpoOv3sRPSv1w9CcaXoXtZ2w9VPteSrSy3miSigCzBMp1pSk2LQPzHa7nJfIiJyHgYi5DZ2l9Pm5wMx4pGHyozDeOL7SxAs5HxY2w1XP9UydvUhKABRMGJvqkW/KR4REbkHc0TILfTLaU2TSPWjGodnLxEHIXXrAuXlSK97k0O74eqnWqJV4mmaaFWI2QgKERF5DkdEyOVsLqcVBGz6bBLa5J+uanzrLeCllwBUbzdcTrUQEXk/BiLkctaW0zYquozf3h8hbjx2DGjVynBY3RofnGohIvJunJohl7M0WvHQkZ9EQcjl2ips/P1vURACsMYHEZG/YyBCLicarRAEbFsxFvN+eM/Q9GbvUej0/BpE1atjdi1rfBAR+TcGIuRy+lGNJuqLyH1nEFoWnDV8dsczH+KTLvfZHNVg4ikRkf9ijgi5nDJAgRXF6bj1g5cMbf+ER+H2MSsAhS4WtjeqwcRTIiL/xECEXEurBZo3x61//21oeqX/WKzuMACA9eqoljDxlIjI/zAQIdfJzgZatBA1VWb/hQFCODpzVIOIiMBAxG/ZLaXuagsXApMnVx23bg0cPQqlQoEU9/WCiIi8HAMRP2S3lDpcGKhUVgKNGwMXL1a1ffQR8PTT1b83ERH5HQYifkZfSt20iqnxBnEA7AYqDjl+XDfyYezsWaBJE8fvSUREfo3Ld/2IzVLq//7vjG//xBgbe76kZeY59vDZs8VBSMeOukRVBiFERGQDAxE/Yq2Uup4A4OqNcqufAbqRkkqtpVDGiooK3QZ1L1UtzcXnnwO//w4omIRKRES2MRDxI1I3iLPG2k62VmVmAoGBQHFxVVteHjBsWLX6QURENQcDET8idYM4eyQFNDNnArfdVnV8++26qZjoaKf0gYiIagYmq/oRfSn1fHWpxTwRqWwGNOXlQEiILujQW7cOeOihajyRiIhqKo6I+BF7G8QpANSrHej4TrZ//AEEBYmDkIsXGYQQEZHDGIj4GXsbxM25XzedInsn2xdfBDp0qDru3x8QBCAy0nmdJyKiGkchCEJ1RvFdqrCwECqVCmq1GuHh4Z7ujk+xVbBMSsEzA41GNxVj7LvvgMGDXfwbEBGRr5Lz/mYgUkNJqqyang4kJ4vbCgqACCtTN0RERJD3/mayag1ldyfbF14AFi+uOh48WDcSQkRE5EQuzRHZunUrkpOTERoaivr162PIkCGufBw5Q0mJrhCZcRDy/fcMQoiIyCVcNiKyfv16jB49Gm+//Tb69OmDiooKZGZmuupx5Ax79wI9eojbrl0DVCqPdIeIiPyfSwKRiooKTJgwAfPmzcOoUaMM7YmJia54HDnD6NHAihVVx48+Cqxd67n+EBFRjeCSQOTQoUM4d+4cAgIC0L59e+Tn56Ndu3aYN28ekpKSrF6n0Wig0WgMx4WFha7oHhkrLtbtFWNs+3agb1/P9IeIiGoUl+SI/PXXXwCA119/Ha+88gq2bNmC+vXro1evXrhyxfo+JrNnz4ZKpTL8xMbGuqJ7pLdrl3kQUlTEIISIiNxGViAyffp0KBQKmz/Hjx+H9t/Kmy+//DIeeOABdOzYEStXroRCocDXX39t9f4zZsyAWq02/Jw9e7Z6vx1ZN2wY0KdP1fFTT+kKlJkGJkRERC4ka2pmypQpGDFihM1zmjdvjry8PADinJDg4GA0b94cZ86csXptcHAwgoOD5XSJ5CosNE8+3b1bt2kdERGRm8kKRCIjIxEpoaR3x44dERwcjBMnTqDHv6swysvLkZubi2bNmjnWU6q+H38EUlPFbcXFQO3anukPERHVeC7JEQkPD8eYMWMwc+ZM/PTTTzhx4gTGjh0LAHiIG6R5xgMPiIOQ557TTcUwCCEiIg9yWR2RefPmoVatWnjiiSdQUlKC5ORk7Ny5E/Xr13fVI8mSq1fNS7Lv329eup2IiMgDuNeMP9u8Gbj3XnFbSYn5JnZEREROJOf97dIS7+QelVoB+7ILsDHjHPZlF6BSKwB33y0OQqZM0U3FMAghIiIvwk3vfFxaZh5mbc5CnroUAFD/hhp/LH5cfNLvvwMdO3qgd0RERLYxEPFhaZl5GLv6EPRza3cf34NlG+eIT9JogKAgt/eNiIhICgYiPqpSK2DW5ixDEPLl2unoerZqU8HFKUOxdtBo7KkVCKVnukhERGQXAxEflZ5zBXnqUjQsvorflzwh+ix15GIcj4oH1KVIz7mClIQGHuolERGRbQxEfNTFolLcm/Uz3ts839B2IzAYt01ch8oApeg8IiIib8VAxBcJAvo+ORCD/8wwNM2/fRiWdHvE7NSoMK6SISIi78VAxNecOwc0aQLjren6jlqG7IbinYoVAKJVIegSb1LMjIiIyIuwjogvWbkSaNLEcFhWrz4Spm7EXxaCEACYOSgRygAFiIiIvBUDEV8gCEBSEvDUU1Vt8+Yh6OoVLH2yM6JV4umXaFUIlg3rgNSkGDd3lIiISB5OzXi7M2cA0x2LT54EWrYEAKQmxaB/YjTSc67gYlEposJ00zEcCSEiIl/AQMSbLVum2yVXr2lTICcHCBAPZCkDFFyiS0REPolTM95IqwUSEsRByOLFwN9/mwUhREREvowjIt4mOxto0ULclpMDxMV5pDtERESuxH9ee5NFi8RBSKtWutERBiFEROSnOCLiDSorgcaNgYsXq9o+/BAYPdpzfSIiInIDBiKeduKEbuTD2NmzonohRERE/opTM540Z444CGnfXjcVwyCEiIhqCI6IeEJFBRARARQVVbV99hnwxBPWryEiIvJDDETcLTMTuO02cdv580AMq6ASEVHNw6kZd3r9dXEQ0r27biqGQQgREdVQHBFxh/JyoHZt3ZSM3pdfAkOHeq5PREREXoCBiKtlZOiSUI1dvAhERnqkO0RERN6EUzOuNH26OAjp31+3k67MIKRSK2BfdgE2ZpzDvuwCVGoFJ3eUiIjIMzgi4goaDRASIm7bsAEYMkT2rdIy8zBrcxby1KWGthhVCGYOSkRqEnNLiIjIt3FExNnS082DkIICh4OQsasPiYIQAMhXl2Ls6kNIy8yrRkeJiIg8j4GIM02cCCQnVx3fe69uKiYiQvatKrUCZm3OgqVJGH3brM1ZnKYhIiKfxqkZZygp0a2KMbZ1K3DPPQ7fMj3nitlIiDEBQJ66FOk5V5CS0MDh5xAREXkSA5Hq2rsX6NFD3HbtGqBSVeu2F4usByGOnEdEROSNODWDaqxKefZZcRAydKhuKqaaQQgARIWF2D9JxnlERETeqMaPiDi0KqW4GKhbV9y2bRvQr5/T+tUlPgIxqhDkq0st5okoAESrQtAlXn7+CRERkbeokSMi+hGQNzcfxRi5q1J27TIPQgoLnRqEAIAyQIGZgxIB6IIOY/rjmYMSoQww/ZSIiMh31LhAJC0zDz3m7sSjH+3Hx3tzLZ5jdVXKk08CffpUHY8cqZuKCQtzSV9Tk2KwbFgHRKvE0y/RqhAsG9aBdUSIiMjn1aipGX1dDikZIKJVKZGB5nkfv/wC9Ozpim6KpCbFoH9iNNJzruBiUSmiwnTTMRwJISIif1BjAhFbdTlsEX5MA8YNEzcWF5sv13UhZYCCS3SJiMgv1ZipGXt1OSx5f8Pb6GYchDz3nG4qxo1BCBERkT+rMSMicupthJdex5F3HxE37tsHdO3q5F4RERHVbDUmEJFab6Pv6d/w8fo3xY0lJeb7xxAREVG11ZipGX1dDlspnp98/bo4CJk8WTcVwyCEiIjIJWrMiIi+LsfY1YegAERJq/VLCvHHe4+JL/j9d6BjR3d2kYiIqMapMSMigOW6HKkn9poHIRoNgxAiIiI3qDEjInrGdTlaPDIIkb/vq/rwpZeAt97yXOeIiIhqmBoXiACAEgJSWjQUNx4+DLRp45kOERER1VA1MhDB0aNV/x0SotsrJjDQc/0hIiKqoWpUjohBYiLwySfAd9/pluYyCCEiIvKImjkiolTqNqwjIiIij3LZiMjJkycxePBgNGzYEOHh4ejRowd27drlqscRERGRD3JZIDJw4EBUVFRg586dOHjwINq2bYuBAwciPz/fVY8kIiIiH+OSQOTy5cs4deoUpk+fjjZt2qBly5aYM2cObty4gczMTFc8koiIiHyQSwKRBg0a4JZbbsFnn32G4uJiVFRUYPny5YiKikJHG4XCNBoNCgsLRT9ERETkv1ySrKpQKLB9+3YMGTIEYWFhCAgIQFRUFNLS0lC/fn2r182ePRuzZs1yRZeIiIjIC8kaEZk+fToUCoXNn+PHj0MQBIwbNw5RUVH49ddfkZ6ejiFDhmDQoEHIy8uzev8ZM2ZArVYbfs6ePVvtX5CIiIi8l0IQBMH+aTqXLl1CQUGBzXOaN2+OX3/9FXfeeSeuXr2K8PBww2ctW7bEqFGjMH36dEnPKywshEqlglqtFt2HiIiIvJec97esqZnIyEhERkbaPe/GjRsAgIAA8YBLQEAAtFqtnEcSERGRH3NJsmpKSgrq16+P4cOH4/Dhwzh58iSmTp2KnJwcDBgwwBWPJCIiIh/kkkCkYcOGSEtLw/Xr19GnTx906tQJe/bswcaNG9G2bVtXPJKIiIh8kKwcEXdjjggREZHvkfP+rpmb3hEREZFXYCBCREREHuPVu+/qZ41YYZWIiMh36N/bUrI/vDoQKSoqAgDExsZ6uCdEREQkV1FREVQqlc1zvDpZVavV4vz58wgLC4NCofB0dxxSWFiI2NhYnD17lgm3EvE7k4/fmXz8zuTjdyZfTf3OBEFAUVERGjdubFZTzJRXj4gEBASgSZMmnu6GU4SHh9eoP4TOwO9MPn5n8vE7k4/fmXw18TuzNxKix2RVIiIi8hgGIkREROQxDERcLDg4GDNnzkRwcLCnu+Iz+J3Jx+9MPn5n8vE7k4/fmX1enaxKRERE/o0jIkREROQxDESIiIjIYxiIEBERkccwECEiIiKPYSDiRvfeey+aNm2KkJAQxMTE4IknnsD58+c93S2vlZubi1GjRiE+Ph6hoaFISEjAzJkzUVZW5umuebW33noL3bp1Q+3atVGvXj1Pd8crLV26FHFxcQgJCUFycjLS09M93SWvtnv3bgwaNAiNGzeGQqHAd9995+kueb3Zs2ejc+fOCAsLQ1RUFIYMGYITJ054ulteiYGIG/Xu3Rvr1q3DiRMnsH79emRnZ+PBBx/0dLe81vHjx6HVarF8+XIcPXoUCxcuxAcffICXXnrJ013zamVlZXjooYcwduxYT3fFK3311VeYPHkyZs6ciUOHDqFt27a46667cPHiRU93zWsVFxejbdu2WLp0qae74jN++eUXjBs3Dvv378e2bdtQXl6OO++8E8XFxZ7umtfh8l0P2rRpE4YMGQKNRoPAwEBPd8cnzJs3D8uWLcNff/3l6a54vVWrVmHixIm4du2ap7viVZKTk9G5c2csWbIEgG5Pq9jYWDz//POYPn26h3vn/RQKBTZs2IAhQ4Z4uis+5dKlS4iKisIvv/yCnj17ero7XoUjIh5y5coVrFmzBt26dWMQIoNarUZERISnu0E+qqysDAcPHkS/fv0MbQEBAejXrx/27dvnwZ6Rv1Or1QDAv78sYCDiZtOmTUOdOnXQoEEDnDlzBhs3bvR0l3zG6dOnsXjxYjz77LOe7gr5qMuXL6OyshKNGjUStTdq1Aj5+fke6hX5O61Wi4kTJ6J79+5ISkrydHe8DgORapo+fToUCoXNn+PHjxvOnzp1Kv744w/89NNPUCqVePLJJ1HTZsfkfmcAcO7cOaSmpuKhhx7C6NGjPdRzz3HkOyMi7zBu3DhkZmbiyy+/9HRXvFItT3fA102ZMgUjRoyweU7z5s0N/92wYUM0bNgQN998M1q3bo3Y2Fjs378fKSkpLu6p95D7nZ0/fx69e/dGt27d8OGHH7q4d95J7ndGljVs2BBKpRIXLlwQtV+4cAHR0dEe6hX5s/Hjx2PLli3YvXs3mjRp4unueCUGItUUGRmJyMhIh67VarUAAI1G48wueT0539m5c+fQu3dvdOzYEStXrkRAQM0cxKvOnzOqEhQUhI4dO2LHjh2GZEutVosdO3Zg/Pjxnu0c+RVBEPD8889jw4YN+PnnnxEfH+/pLnktBiJu8ttvv+HAgQPo0aMH6tevj+zsbLz66qtISEioUaMhcpw7dw69evVCs2bNMH/+fFy6dMnwGf/1at2ZM2dw5coVnDlzBpWVlcjIyAAAtGjRAnXr1vVs57zA5MmTMXz4cHTq1AldunTBokWLUFxcjJEjR3q6a17r+vXrOH36tOE4JycHGRkZiIiIQNOmTT3YM+81btw4rF27Fhs3bkRYWJghB0mlUiE0NNTDvfMyArnFkSNHhN69ewsRERFCcHCwEBcXJ4wZM0b4559/PN01r7Vy5UoBgMUfsm748OEWv7Ndu3Z5umteY/HixULTpk2FoKAgoUuXLsL+/fs93SWvtmvXLot/poYPH+7prnkta393rVy50tNd8zqsI0JEREQeUzMn3ImIiMgrMBAhIiIij2EgQkRERB7DQISIiIg8hoEIEREReQwDESIiIvIYBiJERETkMQxEiIiIyGMYiBAREZHHMBAhIiIij2EgQkRERB7DQISIiIg85v8BGOXhIYHsK3sAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(xs, ys)\n",
    "plt.plot(xs, params.weight * xs + params.bias, c='red', label='Model Prediction')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stateful Computations in JAX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, if we can’t have side effects, how do we update model parameters, the optimizer state, and use stateful layers in our models? This colab will explain this in detail, but the short answer is: with **functional programming**.\n",
    "\n",
    "Solution:\n",
    "- Explicit State. Convert Class Data into Function Arguments. (Making OOP uselesss)\n",
    "- Use Library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------\n",
    "# End"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "jax",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}