04-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mRCQfI4o9f0k"
   },
   "source": [
    "Regression and Stochastic Gradient Descent\n",
    "===================================\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regression consists in finding a model $f$ that depends on parameters $\\theta$ and that verifies $f(\\theta, x_i) = y_i$ for a given set of $(x_i, y_i)_{i=1..n}$. \n",
    "\n",
    "Most often, this can only be verified approximately, typically because there is noise or because the problem is over-determined.\n",
    "\n",
    "In that case, regression can be recast as an optimization problem, for example by minimizing the loss: \n",
    "\n",
    "\\begin{equation}\n",
    "L = \\sum_i \\| y_i - f(\\theta, x_i) \\|_2^2\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regressing between 2 squares"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make 2 sets of points, both on a square, such that there is an affine transformation between them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_2squares(n): \n",
    "    a = np.random.rand(n) * 2 * np.pi\n",
    "    x = np.vstack((np.cos(a), np.sin(a)))\n",
    "    x = x / np.linalg.norm(x, axis=0, ord=1)\n",
    "    y = np.vstack((np.cos(a + np.pi / 4), np.sin(a + np.pi / 4)))\n",
    "    y = y / np.linalg.norm(y, axis=0, ord=np.inf)\n",
    "    y[0] += 3\n",
    "    x += np.random.randn(*x.shape) * 0.02\n",
    "    y += np.random.randn(*x.shape) * 0.02\n",
    "    return x.T, y.T\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = gen_2squares(200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyplot.plot(x[:, 0], x[:, 1], '+', label='x')\n",
    "pyplot.plot(y[:, 0], y[:, 1], '.', label='y')\n",
    "pyplot.axis('equal')\n",
    "pyplot.legend()\n",
    "pyplot.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The affine transformation: the parameters are a 2x2 matrix and a 2D translation. We stack them together in a 2x3 matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(theta, x):\n",
    "    return torch.matmul(x, theta[:2, :]) + theta[2, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt = torch.from_numpy(x.astype('float32'))\n",
    "yt = torch.from_numpy(y.astype('float32'))\n",
    "\n",
    "# starting point\n",
    "theta = torch.tensor([\n",
    "    [1., 0.], \n",
    "    [0., 1.], \n",
    "    [0., 0.]\n",
    "])\n",
    "\n",
    "# set the learning rate\n",
    "learning_rate = 0.1\n",
    "for it in range(20):    \n",
    "    \n",
    "    # we will need a gradient wrt. x\n",
    "    theta.requires_grad = True\n",
    "    \n",
    "    # call the function, record dependencies for the gradient\n",
    "    y_cur = f(theta, xt)\n",
    "    err = ((y_cur - yt) ** 2).sum(1).mean()\n",
    "    \n",
    "    print(err.item())\n",
    "    \n",
    "    # compute gradients\n",
    "    err.backward()    \n",
    "    \n",
    "    # update current solution\n",
    "    theta = theta.data - learning_rate * theta.grad\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercices:** \n",
    "\n",
    "- what is the error we are optimizing?\n",
    "\n",
    "- visualize the intermediate solutions\n",
    "\n",
    "- vary the learning rate. What is the range of paramters where this converges?\n",
    "\n",
    "- vary the noise on the input data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stochastic gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The optimization sees points in \"mini-batches\", ie. the sum $\\sum_i \\| y_i - f(\\theta, x_i) \\|_2^2$ is not computed as a whole but for a subset of points. The points are visited in a random order. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt = torch.from_numpy(x.astype('float32'))\n",
    "yt = torch.from_numpy(y.astype('float32'))\n",
    "\n",
    "# starting point\n",
    "theta = torch.tensor([\n",
    "    [1., 0.], \n",
    "    [0., 1.], \n",
    "    [0., 0.]\n",
    "])\n",
    "\n",
    "# set the learning rate\n",
    "learning_rate = 0.5\n",
    "n = xt.shape[0]\n",
    "\n",
    "for it in range(20):        \n",
    "    # random order \n",
    "    perm = torch.randperm(n)\n",
    "    errs = []\n",
    "    for i0 in range(0, n, 10):\n",
    "        \n",
    "        # i0 is called the batch size\n",
    "        theta.requires_grad = True\n",
    "        \n",
    "        # we handle this subset of points (a minibatch)\n",
    "        xbatch = xt[perm[i0 : i0 + 10]]\n",
    "        ybatch = yt[perm[i0 : i0 + 10]]\n",
    "    \n",
    "        # compute the partial loss on this mini-batch\n",
    "        y_cur_batch = f(theta, xbatch)\n",
    "        err = ((y_cur_batch - ybatch) ** 2).sum(1).mean()\n",
    "        errs.append(err.item())\n",
    "    \n",
    "        # compute gradients\n",
    "        err.backward()    \n",
    "\n",
    "        # update current solution\n",
    "        theta = theta.data - learning_rate * theta.grad\n",
    "\n",
    "    print(np.mean(errs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercices**\n",
    "\n",
    "- What do we observe in terms of learning speed?\n",
    "\n",
    "- try to vary the batch size. Which one works best?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Write it with a Module and an Optimizer object\n",
    "\n",
    "An opimizer manages several parameters and does update them. \n",
    "\n",
    "\n",
    "A module is an object that can be called as a function that automatically identifies its \n",
    "parameters -- tensors with gradients\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AffineTransform(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        nn.Module.__init__(self)\n",
    "        self.A = nn.Parameter(\n",
    "            torch.tensor([\n",
    "                [1., 0.], \n",
    "                [0., 1.]\n",
    "            ])\n",
    "        )\n",
    "        self.b = nn.Parameter(torch.tensor([0., 0.]))\n",
    "\n",
    "    def forward(self, x):\n",
    "        # called when the object is used as a function\n",
    "        return torch.matmul(x, self.A) + self.b\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt = torch.from_numpy(x.astype('float32'))\n",
    "yt = torch.from_numpy(y.astype('float32'))\n",
    "\n",
    "# the \"model\" = function + parameters\n",
    "af = AffineTransform()\n",
    "\n",
    "# the optimizer will take care of the parameters\n",
    "optimizer = optim.SGD(af.parameters(), lr=0.5)\n",
    "\n",
    "n = xt.shape[0]\n",
    "\n",
    "for it in range(20):        \n",
    "    perm = torch.randperm(n)\n",
    "    errs = []\n",
    "    for i0 in range(0, n, 10):\n",
    "\n",
    "        # manipulate through the optimizer object\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        xbatch = xt[perm[i0 : i0 + 10]]\n",
    "        ybatch = yt[perm[i0 : i0 + 10]]\n",
    "    \n",
    "        # call the model with the batch\n",
    "        y_cur_batch = af(xbatch)\n",
    "        err = ((y_cur_batch - ybatch) ** 2).sum(1).mean()\n",
    "        errs.append(err.item())\n",
    "    \n",
    "        # compute gradients\n",
    "        err.backward()    \n",
    "        \n",
    "        # update current solution\n",
    "        optimizer.step()\n",
    "\n",
    "    print(np.mean(errs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regressing between a square and... a circle!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we center the data on (0, 0) and remove the noise.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_square_circle(n): \n",
    "    n4 = int(n/4)\n",
    "    a = np.random.rand(n) * 2 * np.pi\n",
    "    x = np.vstack((np.cos(a), np.sin(a)))\n",
    "    x = x / np.linalg.norm(x, axis=0, ord=2)\n",
    "    y = np.vstack((np.cos(a + np.pi / 4), np.sin(a + np.pi / 4)))\n",
    "    y = y / np.linalg.norm(y, axis=0, ord=np.inf)\n",
    "    return x.T, y.T\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = gen_square_circle(200)\n",
    "\n",
    "pyplot.plot(x[:, 0], x[:, 1], '+', label='x')\n",
    "pyplot.plot(y[:, 0], y[:, 1], '.', label='y')\n",
    "pyplot.axis('equal')\n",
    "pyplot.legend()\n",
    "pyplot.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously we need a non-linear model. One of the great principles of neural nets is that we have the combination of:\n",
    "\n",
    "- large linear transformations: fast to evaluate and lots of parameters\n",
    "\n",
    "- scalar non-linearities.\n",
    "\n",
    "The combination has a great expressive power: many mappings can be expressed with it. Let's make a 2-layer neural net to find a reasonable mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NonLinearTransform(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        nn.Module.__init__(self)\n",
    "        # object that encapsulates an affine transform, initialized randomly\n",
    "        # the parameters of the affine transform are extracted automatically\n",
    "        self.fc1 = nn.Linear(2, 8)\n",
    "        self.fc2 = nn.Linear(8, 2)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = self.fc1(x)\n",
    "        y = self.fc2(F.relu(z))\n",
    "        return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt = torch.from_numpy(x.astype('float32'))\n",
    "yt = torch.from_numpy(y.astype('float32'))\n",
    "\n",
    "# the \"model\" = function + parameters\n",
    "nlt = NonLinearTransform()\n",
    "\n",
    "# the optimizer will take care of the parameters\n",
    "lr = 0.1\n",
    "optimizer = optim.SGD(nlt.parameters(), lr=lr, momentum=0.9)\n",
    "\n",
    "n = xt.shape[0]\n",
    "all_errs = []\n",
    "for it in range(200):        \n",
    "    perm = torch.randperm(n)\n",
    "    errs = []\n",
    "    nlt.train()\n",
    "    bs = 20\n",
    "    for i0 in range(0, n, bs):\n",
    "\n",
    "        # manipulate through the optimizer object\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        xbatch = xt[perm[i0 : i0 + bs]]\n",
    "        ybatch = yt[perm[i0 : i0 + bs]]\n",
    "    \n",
    "        # call the model with the batch\n",
    "        y_cur_batch = nlt(xbatch)\n",
    "        err = ((y_cur_batch - ybatch) ** 2).sum(1).mean()\n",
    "        errs.append(err.item())\n",
    "    \n",
    "        # compute gradients\n",
    "        err.backward()    \n",
    "        \n",
    "        # update current solution\n",
    "        optimizer.step()\n",
    "    # lr /= 1.05\n",
    "    optimizer = optim.SGD(nlt.parameters(), lr=lr, momentum=0.9, weight_decay=1e-3)\n",
    "    nlt.eval()\n",
    "    with torch.no_grad():\n",
    "        points.append(nlt(xt).numpy())\n",
    "    all_errs.append(np.mean(errs))\n",
    "    if it % 10 == 0: \n",
    "        print(it, all_errs[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.semilogy(all_errs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercices**:\n",
    "\n",
    "- visualize the mapping on concentric circles\n",
    "\n",
    "- generate more points, add noise \n",
    "\n",
    "- how can we improve the objective and mapping: use different learning rate? bigger hidden state? deeper network? different non-linearity?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "autograd_tutorial.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}