ml-tensoralgebra-pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensoralgebra in PyTorch\n",
    "\n",
    "PyTorch provides the following functions for **algebraic operations**:\n",
    "\n",
    "```\n",
    "A.mm(B)                    # matrix multiplication; alias: A.matmul(B)\n",
    "A.mv(x)                    # matrix vector multiplication\n",
    "x.dot(y)                   # inner vector product / dot product\n",
    "x.t()                      # matrix transpose\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors\n",
    "\n",
    "In machine learning, **features** $x$ are typically described as a **column vector** of $n$ feature units:\n",
    "\n",
    "$$x = \\begin{pmatrix}x_1 \\\\ \\vdots \\\\ x_n \\end{pmatrix}$$\n",
    "\n",
    "Then a single neuron is the **inner product** or **dot product** `torch.dot()` (a.k.a. scalar product in a cartesian coordinate system) with a **column vector** of **weights** $w$\n",
    "\n",
    "$$w = \\begin{pmatrix}w_1 \\\\ \\vdots \\\\ w_n \\end{pmatrix}$$\n",
    "\n",
    "applied to an **activation function** $f$ which given a **scalar** $y$:\n",
    "\n",
    "$$y = f(w \\cdot x) = f(\\sum_{i=0}^n w_i x_i)$$\n",
    "\n",
    "In `PyTorch`, this is written as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3,)\n",
      "(3,)\n",
      "20\n",
      "()\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.tensor([1, 2, 3])\n",
    "print(x.numpy().shape)\n",
    "\n",
    "w = torch.tensor([2, 3, 4])\n",
    "print(w.numpy().shape)\n",
    "\n",
    "y = torch.dot(w, x)               # alternatively w.dot(x)\n",
    "print(y.item())\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **dot product** is **commutative**, hence we can also switch the order of the operands:\n",
    "\n",
    "$$y = f(x \\cdot w) = f(\\sum_{i=0}^n x_i w_i)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n",
      "()\n"
     ]
    }
   ],
   "source": [
    "y = torch.dot(x, w)               # alternatively x.dot(w)\n",
    "print(y.item())\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrices\n",
    "\n",
    "If we describe features and weights with a $n \\times 1$ **column matrix**, we have \n",
    "\n",
    "$$x = \\begin{bmatrix}x_1 \\\\ \\vdots \\\\ x_n \\end{bmatrix} \\qquad \\text{and} \\qquad w = \\begin{bmatrix}w_1 \\\\ \\vdots \\\\ w_n \\end{bmatrix}$$\n",
    "\n",
    "and the dot product can be written as **matrix product** `torch.matmul()` using the **transpose** $w^\\top$ if the number of **columns** of the **left operand** is equal to the number of **rows** of the **right operand**:\n",
    "\n",
    "$$w^\\top \\cdot x = [w_1, \\ldots, w_n] \\cdot \\begin{bmatrix}x_1 \\\\ \\vdots \\\\ x_n \\end{bmatrix} = \\sum_{i=0}^n w_i x_i$$\n",
    "\n",
    "If we use a **row matrix** $x = [x_1, \\ldots, x_n]$ for the features, we must switch the order of the operands to get the same scalar:\n",
    "\n",
    "$$x \\cdot w = [x_1, \\ldots, x_n] \\cdot \\begin{bmatrix}w_1 \\\\ \\vdots \\\\ w_n \\end{bmatrix} = \\sum_{i=0}^n x_i w_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.6614, 0.2669, 0.0617]])\n",
      "(1, 3)\n",
      "tensor([[1.],\n",
      "        [2.],\n",
      "        [3.]])\n",
      "(3, 1)\n",
      "1.3802322149276733\n",
      "(1, 1)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1)\n",
    "\n",
    "w = torch.randn(3, 1)\n",
    "print(w.t())\n",
    "print(w.t().numpy().shape)\n",
    "\n",
    "x = torch.tensor([[1.], [2.], [3.]])\n",
    "print(x)\n",
    "print(x.numpy().shape)\n",
    "\n",
    "y = torch.matmul(w.t(), x)              # alternatively `w.t().mm(x)` or `w.t() @ x`\n",
    "print(y.item())\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As **matrix multiplication** is **non-communtative** $AB \\neq BA$, we must take care of the order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outer Product (German: Dyadisches Produkt)\n",
    "\n",
    "An outer product is a product of two vectors, or two one-column matrices, which results in a **matrix**:\n",
    "\n",
    "$$x \\otimes y = x \\cdot y^\\top = {\\begin{bmatrix} x_1 \\\\ \\vdots \\\\ x_m \\end{bmatrix}} \\cdot [ y_1, \\cdots, y_n ] = \\begin{bmatrix} x_1 y_1 & \\cdots & x_1 y_n \\\\ \\vdots & & \\vdots \\\\ x_m y_1 & \\cdots & x_m y_n \\end{bmatrix}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.6614],\n",
      "        [0.2669],\n",
      "        [0.0617]])\n",
      "(3, 1)\n",
      "tensor([[1., 2., 3.]])\n",
      "(1, 3)\n",
      "tensor([[0.6614, 1.3227, 1.9841],\n",
      "        [0.2669, 0.5338, 0.8008],\n",
      "        [0.0617, 0.1234, 0.1850]])\n",
      "(3, 3)\n"
     ]
    }
   ],
   "source": [
    "print(w)\n",
    "print(w.numpy().shape)\n",
    "\n",
    "print(x.t())\n",
    "print(x.t().numpy().shape)\n",
    "\n",
    "y = torch.matmul(w, x.t())\n",
    "print(y)\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi Layer Networks\n",
    "\n",
    "When we have a multi layer neural network, the weights are typically a **matrix** $W$ with $n$ **rows** for $n$ **features** and $m$ **columns** for $m$ **hidden nodes**:\n",
    "\n",
    "$$W = \\begin{bmatrix}w_{11} & \\cdots & w_{1m} \\\\ \\vdots & & \\vdots \\\\ w_{n1} & \\cdots & w_{nm} \\end{bmatrix}$$\n",
    "\n",
    "Then the items of the hidden layer are a **column matrix**\n",
    "\n",
    "$$h = f(W^\\top x) = \\begin{bmatrix}h_1 \\\\ \\vdots \\\\ h_m \\end{bmatrix}$$\n",
    "\n",
    "with\n",
    "\n",
    "$$W^\\top x = \\begin{bmatrix}w_{11} & \\cdots & w_{n1} \\\\ \\vdots & & \\vdots \\\\ w_{1m} & \\cdots & w_{nm} \\end{bmatrix} \\cdot \\begin{bmatrix}x_1 \\\\ \\vdots \\\\ x_n \\end{bmatrix}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.6614,  0.0617, -0.4519],\n",
      "        [ 0.2669,  0.6213, -0.1661]])\n",
      "(2, 3)\n",
      "tensor([[1.],\n",
      "        [2.],\n",
      "        [3.]])\n",
      "(3, 1)\n",
      "tensor([[-0.5710],\n",
      "        [ 1.0112]])\n",
      "(2, 1)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1)\n",
    "\n",
    "W = torch.randn(3, 2)\n",
    "print(W.t())\n",
    "print(W.t().numpy().shape)\n",
    "\n",
    "x = torch.tensor([[1.], [2.], [3.]])\n",
    "print(x)\n",
    "print(x.numpy().shape)\n",
    "\n",
    "y = torch.matmul(W.t(), x)\n",
    "print(y)\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the number of **columns** of $W$ is **not equal** the number of **rows** of $x$, we can't calculate $x \\cdot W^\\top$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "size mismatch, m1: [3 x 1], m2: [2 x 3] at ../aten/src/TH/generic/THTensorMath.cpp:41",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-ac248ce1291b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mW\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: size mismatch, m1: [3 x 1], m2: [2 x 3] at ../aten/src/TH/generic/THTensorMath.cpp:41"
     ]
    }
   ],
   "source": [
    "y = torch.matmul(x, W.t())\n",
    "print(y)\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But as $(A \\cdot B)^T = B^T \\cdot A^T$ and $(A^\\top)^\\top = A$, we can caluclate\n",
    "\n",
    "$$x^\\top \\cdot W = ((x^\\top \\cdot W)^\\top)^\\top = (W^\\top \\cdot (x^\\top)^\\top)^\\top = (W^\\top \\cdot x)^\\top = h^\\top$$\n",
    "\n",
    "to get the **transpose** of the **hidden column matrix**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.5710,  1.0112]])\n",
      "(1, 2)\n"
     ]
    }
   ],
   "source": [
    "y = torch.matmul(x.t(), W)\n",
    "print(y)\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is common in PyTorch to use a **row matrix** for a feature vector, hence we calculate the **matrix product** `torch.matmul()` as $x \\cdot W$ to get a **hidden row matrix**:\n",
    "\n",
    "$$h = f(x \\cdot W) = [h_1, \\ldots, h_m]$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Batches\n",
    "\n",
    "In PyTorch it is common to have a **batch** of $k$ **feature vectors**, which are described as a matrix having $k$ rows and $n$ columns. \n",
    "\n",
    "Each row represents one feature vector $x^{(i)} = (x_1^{(i)}, \\ldots, x_n^{(i)})$ (consider features as stacked within a matrix):\n",
    "\n",
    "$$X = \\begin{bmatrix}x^{(1)} \\\\ \\vdots \\\\ x^{(k)} \\end{bmatrix} = \\begin{bmatrix}x_1^{(1)} & \\cdots & x_n^{(1)} \\\\ \\vdots & & \\vdots \\\\ x_1^{(k)} & \\cdots & x_n^{(k)} \\end{bmatrix} = \\begin{bmatrix}x_{11} & \\cdots & x_{1n} \\\\ \\vdots & & \\vdots \\\\ x_{k1} & \\cdots & x_{kn} \\end{bmatrix}$$\n",
    "\n",
    "With a **weights matrix** from a multi layer neural network\n",
    "\n",
    "$$W = \\begin{bmatrix}w_{11} & \\cdots & w_{1m} \\\\ \\vdots & & \\vdots \\\\ w_{n1} & \\cdots & w_{nm} \\end{bmatrix}$$\n",
    "\n",
    "we then calculate the items of the hidden layer as the $k \\times m$ **matrix product** of the $k \\times n$ **feature matrix** and the $n \\times m$ **weight matrix**:\n",
    "\n",
    "$$h = f(X \\cdot W) = \\begin{bmatrix}h_{11} & \\cdots & h_{1m} \\\\ \\vdots & & \\vdots \\\\ h_{k1} & \\cdots & h_{km} \\end{bmatrix} = \\begin{bmatrix}h^{(1)} \\\\ \\vdots \\\\ h^{(k)} \\end{bmatrix}$$\n",
    "\n",
    "with\n",
    "\n",
    "$$X \\cdot W = \\begin{bmatrix}x_{11} & \\cdots & x_{1n} \\\\ \\vdots & & \\vdots \\\\ x_{k1} & \\cdots & x_{kn} \\end{bmatrix} \\cdot \\begin{bmatrix}w_{11} & \\cdots & w_{1m} \\\\ \\vdots & & \\vdots \\\\ w_{n1} & \\cdots & w_{nm} \\end{bmatrix}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 2., 3.]])\n",
      "(1, 3)\n",
      "tensor([[ 0.6614,  0.2669],\n",
      "        [ 0.0617,  0.6213],\n",
      "        [-0.4519, -0.1661]])\n",
      "(3, 2)\n",
      "tensor([[-0.5710,  1.0112]])\n",
      "(1, 2)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1)\n",
    "\n",
    "X = torch.tensor([[1., 2., 3.]])          # 1 batch, feature as a row-matrix\n",
    "print(X)\n",
    "print(X.numpy().shape)\n",
    "\n",
    "W = torch.randn(3, 2)                     # 3 input units, 2 hidden units\n",
    "print(W)\n",
    "print(W.numpy().shape)\n",
    "\n",
    "y = torch.matmul(X, W)\n",
    "print(y)\n",
    "print(y.numpy().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example of a Neural Network\n",
    "\n",
    "This is an example neural network with 3 **input units**, 2 **hidden units** and 1 **output unit** using a non-linearity as activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.1766]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(1)\n",
    "\n",
    "X = torch.randn(1, 3)\n",
    "\n",
    "W_ih = torch.randn(3, 2)\n",
    "W_ho = torch.randn(2, 1)\n",
    "\n",
    "B_h = torch.randn(1, 2)\n",
    "B_o = torch.randn(1, 1)\n",
    "\n",
    "h = torch.sigmoid(torch.mm(X, W_ih) + B_h)\n",
    "y = torch.sigmoid(torch.mm(h, W_ho) + B_o)\n",
    "\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensions in PyTorch\n",
    "\n",
    "### Addressing Dimensions\n",
    "\n",
    "PyTorch provides functions like `torch.sum()`, `torch.mean()`, `torch.max()` or `torch.nn.Softmax()` for **element wise operations** which allow to define the axis of the operation via an `dim` argument.\n",
    "\n",
    "An important concept of PyTorch is, that each **new dimension** gets **prepended** and takes the first position at `dim=0`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 4, 3)"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.ones((4,3))\n",
    "b = torch.ones((4,3))\n",
    "c = torch.stack((a, b))       # create new dimension at dim=0\n",
    "c.numpy().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means, that `dim` represents the **index** of an **n-dimensional tensor**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example for an 2 Dimensional Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.]])\n",
      "(2, 4)\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones((2, 4))\n",
    "print(x)\n",
    "print(x.numpy().shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(8.)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum()                           # sum element wise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 2., 2., 2.])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum(dim=0)                      # sum in y-axis (here: dim index 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4., 4.])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum(dim=1)                      # sum in x-axis (here: dim index 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example for an 3 Dimensional Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[1., 1., 1., 1.],\n",
      "         [1., 1., 1., 1.],\n",
      "         [1., 1., 1., 1.]],\n",
      "\n",
      "        [[1., 1., 1., 1.],\n",
      "         [1., 1., 1., 1.],\n",
      "         [1., 1., 1., 1.]]])\n",
      "(2, 3, 4)\n"
     ]
    }
   ],
   "source": [
    "x = torch.stack((torch.ones((3,4)), torch.ones((3,4))))\n",
    "print(x)\n",
    "print(x.numpy().shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(24.)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 2., 2., 2.],\n",
       "        [2., 2., 2., 2.],\n",
       "        [2., 2., 2., 2.]])"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum(dim=0)                       # sum in  z-axis (here: dim index 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3., 3., 3., 3.],\n",
       "        [3., 3., 3., 3.]])"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum(dim=1)                       # sum in  y-axis (here: dim index 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[4., 4., 4.],\n",
       "        [4., 4., 4.]])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.sum(dim=2)                       # sum in  x-axis (here: dim index 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Dimensions\n",
    "\n",
    "In order to add new dimensions to a PyTorch tensor, you can add a new index with `[None]`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2,)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.ones(2)\n",
    "x.numpy().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 32)"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[None].numpy().shape              # a trailing `:` can be skipped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32, 1)"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[:, None].numpy().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 1, 32, 1)"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[None, None, :, None].numpy().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `unsqueeze()` function allows to add a new dimension at a **specific dimension**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3, 4)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.ones((2, 3, 4))\n",
    "x.numpy().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 2, 3, 4)"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.unsqueeze(dim=0).numpy().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3, 1, 4)"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.unsqueeze(dim=2).numpy().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `view()` function allows to reshape tensors and to add new dimensions via the `-1` dimension placeholder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32,)"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.ones(32)\n",
    "x.numpy().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8, 2, 2)"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.view(8, 2, 2).numpy().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8, 4)"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.view(-1, 4).numpy().shape              # `-1` can only be used once!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting\n",
    "\n",
    "When operating on two arrays, PyTorch compares their shapes element-wise.\n",
    "\n",
    "Two dimensions are compatible when:\n",
    "\n",
    "1. they are **equal**, or\n",
    "2. one of them is **1**\n",
    "\n",
    "When either of the dimensions compared is one, the other is used. In other words, **dimensions with size 1 are stretched or copied to match the other**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 3, 2])"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.ones((4, 3, 2))\n",
    "b = torch.ones((   3, 1))\n",
    "(a + b).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 3, 2])"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.ones((4, 3, 2))\n",
    "b = torch.ones((   3, 2))\n",
    "(a + b).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 2",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-156-abc8a43b6bd8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m   \u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 2"
     ]
    }
   ],
   "source": [
    "a = torch.ones((4, 3, 2))\n",
    "b = torch.ones((   3, 3))\n",
    "(a + b).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 3])"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.ones((1, 3))\n",
    "b = torch.ones((3, 1))\n",
    "(a + b).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}