DCGAN-MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Teaching a Deep Convolutional Generative Adversarial Network (DCGAN) to draw MNIST characters\n",
    "\n",
    "In the last tutorial, we learnt using tensorflow for designing a Variational Autoencoder (VAE) that could draw MNIST characters. Most of the created digits looked nice. There was only one drawback -- some of the created images looked a bit fuzzy. The VAE was trained with the _mean squared error_ loss function. However, it's quite difficult to encode exact character edge locations, which leads to the network being unsure about those edges. And does it really matter if the edge of a character starts a few pixels more to the left or right? Not really.\n",
    "In this article, we will see how we can train a network that does not depend on the mean squared error or any related loss functions--instead, it will learn all by itself what a real image should look like.\n",
    "\n",
    "## Deep Convolutional Generative Adversarial Networks\n",
    "Another network architecture for learning to generate new content is the DCGAN. Like the VAE, our DCGAN consists of two parts:\n",
    "* The _discriminator_ learns how to distinguish fake from real objects of the type we'd like to create\n",
    "* The _generator_ creates new content and tries to fool the discriminator\n",
    "\n",
    "There is a HackerNoon article by Chanchana Sornsoontorn that explains the concept in more detail and describes some creative projects DCGANs have been applied to. One of these projects is the generation of MNIST characters. Let's try to use python and tensorflow for the same purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Code by Parag Mital (github.com/pkmital/CADL)\n",
    "def montage(images):\n",
    "    if isinstance(images, list):\n",
    "        images = np.array(images)\n",
    "    img_h = images.shape[1]\n",
    "    img_w = images.shape[2]\n",
    "    n_plots = int(np.ceil(np.sqrt(images.shape[0])))\n",
    "    m = np.ones((images.shape[1] * n_plots + n_plots + 1, images.shape[2] * n_plots + n_plots + 1)) * 0.5\n",
    "    for i in range(n_plots):\n",
    "        for j in range(n_plots):\n",
    "            this_filter = i * n_plots + j\n",
    "            if this_filter < images.shape[0]:\n",
    "                this_img = images[this_filter]\n",
    "                m[1 + i + i * img_h:1 + i + (i + 1) * img_h,\n",
    "                  1 + j + j * img_w:1 + j + (j + 1) * img_w] = this_img\n",
    "    return m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the basics\n",
    "Like in the last tutorial, we use tensorflow's own method for accessing batches of MNIST characters. We set our batch size to be 64. Our generator will take noise as input. The number of these inputs is being set to 100. Batch normalization considerably improved the training of this network. For tensorflow to apply batch normalization, we need to let it know whether we are in training mode. The variable _keep_prob_ will be used by our dropout layers, which we introduce for more stable learning outcomes.\n",
    "_lrelu_ defines the popular leaky ReLU, that hopefully will be supported by future versions of tensorflow! I firstly tried to apply standard ReLUs to this network, but this lead to the well-known _dying ReLU problem_, and I received generated images that looked like artwork by Kazimir Malevich--I just got black squares. \n",
    "\n",
    "Then, we define a function _binary_cross_entropy_, which we will use later, when computing losses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data')\n",
    "tf.reset_default_graph()\n",
    "batch_size = 64\n",
    "n_noise = 64\n",
    "\n",
    "X_in = tf.placeholder(dtype=tf.float32, shape=[None, 28, 28], name='X')\n",
    "noise = tf.placeholder(dtype=tf.float32, shape=[None, n_noise])\n",
    "\n",
    "keep_prob = tf.placeholder(dtype=tf.float32, name='keep_prob')\n",
    "is_training = tf.placeholder(dtype=tf.bool, name='is_training')\n",
    "\n",
    "def lrelu(x):\n",
    "    return tf.maximum(x, tf.multiply(x, 0.2))\n",
    "\n",
    "def binary_cross_entropy(x, z):\n",
    "    eps = 1e-12\n",
    "    return (-(x * tf.log(z + eps) + (1. - x) * tf.log(1. - z + eps)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The discriminator\n",
    "Now, we can define the discriminator. It looks similar to the encoder part of our VAE. As input, it takes real or fake MNIST digits (28 x 28 pixel grayscale images) and applies a series of convolutions. Finally, we use a sigmoid to make sure our output can be interpreted as the probability to that the input image is a real MNIST character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "def discriminator(img_in, reuse=None, keep_prob=keep_prob):\n",
    "    activation = lrelu\n",
    "    with tf.variable_scope(\"discriminator\", reuse=reuse):\n",
    "        x = tf.reshape(img_in, shape=[-1, 28, 28, 1])\n",
    "        x = tf.layers.conv2d(x, kernel_size=5, filters=64, strides=2, padding='same', activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)\n",
    "        x = tf.layers.conv2d(x, kernel_size=5, filters=64, strides=1, padding='same', activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)\n",
    "        x = tf.layers.conv2d(x, kernel_size=5, filters=64, strides=1, padding='same', activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)\n",
    "        x = tf.contrib.layers.flatten(x)\n",
    "        x = tf.layers.dense(x, units=128, activation=activation)\n",
    "        x = tf.layers.dense(x, units=1, activation=tf.nn.sigmoid)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The generator\n",
    "The generator--just like the decoder part in our VAE--takes noise and tries to learn how to transform this noise into digits. To this end, it applies several transpose convolutions. At first, I didn't apply batch normalization to the generator, and its learning seemed to be really unefficient. After applying batch normalization layers, learning improved considerably. Also, I firstly had a much larger dense layer accepting the generator input. This led to the generator creating the same output always, no matter what the input noise was. Tuning the generator honestly took quite some effort!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generator(z, keep_prob=keep_prob, is_training=is_training):\n",
    "    activation = lrelu\n",
    "    momentum = 0.99\n",
    "    with tf.variable_scope(\"generator\", reuse=None):\n",
    "        x = z\n",
    "        d1 = 4\n",
    "        d2 = 1\n",
    "        x = tf.layers.dense(x, units=d1 * d1 * d2, activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)      \n",
    "        x = tf.contrib.layers.batch_norm(x, is_training=is_training, decay=momentum)  \n",
    "        x = tf.reshape(x, shape=[-1, d1, d1, d2])\n",
    "        x = tf.image.resize_images(x, size=[7, 7])\n",
    "        x = tf.layers.conv2d_transpose(x, kernel_size=5, filters=64, strides=2, padding='same', activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)\n",
    "        x = tf.contrib.layers.batch_norm(x, is_training=is_training, decay=momentum)\n",
    "        x = tf.layers.conv2d_transpose(x, kernel_size=5, filters=64, strides=2, padding='same', activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)\n",
    "        x = tf.contrib.layers.batch_norm(x, is_training=is_training, decay=momentum)\n",
    "        x = tf.layers.conv2d_transpose(x, kernel_size=5, filters=64, strides=1, padding='same', activation=activation)\n",
    "        x = tf.layers.dropout(x, keep_prob)\n",
    "        x = tf.contrib.layers.batch_norm(x, is_training=is_training, decay=momentum)\n",
    "        x = tf.layers.conv2d_transpose(x, kernel_size=5, filters=1, strides=1, padding='same', activation=tf.nn.sigmoid)\n",
    "        return x    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss functions and optimizers\n",
    "Now, we wire both parts together, like we did for the encoder and the decoder of our VAE in the last tutorial.\n",
    "However, we have to create two objects of our discriminator\n",
    "* The first object receives the real images\n",
    "* The second object receives the fake images\n",
    "\n",
    "_reuse_ of the second object is set to _True_ so both objects share their variables. We need both instances for computing two types of losses:\n",
    "* when receiving real images, the discriminator should learn to compute high values (near _1_), meaning that it is confident the input images are real\n",
    "* when receiving fake images, it should compute low values (near _0_), meaning it is confident the input images are not real\n",
    "\n",
    "To accomplish this, we use the _binary cross entropy_ function defined earlier. The generator tries to achieve the opposite goal, it tries to make the discriminator assign high values to fake images.\n",
    "\n",
    "Now, we also apply some regularization. We create two distinct optimizers, one for the discriminator, one for the generator. We have to define which variables we allow these optimizers to modify, otherwise the generator's optimizer could just mess up the discriminator's variables and vice-versa.\n",
    "\n",
    "We have to provide the __update_ops__ to our optimizers when applying batch normalization--take a look at the tensorflow documentation for more information on this topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "g = generator(noise, keep_prob, is_training)\n",
    "d_real = discriminator(X_in)\n",
    "d_fake = discriminator(g, reuse=True)\n",
    "\n",
    "vars_g = [var for var in tf.trainable_variables() if var.name.startswith(\"generator\")]\n",
    "vars_d = [var for var in tf.trainable_variables() if var.name.startswith(\"discriminator\")]\n",
    "\n",
    "\n",
    "d_reg = tf.contrib.layers.apply_regularization(tf.contrib.layers.l2_regularizer(1e-6), vars_d)\n",
    "g_reg = tf.contrib.layers.apply_regularization(tf.contrib.layers.l2_regularizer(1e-6), vars_g)\n",
    "\n",
    "loss_d_real = binary_cross_entropy(tf.ones_like(d_real), d_real)\n",
    "loss_d_fake = binary_cross_entropy(tf.zeros_like(d_fake), d_fake)\n",
    "loss_g = tf.reduce_mean(binary_cross_entropy(tf.ones_like(d_fake), d_fake))\n",
    "loss_d = tf.reduce_mean(0.5 * (loss_d_real + loss_d_fake))\n",
    "\n",
    "update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "with tf.control_dependencies(update_ops):\n",
    "    optimizer_d = tf.train.RMSPropOptimizer(learning_rate=0.00015).minimize(loss_d + d_reg, var_list=vars_d)\n",
    "    optimizer_g = tf.train.RMSPropOptimizer(learning_rate=0.00015).minimize(loss_g + g_reg, var_list=vars_g)\n",
    "    \n",
    "    \n",
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the DCGAN\n",
    "Finally, the fun part begins--let's train our network! \n",
    "We feed random values to our generator, which will learn to create digits out of this noise. We also take care that neither the generator nor the discriminator becomes too strong--otherwise, this would inhibit the learning of the other part and could even stop the network from learning anything at all (I unfortunately have made this experience)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(60000):\n",
    "    train_d = True\n",
    "    train_g = True\n",
    "    keep_prob_train = 0.6 # 0.5\n",
    "    \n",
    "    \n",
    "    n = np.random.uniform(0.0, 1.0, [batch_size, n_noise]).astype(np.float32)   \n",
    "    batch = [np.reshape(b, [28, 28]) for b in mnist.train.next_batch(batch_size=batch_size)[0]]  \n",
    "    \n",
    "    d_real_ls, d_fake_ls, g_ls, d_ls = sess.run([loss_d_real, loss_d_fake, loss_g, loss_d], feed_dict={X_in: batch, noise: n, keep_prob: keep_prob_train, is_training:True})\n",
    "    \n",
    "    d_real_ls = np.mean(d_real_ls)\n",
    "    d_fake_ls = np.mean(d_fake_ls)\n",
    "    g_ls = g_ls\n",
    "    d_ls = d_ls\n",
    "    \n",
    "    if g_ls * 1.5 < d_ls:\n",
    "        train_g = False\n",
    "        pass\n",
    "    if d_ls * 2 < g_ls:\n",
    "        train_d = False\n",
    "        pass\n",
    "    \n",
    "    if train_d:\n",
    "        sess.run(optimizer_d, feed_dict={noise: n, X_in: batch, keep_prob: keep_prob_train, is_training:True})\n",
    "        \n",
    "        \n",
    "    if train_g:\n",
    "        sess.run(optimizer_g, feed_dict={noise: n, keep_prob: keep_prob_train, is_training:True})\n",
    "        \n",
    "        \n",
    "    if not i % 50:\n",
    "        print (i, d_ls, g_ls, d_real_ls, d_fake_ls)\n",
    "        if not train_g:\n",
    "            print(\"not training generator\")\n",
    "        if not train_d:\n",
    "            print(\"not training discriminator\")\n",
    "        gen_img = sess.run(g, feed_dict = {noise: n, keep_prob: 1.0, is_training:False})\n",
    "        imgs = [img[:,:,0] for img in gen_img]\n",
    "        m = montage(imgs)\n",
    "        gen_img = m\n",
    "        plt.axis('off')\n",
    "        plt.imshow(gen_img, cmap='gray')\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Results\n",
    "Take a look at the pictures drawn by our generator--they look more realistic than the pictures drawn by the VAE, which looked more fuzzy at their edges. Training however took much longer than training the other model.\n",
    "\n",
    "In conclusion, training the DCGAN took me much longer than training the VAE. Maybe fine-tuning the architecture could speed up the network's learning. Nonetheless, it's a real advantage that we are not dependent on loss functions based on pixel positions, making the results look less fuzzy. This is especially important when creating more complex data--e.g. pictures of human faces. So, just be a little patient--then everything is possible in the world of deep learning!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}