Intro mxnet/Intro mxnet NDArray, Symbol, Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Intro mxnet NDArray, Symbol, Model**\n",
    "* 목표 : \n",
    "  * mxnet의 NDArray 실습해보기\n",
    "  * Symbolic model\n",
    "  * mx.mod.Module을 이용한 훈련하기\n",
    "* 출처 : https://mxnet.incubator.apache.org/tutorials/basic/ndarray.html / http://www.popit.kr/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **1. Intro MXNet NDArray**\n",
    "* NDArray : the core data structure for all mathematical computations.\n",
    "  * similar to **numpy.ndarray**\n",
    "* key advantages\n",
    "  * CPU, GPU, multi-GPU 연산\n",
    "  * 분산 학습 지원\n",
    "  * 멀티 클라이언트 프로그래밍 지원"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create an NDArray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "from mxnet import nd\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.  2.  3.]\n"
     ]
    }
   ],
   "source": [
    "# 1차원 배열 \n",
    "a = nd.array([1,2,3])\n",
    "print(a.asnumpy()) #mxnet -> numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a.shape': (3,), 'a.type': numpy.float32}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{'a.shape':a.shape,'a.type':a.dtype} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.int32'>\n"
     ]
    }
   ],
   "source": [
    "# 기본 배열 타입은 numpy.float32, 배열 생성시 배열 타입을 지정 가능(dtype=__)\n",
    "a = nd.array([1, 2, 3], dtype=np.int32)\n",
    "print(a.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.  3.]\n",
      " [ 2.  3.  4.]]\n"
     ]
    }
   ],
   "source": [
    "# 2차원 배열\n",
    "b = nd.array([[1,2,3], [2,3,4]])\n",
    "print(b.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'b.shape': (2, 3), 'b.type': numpy.float32}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{'b.shape':b.shape,'b.type':b.dtype}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### special initializers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[ 0.  0.  0.]\n",
       " [ 0.  0.  0.]]\n",
       "<NDArray 2x3 @cpu(0)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nd.zeros((2,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[ 1.  1.  1.]\n",
       " [ 1.  1.  1.]]\n",
       "<NDArray 2x3 @cpu(0)>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nd.ones((2,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[ 7.  7.  7.]\n",
       " [ 7.  7.  7.]]\n",
       "<NDArray 2x3 @cpu(0)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nd.full((2,3), 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[ 0.23107235  0.30030754 -0.32433933  1.04932892]\n",
       " [ 0.73686236 -0.0097888   0.46656415  1.72023427]\n",
       " [ 0.87809837 -1.07333767 -0.8692565  -0.26717702]]\n",
       "<NDArray 3x4 @cpu(0)>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#random value : a standard normal distribution with zero mean and unit variance.\n",
    "nd.random_normal(0, 1, shape=(3, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reshape \n",
    ": 원본 메모리의 데이터는 그대로 둔 채 형태만 변경"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0.   1.   2.   3.   4.   5.   6.   7.   8.   9.  10.  11.  12.  13.  14.\n",
      "  15.  16.  17.  18.  19.  20.  21.  22.  23.]\n"
     ]
    }
   ],
   "source": [
    "a = nd.array(np.arange(24))\n",
    "print(a.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[  0.   1.   2.   3.]\n",
      "  [  4.   5.   6.   7.]\n",
      "  [  8.   9.  10.  11.]]\n",
      "\n",
      " [[ 12.  13.  14.  15.]\n",
      "  [ 16.  17.  18.  19.]\n",
      "  [ 20.  21.  22.  23.]]]\n"
     ]
    }
   ],
   "source": [
    "b = a.reshape((2, 3, 4)) #channel, row, col\n",
    "print(b.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 2.  2.  2.]\n",
      " [ 2.  2.  2.]]\n",
      "[[ 0.  0.  0.]\n",
      " [ 0.  0.  0.]]\n"
     ]
    }
   ],
   "source": [
    "a = nd.ones((2,3))\n",
    "b = nd.ones((2,3))\n",
    "c = a + b\n",
    "print(c.asnumpy())\n",
    "d = a - b\n",
    "print(d.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.,  2.,  2.],\n",
       "       [ 2.,  2.,  2.]], dtype=float32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# elementwise pow and sin, and then transpose\n",
    "d = nd.sin(c**2).T\n",
    "# elementwise max\n",
    "f = nd.maximum(a, c)\n",
    "f.asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing and Slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  1.]\n",
      " [ 2.  3.]\n",
      " [ 4.  5.]]\n"
     ]
    }
   ],
   "source": [
    "a = nd.array(np.arange(6).reshape(3, 2))\n",
    "print(a.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[ 1.]\n",
       "<NDArray 1 @cpu(0)>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# indexing\n",
    "a[0][1]  #(1,2)요소"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  1.],\n",
       "       [ 1.,  1.],\n",
       "       [ 4.,  5.]], dtype=float32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[1:2] = 1 #2nd row에 1 입력\n",
    "a[:].asnumpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  1.]]\n"
     ]
    }
   ],
   "source": [
    "# slicing\n",
    "print(a[1:2].asnumpy()) #2nd row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  1.]], dtype=float32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# slice with particular axis(axis=0)\n",
    "d = nd.slice_axis(a, axis=0, begin=1, end=2) \n",
    "d.asnumpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.],\n",
       "       [ 1.],\n",
       "       [ 5.]], dtype=float32)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using another axis(axis=1)\n",
    "d = nd.slice_axis(a, axis=1, begin=1, end=2)\n",
    "d.asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### concatenate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  1.  1.]\n",
      " [ 1.  1.  1.]]\n",
      "[[ 2.  2.  2.]\n",
      " [ 2.  2.  2.]]\n"
     ]
    }
   ],
   "source": [
    "a = nd.ones((2, 3))\n",
    "b = nd.ones((2, 3)) * 2\n",
    "print(a.asnumpy())\n",
    "print(b.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  1.  1.]\n",
      " [ 1.  1.  1.]\n",
      " [ 2.  2.  2.]\n",
      " [ 2.  2.  2.]]\n"
     ]
    }
   ],
   "source": [
    "c = nd.concatenate([a, b]) #기본 axis는 0(세로 결합)\n",
    "print(c.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  1.  1.  2.  2.  2.]\n",
      " [ 1.  1.  1.  2.  2.  2.]]\n"
     ]
    }
   ],
   "source": [
    "# concatenate with a particular axis(가로 결합)\n",
    "c = nd.concatenate([a, b], axis=1)\n",
    "print(c.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reduce : scalar 값으로 변환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  1.  1.]\n",
      " [ 1.  1.  1.]]\n"
     ]
    }
   ],
   "source": [
    "a = nd.ones((2, 3))\n",
    "print(a.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 6.]\n"
     ]
    }
   ],
   "source": [
    "b = nd.sum(a)\n",
    "print(b.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 2.  2.  2.]\n"
     ]
    }
   ],
   "source": [
    "# reduce along a particular axis\n",
    "c = nd.sum_axis(a, axis=0) # 세로 합\n",
    "print(c.asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 3.  3.]\n"
     ]
    }
   ],
   "source": [
    "c = mx.nd.sum_axis(a, axis=1) # 가로 합\n",
    "print(c.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU Support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "MXNetError",
     "evalue": "[23:44:10] src/imperative/imperative.cc:78: Operator _ones is not implemented for GPU.\n\nStack trace returned 6 entries:\n[bt] (0) 0   libmxnet.so                         0x00000001054f5b98 _ZN4dmlc15LogMessageFatalD2Ev + 40\n[bt] (1) 1   libmxnet.so                         0x000000010646bcd4 _ZN5mxnet10Imperative8InvokeOpERKNS_7ContextERKN4nnvm9NodeAttrsERKNSt3__16vectorIPNS_7NDArrayENS8_9allocatorISB_EEEESG_RKNS9_INS_9OpReqTypeENSC_ISH_EEEENS_12DispatchModeENS_10OpStatePtrE + 1972\n[bt] (2) 2   libmxnet.so                         0x000000010646f8da _ZN5mxnet10Imperative6InvokeERKNS_7ContextERKN4nnvm9NodeAttrsERKNSt3__16vectorIPNS_7NDArrayENS8_9allocatorISB_EEEESG_ + 794\n[bt] (3) 3   libmxnet.so                         0x00000001063f0667 _Z22MXImperativeInvokeImplPviPS_PiPS0_iPPKcS5_ + 391\n[bt] (4) 4   libmxnet.so                         0x00000001063f1834 MXImperativeInvokeEx + 164\n[bt] (5) 5   _ctypes.cpython-36m-darwin.so       0x00000001045df2c7 ffi_call_unix64 + 79\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mMXNetError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-29-43e7da3b77e1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# CPU/GPU간 스위치는 context를 기반으로 하며, 여러개인 경우 CPU/GPU 숫자를 입력한다.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hyemin/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py\u001b[0m in \u001b[0;36mones\u001b[0;34m(shape, ctx, dtype, **kwargs)\u001b[0m\n\u001b[1;32m   2175\u001b[0m     \u001b[0mdtype\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmx_real_t\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdtype\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2176\u001b[0m     \u001b[0;31m# pylint: disable= no-member, protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2177\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0m_internal\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mctx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2178\u001b[0m     \u001b[0;31m# pylint: enable= no-member, protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2179\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hyemin/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/register.py\u001b[0m in \u001b[0;36m_ones\u001b[0;34m(shape, ctx, dtype, out, name, **kwargs)\u001b[0m\n",
      "\u001b[0;32m/Users/hyemin/anaconda3/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py\u001b[0m in \u001b[0;36m_imperative_invoke\u001b[0;34m(handle, ndargs, keys, vals, out)\u001b[0m\n\u001b[1;32m     90\u001b[0m         \u001b[0mc_str_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m         \u001b[0mc_str_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ms\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvals\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 92\u001b[0;31m         ctypes.byref(out_stypes)))\n\u001b[0m\u001b[1;32m     93\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0moriginal_output\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hyemin/anaconda3/lib/python3.6/site-packages/mxnet/base.py\u001b[0m in \u001b[0;36mcheck_call\u001b[0;34m(ret)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \"\"\"\n\u001b[1;32m    145\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mret\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mMXNetError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_str\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_LIB\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMXGetLastError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mMXNetError\u001b[0m: [23:44:10] src/imperative/imperative.cc:78: Operator _ones is not implemented for GPU.\n\nStack trace returned 6 entries:\n[bt] (0) 0   libmxnet.so                         0x00000001054f5b98 _ZN4dmlc15LogMessageFatalD2Ev + 40\n[bt] (1) 1   libmxnet.so                         0x000000010646bcd4 _ZN5mxnet10Imperative8InvokeOpERKNS_7ContextERKN4nnvm9NodeAttrsERKNSt3__16vectorIPNS_7NDArrayENS8_9allocatorISB_EEEESG_RKNS9_INS_9OpReqTypeENSC_ISH_EEEENS_12DispatchModeENS_10OpStatePtrE + 1972\n[bt] (2) 2   libmxnet.so                         0x000000010646f8da _ZN5mxnet10Imperative6InvokeERKNS_7ContextERKN4nnvm9NodeAttrsERKNSt3__16vectorIPNS_7NDArrayENS8_9allocatorISB_EEEESG_ + 794\n[bt] (3) 3   libmxnet.so                         0x00000001063f0667 _Z22MXImperativeInvokeImplPviPS_PiPS0_iPPKcS5_ + 391\n[bt] (4) 4   libmxnet.so                         0x00000001063f1834 MXImperativeInvokeEx + 164\n[bt] (5) 5   _ctypes.cpython-36m-darwin.so       0x00000001045df2c7 ffi_call_unix64 + 79\n"
     ]
    }
   ],
   "source": [
    "# CPU/GPU간 스위치는 context를 기반으로 하며, 여러개인 경우 CPU/GPU 숫자를 입력\n",
    "# -> Google Cloud Platform 이용하기\n",
    "a = nd.ones((100, 100), mx.cpu(0))\n",
    "b = nd.ones((100, 100), mx.gpu(0))\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **2. Symbol - Neural network graphs and auto-differentiation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* MXNet provides the **Symbol API**, an interface for symbolic programming.\n",
    "* MXNet’s Symbol API is similar to the network configurations used by **Caffe** and the symbolic programming in **Theano.**\n",
    "* symbolic approach : we can optimize our functions before using them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imperative vs. Symbolic\n",
    "* mxnet은 imperative 모델과 Symbolic 모델 모두 가능"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1) Imperative model\n",
    "* 장점) 기반 프로그래밍 모델의 구문을 모두 활용 가능하여 좀 더 유연하며, 디버깅이 용이"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 3.  5.  7.]\n"
     ]
    }
   ],
   "source": [
    "# Imperative 모델은 일반적인 프로그래밍 모델\n",
    "a = nd.array([1, 2, 3])\n",
    "b = a * 2 + 1\n",
    "print(b.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2) Symbolic model\n",
    "* (1) 심볼을 이용하여 오퍼레이션에 대한 그래프를 먼저 정의한 후,\n",
    "* (2) 그래프에 실제값을 바인딩하여 실행시키는 과정을 거침. \n",
    "* 즉, 오퍼레이션을 정의하는 과정과 오퍼레이션을 실행하는 단계가 분리\n",
    "----\n",
    "* 단점) 기반 프로그래밍 요소를 사용할 수 없으며, 주어진 심볼릭 구분만을 이용해야 하므로 제약이 따른다.\n",
    "* 장점) 메모리 재사용이 가능하고 실행 과정이 최적화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Symbol a>\n",
      "<Symbol _plusscalar0>\n"
     ]
    }
   ],
   "source": [
    "#위 내용을 symbolic 코드로 구현\n",
    "#1)먼저 심볼을 기반으로 computation graph를 구성\n",
    "a = mx.sym.Variable('a')\n",
    "b = a * 2 + 1\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"102pt\" height=\"254pt\"\n",
       " viewBox=\"0.00 0.00 102.00 254.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 250)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-250 98,-250 98,4 -4,4\"/>\n",
       "<!-- a -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>a</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"#000000\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-24.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">a</text>\n",
       "</g>\n",
       "<!-- _mulscalar0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>_mulscalar0</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"#000000\" points=\"94,-152 0,-152 0,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-118.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">_mulscalar0</text>\n",
       "</g>\n",
       "<!-- _mulscalar0&#45;&gt;a -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_mulscalar0&#45;&gt;a</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-83.6321C47,-75.1148 47,-66.2539 47,-58.2088\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-93.7731 42.5001,-83.773 47,-88.7731 47.0001,-83.7731 47.0001,-83.7731 47.0001,-83.7731 47,-88.7731 51.5001,-83.7731 47,-93.7731 47,-93.7731\"/>\n",
       "</g>\n",
       "<!-- _plusscalar0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>_plusscalar0</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"#000000\" points=\"94,-246 0,-246 0,-188 94,-188 94,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-212.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">_plusscalar0</text>\n",
       "</g>\n",
       "<!-- _plusscalar0&#45;&gt;_mulscalar0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>_plusscalar0&#45;&gt;_mulscalar0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-177.6321C47,-169.1148 47,-160.2539 47,-152.2088\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-187.7731 42.5001,-177.773 47,-182.7731 47.0001,-177.7731 47.0001,-177.7731 47.0001,-177.7731 47,-182.7731 51.5001,-177.7731 47,-187.7731 47,-187.7731\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x11199fcc0>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Plot computation graph\n",
    "mx.viz.plot_network(symbol=b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\n",
       " [ 3.  5.  7.]\n",
       " <NDArray 3 @cpu(0)>]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#2) 그래프에 실제 값은 bind\n",
    "ex = b.bind(ctx=mx.cpu(), args={'a': nd.array([1, 2, 3])}) #a값 입력\n",
    "# Execute computation graph\n",
    "ex.forward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "[ 3.  5.  7.]\n",
      "<NDArray 3 @cpu(0)>]\n"
     ]
    }
   ],
   "source": [
    "print(ex.outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 3.  5.  7.]\n"
     ]
    }
   ],
   "source": [
    "print(ex.outputs[0].asnumpy()) #imperative 모델과 동일한 결과"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Neural Networks 만들기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"498pt\"\n",
       " viewBox=\"0.00 0.00 214.00 498.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 494)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-494 210,-494 210,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"#000000\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-24.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">data</text>\n",
       "</g>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"#000000\" points=\"94,-166 0,-166 0,-108 94,-108 94,-166\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-139.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">128</text>\n",
       "</g>\n",
       "<!-- fc1&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>fc1&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-97.6567C47,-84.6329 47,-70.3785 47,-58.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-107.7736 42.5001,-97.7736 47,-102.7736 47.0001,-97.7736 47.0001,-97.7736 47.0001,-97.7736 47,-102.7736 51.5001,-97.7737 47,-107.7736 47,-107.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-78.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">200</text>\n",
       "</g>\n",
       "<!-- relu1 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>relu1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"#000000\" points=\"94,-274 0,-274 0,-216 94,-216 94,-274\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-247.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-233.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">relu</text>\n",
       "</g>\n",
       "<!-- relu1&#45;&gt;fc1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>relu1&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-205.6567C47,-192.6329 47,-178.3785 47,-166.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-215.7736 42.5001,-205.7736 47,-210.7736 47.0001,-205.7736 47.0001,-205.7736 47.0001,-205.7736 47,-210.7736 51.5001,-205.7737 47,-215.7736 47,-215.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-186.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">128</text>\n",
       "</g>\n",
       "<!-- fc2 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>fc2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"#000000\" points=\"94,-382 0,-382 0,-324 94,-324 94,-382\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-355.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-341.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">10</text>\n",
       "</g>\n",
       "<!-- fc2&#45;&gt;relu1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>fc2&#45;&gt;relu1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-313.6567C47,-300.6329 47,-286.3785 47,-274.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-323.7736 42.5001,-313.7736 47,-318.7736 47.0001,-313.7736 47.0001,-313.7736 47.0001,-313.7736 47,-318.7736 51.5001,-313.7737 47,-323.7736 47,-323.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-294.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">128</text>\n",
       "</g>\n",
       "<!-- out_label -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>out_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"#000000\" cx=\"159\" cy=\"-353\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-348.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">out_label</text>\n",
       "</g>\n",
       "<!-- out -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>out</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"#000000\" points=\"170,-490 76,-490 76,-432 170,-432 170,-490\"/>\n",
       "<text text-anchor=\"middle\" x=\"123\" y=\"-456.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">out</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;fc2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>out&#45;&gt;fc2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M96.6685,-423.5815C87.1459,-410.0495 76.5407,-394.9789 67.5831,-382.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"102.4333,-431.7736 92.9982,-426.1853 99.5558,-427.6846 96.6783,-423.5956 96.6783,-423.5956 96.6783,-423.5956 99.5558,-427.6846 100.3585,-421.0058 102.4333,-431.7736 102.4333,-431.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"97\" y=\"-402.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">10</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;out_label -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>out&#45;&gt;out_label</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M135.9092,-422.2723C140.3948,-408.8157 145.3399,-393.9804 149.4991,-381.5026\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"132.7421,-431.7736 131.6354,-420.8638 134.3233,-427.0302 135.9045,-422.2868 135.9045,-422.2868 135.9045,-422.2868 134.3233,-427.0302 140.1735,-423.7099 132.7421,-431.7736 132.7421,-431.7736\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1119af550>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#2개의 레이어로 구성된 MLP(Multi Layer Perceptron)을 구성\n",
    "#(error 발생시, 'brew install graphviz' 실행)\n",
    "net = mx.sym.Variable('data')                                        # 입력값 key\n",
    "net = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)    # FullyConnected layer\n",
    "net = mx.sym.Activation(data=net, name='relu1', act_type=\"relu\")     # Activataion function\n",
    "net = mx.sym.FullyConnected(data=net, name='fc2', num_hidden=10)     # FullyConnected layer\n",
    "net = mx.sym.SoftmaxOutput(data=net, name='out')                     # 모델의 예측값 key\n",
    "mx.viz.plot_network(net, shape={'data':(100,200)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data', 'fc1_weight', 'fc1_bias', 'fc2_weight', 'fc2_bias', 'out_label']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.list_arguments()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 모듈화 하기\n",
    "* CNN과 같은 신경망은 유사한 레이어가 반복적으로 필요.\n",
    "* 따라서 각 레이어를 하나씩 추가하기 보다는 모듈화하여 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Modulelized Construction for Deep Networks\n",
    "def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0),name=None, suffix=''):\n",
    "    conv = mx.sym.Convolution(data=data, num_filter=num_filter, kernel=kernel,\n",
    "                  stride=stride, pad=pad, name='conv_%s%s' %(name, suffix))\n",
    "    bn = mx.sym.BatchNorm(data=conv, name='bn_%s%s' %(name, suffix))\n",
    "    act = mx.sym.Activation(data=bn, act_type='relu', name='relu_%s%s'\n",
    "                  %(name, suffix))\n",
    "    return act"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"110pt\" height=\"714pt\"\n",
       " viewBox=\"0.00 0.00 109.97 714.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 710)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-710 105.9746,-710 105.9746,4 -4,4\"/>\n",
       "<!-- Previos Output -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>Previos Output</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"#000000\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-24.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Previos Output</text>\n",
       "</g>\n",
       "<!-- conv_Conv1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>conv_Conv1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"#000000\" points=\"94,-166 0,-166 0,-108 94,-108 94,-166\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-139.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">7x7/2x2, 64</text>\n",
       "</g>\n",
       "<!-- conv_Conv1&#45;&gt;Previos Output -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>conv_Conv1&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-97.6567C47,-84.6329 47,-70.3785 47,-58.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-107.7736 42.5001,-97.7736 47,-102.7736 47.0001,-97.7736 47.0001,-97.7736 47.0001,-97.7736 47,-102.7736 51.5001,-97.7737 47,-107.7736 47,-107.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"71.5\" y=\"-78.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_Conv1 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>bn_Conv1</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"#000000\" points=\"94,-274 0,-274 0,-216 94,-216 94,-274\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-240.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bn_Conv1</text>\n",
       "</g>\n",
       "<!-- bn_Conv1&#45;&gt;conv_Conv1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>bn_Conv1&#45;&gt;conv_Conv1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-205.6567C47,-192.6329 47,-178.3785 47,-166.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-215.7736 42.5001,-205.7736 47,-210.7736 47.0001,-205.7736 47.0001,-205.7736 47.0001,-205.7736 47,-210.7736 51.5001,-205.7737 47,-215.7736 47,-215.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.4873\" y=\"-186.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">64x11x11</text>\n",
       "</g>\n",
       "<!-- relu_Conv1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>relu_Conv1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"#000000\" points=\"94,-382 0,-382 0,-324 94,-324 94,-382\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-355.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-341.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">relu</text>\n",
       "</g>\n",
       "<!-- relu_Conv1&#45;&gt;bn_Conv1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>relu_Conv1&#45;&gt;bn_Conv1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-313.6567C47,-300.6329 47,-286.3785 47,-274.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-323.7736 42.5001,-313.7736 47,-318.7736 47.0001,-313.7736 47.0001,-313.7736 47.0001,-313.7736 47,-318.7736 51.5001,-313.7737 47,-323.7736 47,-323.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.4873\" y=\"-294.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">64x11x11</text>\n",
       "</g>\n",
       "<!-- conv_Conv2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>conv_Conv2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"#000000\" points=\"94,-490 0,-490 0,-432 94,-432 94,-490\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-463.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-449.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">7x7/2x2, 32</text>\n",
       "</g>\n",
       "<!-- conv_Conv2&#45;&gt;relu_Conv1 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>conv_Conv2&#45;&gt;relu_Conv1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-421.6567C47,-408.6329 47,-394.3785 47,-382.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-431.7736 42.5001,-421.7736 47,-426.7736 47.0001,-421.7736 47.0001,-421.7736 47.0001,-421.7736 47,-426.7736 51.5001,-421.7737 47,-431.7736 47,-431.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.4873\" y=\"-402.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">64x11x11</text>\n",
       "</g>\n",
       "<!-- bn_Conv2 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>bn_Conv2</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"#000000\" points=\"94,-598 0,-598 0,-540 94,-540 94,-598\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-564.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">bn_Conv2</text>\n",
       "</g>\n",
       "<!-- bn_Conv2&#45;&gt;conv_Conv2 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>bn_Conv2&#45;&gt;conv_Conv2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-529.6567C47,-516.6329 47,-502.3785 47,-490.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-539.7736 42.5001,-529.7736 47,-534.7736 47.0001,-529.7736 47.0001,-529.7736 47.0001,-529.7736 47,-534.7736 51.5001,-529.7737 47,-539.7736 47,-539.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"68\" y=\"-510.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">32x3x3</text>\n",
       "</g>\n",
       "<!-- relu_Conv2 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>relu_Conv2</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"#000000\" points=\"94,-706 0,-706 0,-648 94,-648 94,-706\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-679.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-665.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">relu</text>\n",
       "</g>\n",
       "<!-- relu_Conv2&#45;&gt;bn_Conv2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>relu_Conv2&#45;&gt;bn_Conv2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-637.6567C47,-624.6329 47,-610.3785 47,-598.2497\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-647.7736 42.5001,-637.7736 47,-642.7736 47.0001,-637.7736 47.0001,-637.7736 47.0001,-637.7736 47,-642.7736 51.5001,-637.7737 47,-647.7736 47,-647.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"68\" y=\"-618.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">32x3x3</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1119bb518>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#ConvFactory() 함수를 이용하여 2개의 Convolution 레이어로 구성된 네트워크를 아래와 같이 구성\n",
    "prev = mx.symbol.Variable(name=\"Previos Output\")\n",
    "conv_comp1 = ConvFactory(data=prev, num_filter=64, kernel=(7, 7), stride=(2, 2), name='Conv1')\n",
    "conv_comp2 = ConvFactory(data=conv_comp1, num_filter=32, kernel=(7, 7), stride=(2, 2), name='Conv2')\n",
    "shape = {\"Previos Output\": (128, 3, 28, 28)}\n",
    "mx.viz.plot_network(symbol=conv_comp2, shape=shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### computation graph 저장 및 로드\n",
    "* 구성한 그래프는 JSON 형태로 저장하고, 로드하여 재사용 가능"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "# Load and save computation graph\n",
    "a = mx.sym.Variable('a')\n",
    "b = mx.sym.Variable('b')\n",
    "c = a + b\n",
    "c.save('symbol-b.json') #JSON 형태로 저장\n",
    "c2 = mx.symbol.load('symbol-b.json') #모델 로드\n",
    "print(c.tojson() == c2.tojson())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Module - Neural network training and inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### mx.mod.Module을 이용한 훈련하기\n",
    ": 텐서플로우와 마찬가지로 저수준 API와 고수준 API 제공"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 그래프 구성하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:letter-recognition.data exists, skipping download\n"
     ]
    }
   ],
   "source": [
    "# 먼저 학습할 데이터셋을 로드한다.\n",
    "# data: Letter Image Recognition Data(26 classes, A~Z)\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "import mxnet as mx\n",
    "import numpy as np\n",
    "\n",
    "fname = mx.test_utils.download('http://archive.ics.uci.edu/ml/machine-learning-databases/letter-recognition/letter-recognition.data')\n",
    "data = np.genfromtxt(fname, delimiter=',')[:,1:]\n",
    "label = np.array([ord(l.split(',')[0])-ord('A') for l in open(fname, 'r')])\n",
    "\n",
    "batch_size = 32\n",
    "ntrain = int(data.shape[0]*0.8) #train:val = 8:2\n",
    "\n",
    "#NDArrayIter : 데이터 Batch Iterator 생성\n",
    "train_iter = mx.io.NDArrayIter(data[:ntrain, :], label[:ntrain], batch_size, shuffle=True) \n",
    "val_iter = mx.io.NDArrayIter(data[ntrain:, :], label[ntrain:], batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  2.,   8.,   3., ...,   8.,   0.,   8.],\n",
       "       [  5.,  12.,   3., ...,   8.,   4.,  10.],\n",
       "       [  4.,  11.,   6., ...,   7.,   3.,   9.],\n",
       "       ..., \n",
       "       [  6.,   9.,   6., ...,  12.,   2.,   4.],\n",
       "       [  2.,   3.,   4., ...,   9.,   5.,   8.],\n",
       "       [  4.,   9.,   6., ...,   7.,   2.,   8.]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20000, 16)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"442pt\"\n",
       " viewBox=\"0.00 0.00 214.00 442.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 438)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-438 210,-438 210,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"#000000\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-24.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">data</text>\n",
       "</g>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"#000000\" points=\"94,-152 0,-152 0,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">64</text>\n",
       "</g>\n",
       "<!-- fc1&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>fc1&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-83.6321C47,-75.1148 47,-66.2539 47,-58.2088\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-93.7731 42.5001,-83.773 47,-88.7731 47.0001,-83.7731 47.0001,-83.7731 47.0001,-83.7731 47,-88.7731 51.5001,-83.7731 47,-93.7731 47,-93.7731\"/>\n",
       "</g>\n",
       "<!-- relu1 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>relu1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"#000000\" points=\"94,-246 0,-246 0,-188 94,-188 94,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-219.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-205.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">relu</text>\n",
       "</g>\n",
       "<!-- relu1&#45;&gt;fc1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>relu1&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-177.6321C47,-169.1148 47,-160.2539 47,-152.2088\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-187.7731 42.5001,-177.773 47,-182.7731 47.0001,-177.7731 47.0001,-177.7731 47.0001,-177.7731 47,-182.7731 51.5001,-177.7731 47,-187.7731 47,-187.7731\"/>\n",
       "</g>\n",
       "<!-- fc2 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>fc2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"#000000\" points=\"94,-340 0,-340 0,-282 94,-282 94,-340\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-313.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">26</text>\n",
       "</g>\n",
       "<!-- fc2&#45;&gt;relu1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>fc2&#45;&gt;relu1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M47,-271.6321C47,-263.1148 47,-254.2539 47,-246.2088\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"47,-281.7731 42.5001,-271.773 47,-276.7731 47.0001,-271.7731 47.0001,-271.7731 47.0001,-271.7731 47,-276.7731 51.5001,-271.7731 47,-281.7731 47,-281.7731\"/>\n",
       "</g>\n",
       "<!-- softmax_label -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>softmax_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"#000000\" cx=\"159\" cy=\"-311\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-306.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">softmax_label</text>\n",
       "</g>\n",
       "<!-- softmax -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>softmax</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"#000000\" points=\"150,-434 56,-434 56,-376 150,-376 150,-434\"/>\n",
       "<text text-anchor=\"middle\" x=\"103\" y=\"-400.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">softmax</text>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;fc2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>softmax&#45;&gt;fc2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M80.3812,-367.0328C75.0542,-358.091 69.4571,-348.6959 64.401,-340.2088\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"85.5882,-375.7731 76.6042,-369.4852 83.0292,-371.4776 80.4701,-367.1821 80.4701,-367.1821 80.4701,-367.1821 83.0292,-371.4776 84.3361,-364.8789 85.5882,-375.7731 85.5882,-375.7731\"/>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;softmax_label -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>softmax&#45;&gt;softmax_label</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M125.5712,-367.1126C131.3182,-357.4659 137.3825,-347.2865 142.7449,-338.2854\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"120.4118,-375.7731 121.6639,-364.8789 122.9708,-371.4776 125.5299,-367.1821 125.5299,-367.1821 125.5299,-367.1821 122.9708,-371.4776 129.3958,-369.4852 120.4118,-375.7731 120.4118,-375.7731\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x11196f2b0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 신경망 그래프를 구성\n",
    "net = mx.sym.Variable('data')\n",
    "net = mx.sym.FullyConnected(net, name='fc1', num_hidden=64)\n",
    "net = mx.sym.Activation(net, name='relu1', act_type=\"relu\")\n",
    "net = mx.sym.FullyConnected(net, name='fc2', num_hidden=26)\n",
    "net = mx.sym.SoftmaxOutput(net, name='softmax')\n",
    "mx.viz.plot_network(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ‘data’: 입력값을 가리키는 key\n",
    "* ‘softmax_label’: 출력 레이블을 가리키는 key\n",
    "* ‘softmax_output’: 모델의 예측값을 가리키는 key\n",
    "* ‘opname_varname’: 모델의 파라미터를 가리키는 key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['data', 'fc1_weight', 'fc1_bias', 'fc2_weight', 'fc2_bias', 'softmax_label']\n",
      "['softmax_output']\n"
     ]
    }
   ],
   "source": [
    "print(net.list_arguments())\n",
    "print(net.list_outputs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Module : 아래의 상태 단계 별로 진행\n",
    "\n",
    "* 초기 상태 : \n",
    "  * 심볼 기반의 그래프만 존재하며 메모리가 할당되지 않은 상태\n",
    "  * 계산(computation)이 실행될 수 없는 상태\n",
    "* 데이터 바인딩 상태 :\n",
    "  * 입력, 출력, 파라미터 등에 대해 메모리가 할당된 상태\n",
    "  * 계산(computation)이 실행될 수 있는 상태\n",
    "* 파라미터 초기화 상태 : \n",
    "  * 훈련 과정의 최적화를 위해 파라미터를 적절한 값으로 초기화환 상태\n",
    "  * 옵티마이저 할당 상태\n",
    "* 옵티마이저가 할당된 상태 : \n",
    "  * 계산(computation)을 반복적으로 실행하여 파라미터를 업데이트 할 수 있는 상태"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Intermediate-level Interface\n",
    "* These APIs give developers **flexibility** to do step-by-step computation by running forward and backward passes. \n",
    "* It’s also useful for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## module 학습 위해서는 아래의 단계 거쳐야 함.\n",
    "#(1)Create Module\n",
    "# symbol : the network Symbol(신경망 그래프)\n",
    "# context : the device (or a list of devices) for execution(훈련 모드 : CPU/GPU)\n",
    "# data_names : the list of data variable names(훈련 데이터 X의 key 값)\n",
    "# label_names : the list of label variable names(훈련데이터 Y의 key 값(레이블))\n",
    "mod = mx.mod.Module(symbol=net,\n",
    "                    context=mx.cpu(),\n",
    "                    data_names=['data'],\n",
    "                    label_names=['softmax_label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# (2) 데이터 바인딩 : input data와 label에 따른 memory 할당\n",
    "mod.bind(data_shapes=train_iter.provide_data, label_shapes=train_iter.provide_label)\n",
    "# (3) 파라미터 초기화 상태 : Assigns and initializes parameters.\n",
    "mod.init_params(initializer=mx.init.Uniform(scale=.1))\n",
    "# (4) 옵티마이저 할당 : Initializes optimizers. Defaults to sgd.\n",
    "mod.init_optimizer(optimizer='sgd', optimizer_params=(('learning_rate', 0.1), ))\n",
    "# (5) Creates evaluation metric\n",
    "metric = mx.metric.create('acc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, Training ('accuracy', 0.44612499999999999)\n",
      "Epoch 1, Training ('accuracy', 0.65131249999999996)\n",
      "Epoch 2, Training ('accuracy', 0.71768750000000003)\n",
      "Epoch 3, Training ('accuracy', 0.75431250000000005)\n",
      "Epoch 4, Training ('accuracy', 0.7724375)\n",
      "Epoch 5, Training ('accuracy', 0.79525000000000001)\n",
      "Epoch 6, Training ('accuracy', 0.80925000000000002)\n",
      "Epoch 7, Training ('accuracy', 0.81925000000000003)\n",
      "Epoch 8, Training ('accuracy', 0.82706250000000003)\n",
      "Epoch 9, Training ('accuracy', 0.83618749999999997)\n"
     ]
    }
   ],
   "source": [
    "# train 10 epochs\n",
    "for epoch in range(10):\n",
    "    train_iter.reset()\n",
    "    metric.reset()\n",
    "    for batch in train_iter:\n",
    "        mod.forward(batch, is_train=True)       # Forward computation.\n",
    "        mod.update_metric(metric, batch.label)  # Evaluates and accumulates evaluation metric\n",
    "        mod.backward()                          # Backward computation.\n",
    "        mod.update()                            # update parameters\n",
    "    print('Epoch %d, Training %s' % (epoch, metric.get()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) High-level Interface\n",
    "* Module also provides high-level APIs for training, predicting and evaluating for **user convenience**\n",
    "* fit() 함수 이용"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (1) train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Train-accuracy=0.370188\n",
      "INFO:root:Epoch[0] Time cost=0.352\n",
      "INFO:root:Epoch[0] Validation-accuracy=0.617500\n",
      "INFO:root:Epoch[1] Train-accuracy=0.628750\n",
      "INFO:root:Epoch[1] Time cost=0.295\n",
      "INFO:root:Epoch[1] Validation-accuracy=0.690750\n",
      "INFO:root:Epoch[2] Train-accuracy=0.701375\n",
      "INFO:root:Epoch[2] Time cost=0.268\n",
      "INFO:root:Epoch[2] Validation-accuracy=0.766750\n",
      "INFO:root:Epoch[3] Train-accuracy=0.746250\n",
      "INFO:root:Epoch[3] Time cost=0.265\n",
      "INFO:root:Epoch[3] Validation-accuracy=0.773500\n",
      "INFO:root:Epoch[4] Train-accuracy=0.769563\n",
      "INFO:root:Epoch[4] Time cost=0.271\n",
      "INFO:root:Epoch[4] Validation-accuracy=0.779750\n",
      "INFO:root:Epoch[5] Train-accuracy=0.792188\n",
      "INFO:root:Epoch[5] Time cost=0.278\n",
      "INFO:root:Epoch[5] Validation-accuracy=0.776750\n",
      "INFO:root:Epoch[6] Train-accuracy=0.805063\n",
      "INFO:root:Epoch[6] Time cost=0.422\n",
      "INFO:root:Epoch[6] Validation-accuracy=0.803000\n",
      "INFO:root:Epoch[7] Train-accuracy=0.816312\n",
      "INFO:root:Epoch[7] Time cost=0.368\n",
      "INFO:root:Epoch[7] Validation-accuracy=0.809500\n",
      "INFO:root:Epoch[8] Train-accuracy=0.824187\n",
      "INFO:root:Epoch[8] Time cost=0.295\n",
      "INFO:root:Epoch[8] Validation-accuracy=0.793000\n",
      "INFO:root:Epoch[9] Train-accuracy=0.827875\n",
      "INFO:root:Epoch[9] Time cost=0.443\n",
      "INFO:root:Epoch[9] Validation-accuracy=0.797250\n"
     ]
    }
   ],
   "source": [
    "# reset train_iter to the beginning\n",
    "train_iter.reset()\n",
    "\n",
    "# create a module(동일)\n",
    "mod = mx.mod.Module(symbol=net,\n",
    "                    context=mx.cpu(),\n",
    "                    data_names=['data'],\n",
    "                    label_names=['softmax_label'])\n",
    "\n",
    "# fit the module\n",
    "mod.fit(train_iter,\n",
    "        eval_data=val_iter,\n",
    "        optimizer='sgd',\n",
    "        optimizer_params={'learning_rate':0.1},\n",
    "        eval_metric='acc',\n",
    "        num_epoch=10)\n",
    "# 각 에폭이 끝날 때 마다 validation set 의 accuracy도 계산됨."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (2) Predict and Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# predict\n",
    "y = mod.predict(val_iter)\n",
    "assert y.shape == (4000, 26)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4000, 26)\n"
     ]
    }
   ],
   "source": [
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  6.67226908e-04   7.58538628e-03   1.75329784e-04   3.97842057e-04\n",
      "   3.77875149e-05   1.96039127e-06   3.94954495e-02   3.33127985e-03\n",
      "   5.67132815e-07   3.14889485e-06   1.12999824e-03   5.79085099e-05\n",
      "   4.67386954e-02   4.52895678e-04   2.03699595e-03   2.99512548e-03\n",
      "   2.27701585e-05   9.66484775e-04   5.28426312e-07   1.48907895e-06\n",
      "   5.39539278e-01   4.13565896e-02   3.12916905e-01   2.03440376e-10\n",
      "   8.83944667e-05   1.17190202e-10]\n"
     ]
    }
   ],
   "source": [
    "print(y[0].asnumpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 0, accuracy 0.781250\n",
      "batch 1, accuracy 0.781250\n",
      "batch 2, accuracy 0.812500\n",
      "batch 3, accuracy 0.906250\n",
      "batch 4, accuracy 0.906250\n",
      "batch 5, accuracy 0.906250\n",
      "batch 6, accuracy 0.781250\n",
      "batch 7, accuracy 0.687500\n",
      "batch 8, accuracy 0.937500\n",
      "batch 9, accuracy 0.718750\n",
      "batch 10, accuracy 0.875000\n",
      "batch 11, accuracy 0.781250\n",
      "batch 12, accuracy 0.625000\n",
      "batch 13, accuracy 0.781250\n",
      "batch 14, accuracy 0.875000\n",
      "batch 15, accuracy 0.906250\n",
      "batch 16, accuracy 0.843750\n",
      "batch 17, accuracy 0.781250\n",
      "batch 18, accuracy 0.843750\n",
      "batch 19, accuracy 0.875000\n",
      "batch 20, accuracy 0.843750\n",
      "batch 21, accuracy 0.843750\n",
      "batch 22, accuracy 0.781250\n",
      "batch 23, accuracy 0.750000\n",
      "batch 24, accuracy 0.750000\n",
      "batch 25, accuracy 0.812500\n",
      "batch 26, accuracy 0.843750\n",
      "batch 27, accuracy 0.812500\n",
      "batch 28, accuracy 0.875000\n",
      "batch 29, accuracy 0.843750\n",
      "batch 30, accuracy 0.718750\n",
      "batch 31, accuracy 0.781250\n",
      "batch 32, accuracy 0.812500\n",
      "batch 33, accuracy 0.812500\n",
      "batch 34, accuracy 0.781250\n",
      "batch 35, accuracy 0.781250\n",
      "batch 36, accuracy 0.906250\n",
      "batch 37, accuracy 0.781250\n",
      "batch 38, accuracy 0.781250\n",
      "batch 39, accuracy 0.812500\n",
      "batch 40, accuracy 0.687500\n",
      "batch 41, accuracy 0.687500\n",
      "batch 42, accuracy 0.812500\n",
      "batch 43, accuracy 0.812500\n",
      "batch 44, accuracy 0.750000\n",
      "batch 45, accuracy 0.812500\n",
      "batch 46, accuracy 0.843750\n",
      "batch 47, accuracy 0.750000\n",
      "batch 48, accuracy 0.750000\n",
      "batch 49, accuracy 0.906250\n",
      "batch 50, accuracy 0.812500\n",
      "batch 51, accuracy 0.937500\n",
      "batch 52, accuracy 0.750000\n",
      "batch 53, accuracy 0.812500\n",
      "batch 54, accuracy 0.812500\n",
      "batch 55, accuracy 0.656250\n",
      "batch 56, accuracy 0.906250\n",
      "batch 57, accuracy 0.593750\n",
      "batch 58, accuracy 0.812500\n",
      "batch 59, accuracy 0.718750\n",
      "batch 60, accuracy 0.812500\n",
      "batch 61, accuracy 0.875000\n",
      "batch 62, accuracy 0.687500\n",
      "batch 63, accuracy 0.750000\n",
      "batch 64, accuracy 0.812500\n",
      "batch 65, accuracy 0.812500\n",
      "batch 66, accuracy 0.718750\n",
      "batch 67, accuracy 0.843750\n",
      "batch 68, accuracy 0.718750\n",
      "batch 69, accuracy 0.937500\n",
      "batch 70, accuracy 0.750000\n",
      "batch 71, accuracy 0.843750\n",
      "batch 72, accuracy 0.750000\n",
      "batch 73, accuracy 0.687500\n",
      "batch 74, accuracy 0.937500\n",
      "batch 75, accuracy 0.937500\n",
      "batch 76, accuracy 0.812500\n",
      "batch 77, accuracy 0.750000\n",
      "batch 78, accuracy 0.812500\n",
      "batch 79, accuracy 0.843750\n",
      "batch 80, accuracy 0.750000\n",
      "batch 81, accuracy 0.906250\n",
      "batch 82, accuracy 0.906250\n",
      "batch 83, accuracy 0.781250\n",
      "batch 84, accuracy 0.843750\n",
      "batch 85, accuracy 0.718750\n",
      "batch 86, accuracy 0.875000\n",
      "batch 87, accuracy 0.750000\n",
      "batch 88, accuracy 0.687500\n",
      "batch 89, accuracy 0.843750\n",
      "batch 90, accuracy 0.906250\n",
      "batch 91, accuracy 0.843750\n",
      "batch 92, accuracy 0.843750\n",
      "batch 93, accuracy 0.718750\n",
      "batch 94, accuracy 0.718750\n",
      "batch 95, accuracy 0.937500\n",
      "batch 96, accuracy 0.656250\n",
      "batch 97, accuracy 0.718750\n",
      "batch 98, accuracy 0.781250\n",
      "batch 99, accuracy 0.781250\n",
      "batch 100, accuracy 0.812500\n",
      "batch 101, accuracy 0.781250\n",
      "batch 102, accuracy 0.718750\n",
      "batch 103, accuracy 0.750000\n",
      "batch 104, accuracy 0.875000\n",
      "batch 105, accuracy 0.750000\n",
      "batch 106, accuracy 0.875000\n",
      "batch 107, accuracy 0.875000\n",
      "batch 108, accuracy 0.812500\n",
      "batch 109, accuracy 0.843750\n",
      "batch 110, accuracy 0.781250\n",
      "batch 111, accuracy 0.843750\n",
      "batch 112, accuracy 0.656250\n",
      "batch 113, accuracy 0.656250\n",
      "batch 114, accuracy 0.812500\n",
      "batch 115, accuracy 0.750000\n",
      "batch 116, accuracy 0.750000\n",
      "batch 117, accuracy 0.812500\n",
      "batch 118, accuracy 0.812500\n",
      "batch 119, accuracy 0.687500\n",
      "batch 120, accuracy 0.718750\n",
      "batch 121, accuracy 0.750000\n",
      "batch 122, accuracy 0.812500\n",
      "batch 123, accuracy 0.718750\n",
      "batch 124, accuracy 0.781250\n"
     ]
    }
   ],
   "source": [
    "# Predcit iteratively(Batch별 metric)\n",
    "for preds, i_batch, batch in mod.iter_predict(val_iter):\n",
    "    pred_label = preds[0].asnumpy().argmax(axis=1)\n",
    "    label = batch.label[0].asnumpy().astype('int32')\n",
    "    print('batch %d, accuracy %f' % (i_batch, float(sum(pred_label==label))/len(label)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mse', 211.26472070312499), ('accuracy', 0.79725000000000001)]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluation metric: top_k_acc(top-k-accuracy), F1, RMSE, MSE, MAE, ce(CrossEntropy).\n",
    "score = mod.score(val_iter, ['mse','acc']) \n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (3)  Save and Load\n",
    "* 학습이 진행되는 동안 각 에폭별로 체크포인트 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Train-accuracy=0.084500\n",
      "INFO:root:Epoch[0] Time cost=0.305\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0001.params\"\n",
      "INFO:root:Epoch[1] Train-accuracy=0.270875\n",
      "INFO:root:Epoch[1] Time cost=0.338\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0002.params\"\n",
      "INFO:root:Epoch[2] Train-accuracy=0.440437\n",
      "INFO:root:Epoch[2] Time cost=0.254\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0003.params\"\n",
      "INFO:root:Epoch[3] Train-accuracy=0.551312\n",
      "INFO:root:Epoch[3] Time cost=0.249\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0004.params\"\n",
      "INFO:root:Epoch[4] Train-accuracy=0.614750\n",
      "INFO:root:Epoch[4] Time cost=0.248\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0005.params\"\n",
      "INFO:root:Epoch[5] Train-accuracy=0.651375\n",
      "INFO:root:Epoch[5] Time cost=0.249\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0006.params\"\n",
      "INFO:root:Epoch[6] Train-accuracy=0.679375\n",
      "INFO:root:Epoch[6] Time cost=0.257\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0007.params\"\n",
      "INFO:root:Epoch[7] Train-accuracy=0.697250\n",
      "INFO:root:Epoch[7] Time cost=0.245\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0008.params\"\n",
      "INFO:root:Epoch[8] Train-accuracy=0.710187\n",
      "INFO:root:Epoch[8] Time cost=0.248\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0009.params\"\n",
      "INFO:root:Epoch[9] Train-accuracy=0.720625\n",
      "INFO:root:Epoch[9] Time cost=0.253\n",
      "INFO:root:Saved checkpoint to \"mx_mlp-0010.params\"\n"
     ]
    }
   ],
   "source": [
    "# model save\n",
    "# construct a callback function to save checkpoints\n",
    "model_prefix = 'mx_mlp'\n",
    "checkpoint = mx.callback.do_checkpoint(model_prefix)\n",
    "\n",
    "mod = mx.mod.Module(symbol=net)\n",
    "mod.fit(train_iter, num_epoch=10, epoch_end_callback=checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:14 mx_mlp-0001.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0002.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0003.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0004.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0005.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0006.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0007.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0008.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0009.params\r\n",
      "-rw-r--r--  1 hyemin  staff  11372  1  9 00:15 mx_mlp-0010.params\r\n",
      "-rw-r--r--  1 hyemin  staff   1432  1  9 00:15 mx_mlp-symbol.json\r\n"
     ]
    }
   ],
   "source": [
    "# 에폭별 체크포인트 파일과 모델에 대한 symbol.json 파일이 생성됨.\n",
    "! ls -al mx_mlp*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#model load\n",
    "#저장된 체크포인트 파일에 대해 에폭번호를 지정하여 로드\n",
    "#로드하는 항목은 모델(sym), 파라미터(arg_params), 부가 파라미터(aux_params)로 구성됨.\n",
    "sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 9) #epoch = 9 model load\n",
    "\n",
    "assert sym.tojson() == net.tojson() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 로드한 파라미터를 초기값으로 이용하여 9 epoch 이후부터 training 가능\n",
    "mod.set_params(arg_params, aux_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[9] Train-accuracy=0.720625\n",
      "INFO:root:Epoch[9] Time cost=0.323\n",
      "INFO:root:Epoch[10] Train-accuracy=0.729437\n",
      "INFO:root:Epoch[10] Time cost=0.298\n",
      "INFO:root:Epoch[11] Train-accuracy=0.736625\n",
      "INFO:root:Epoch[11] Time cost=0.259\n",
      "INFO:root:Epoch[12] Train-accuracy=0.744125\n",
      "INFO:root:Epoch[12] Time cost=0.261\n",
      "INFO:root:Epoch[13] Train-accuracy=0.751250\n",
      "INFO:root:Epoch[13] Time cost=0.355\n",
      "INFO:root:Epoch[14] Train-accuracy=0.757687\n",
      "INFO:root:Epoch[14] Time cost=0.297\n",
      "INFO:root:Epoch[15] Train-accuracy=0.763250\n",
      "INFO:root:Epoch[15] Time cost=0.246\n",
      "INFO:root:Epoch[16] Train-accuracy=0.768000\n",
      "INFO:root:Epoch[16] Time cost=0.319\n",
      "INFO:root:Epoch[17] Train-accuracy=0.773750\n",
      "INFO:root:Epoch[17] Time cost=0.326\n",
      "INFO:root:Epoch[18] Train-accuracy=0.778438\n",
      "INFO:root:Epoch[18] Time cost=0.273\n",
      "INFO:root:Epoch[19] Train-accuracy=0.783813\n",
      "INFO:root:Epoch[19] Time cost=0.277\n"
     ]
    }
   ],
   "source": [
    "mod = mx.mod.Module(symbol=sym)\n",
    "mod.fit(train_iter,\n",
    "        num_epoch=20,\n",
    "        arg_params=arg_params,\n",
    "        aux_params=aux_params,\n",
    "        begin_epoch=9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}