Keras Deep Learning Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Tutorial\n",
    "\n",
    "#### Abram Hindle\n",
    "#### <abram.hindle@ualberta.ca>\n",
    "#### http://softwareprocess.ca/\n",
    "\n",
    "Slides stolen gracefully from Ben Zittlau\n",
    "\n",
    "Slide content under CC-BY-SA 4.0 and MIT License for source code or the same license as Python3 or Keras. Slide Source code is MIT License as well.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "### What is machine learning?\n",
    "\n",
    "Building a function from data to classify, predict, group, or represent data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "### Machine Learning\n",
    "\n",
    "There are a few kinds of tasks or functions that could help us here.\n",
    "\n",
    "* Classification: given some input, predict the class that it belongs\n",
    "  to. Given a point is it in the red or in the blue?\n",
    "* Regression: Given a point what will its value be? In the case of a\n",
    "  function with a continuous or numerous discrete outputs it might be\n",
    "  appropriate.\n",
    "* Representation: Learn a smaller representation of the input\n",
    "  data. E.g. we have 300 features lets describe them in a 128-bit hash.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Intro\n",
    "### Motivational Example\n",
    "\n",
    "Imagine we have this data:\n",
    "\n",
    "![2 crescent slices](images/slice.png \"A function we want to learn\n",
    " f(x,y) -> z where z is red\")\n",
    "\n",
    "[See src/genslice.py to see how we made it.](src/genslice.py)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# purpose: make a slight difference of circles be the dataset to learn.\n",
    "import numpy as np\n",
    "gY, gX = np.meshgrid(np.arange(1000)/1000.0,np.arange(1000)/1000.0)\n",
    "\n",
    "def intersect_circle(cx,cy,radius):\n",
    "    equation = (gX - float(cx)) ** 2 + (gY - float(cy)) ** 2\n",
    "    matches = equation < (radius**3) \n",
    "    return matches\n",
    "\n",
    "# rad = 0.1643167672515498\n",
    "rad = 0.3\n",
    "x = intersect_circle(0.5,0.5,rad) ^ intersect_circle(0.51,0.51,rad)\n",
    "\n",
    "def plotit(x):\n",
    "    import matplotlib.pyplot as plt\n",
    "    plt.imshow(x)\n",
    "    plt.savefig('new-slice.png') # was slice.png\n",
    "    plt.imshow(x)\n",
    "    plt.savefig('new-slice.pdf') # was slice.pdf\n",
    "    plt.show()\n",
    "\n",
    "# plotit(x)\n",
    "\n",
    "def mkcol(x):\n",
    "    return x.reshape((x.shape[0]*x.shape[1],1))\n",
    "\n",
    "# make the data set\n",
    "big = np.concatenate((mkcol(gX),mkcol(gY),mkcol(1*x)),axis=1)\n",
    "np.savetxt(\"new-big-slice.csv\", big, delimiter=\",\")\n",
    "\n",
    "# make a 50/50 data set\n",
    "nots = big[big[0:,2]==0.0,]\n",
    "np.random.shuffle(nots)\n",
    "nots = nots[0:1000,]\n",
    "trues = big[big[0:,2]==1.0,]\n",
    "np.random.shuffle(trues)\n",
    "trues = trues[0:1000,]\n",
    "small = np.concatenate((trues,nots))\n",
    "np.savetxt(\"new-small-slice.csv\", small, delimiter=\",\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "### Make your own function\n",
    "\n",
    "``` python\n",
    "def in_circle(x,y,cx,cy,radius):\n",
    "    return (x - float(cx)) ** 2 + (y - float(cy)) ** 2 < radius**2\n",
    "\n",
    "def mysolution(pt,outer=0.3):\n",
    "    return in_circle(pt[0],pt[1],0.5,0.5,outer) and not in_circle(pt[0],pt[1],0.5,0.5,0.1)\n",
    "```\n",
    "\n",
    "```\n",
    ">>> myclasses = np.apply_along_axis(mysolution,1,test[0])\n",
    ">>> print \"My classifier!\"\n",
    "My classifier!\n",
    ">>> print \"%s / %s \" % (sum(myclasses == test[1]),len(test[1]))\n",
    "181 / 200 \n",
    ">>> print theautil.classifications(myclasses,test[1])\n",
    "[('tp', 91), ('tn', 90), ('fp', 19), ('fn', 0)]\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Intro \n",
    "### An example classifier\n",
    "\n",
    "1-NN: 1 Nearest Neighbor.\n",
    "\n",
    "Given the data, we produce a function that\n",
    "outputs the CLASS of the nearest neighbour to the input data.\n",
    "\n",
    "Whoever is closer, is the class. 3-NN is 3-nearest neighbors whereby\n",
    "we use voting of the 3 neighbors instead.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "### An example classifier: 1-NN\n",
    "\n",
    "[src/slice-classifier.py](src/slice-classifier.py)\n",
    "\n",
    "``` python\n",
    "def euclid(pt1,pt2):\n",
    "    return sum([ (pt1[i] - pt2[i])**2 for i in range(0,len(pt1)) ])\n",
    "\n",
    "def oneNN(data,labels):\n",
    "    def func(input):\n",
    "        distance = None\n",
    "        label = None\n",
    "        for i in range(0,len(data)):\n",
    "            d = euclid(input,data[i])\n",
    "            if distance == None or d < distance:\n",
    "                distance = d\n",
    "                label = labels[i]\n",
    "        return label\n",
    "    return func\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Intro\n",
    "### An example classifier: 1-NN\n",
    "\n",
    "``` python\n",
    ">>> learner = oneNN(train[0],train[1])\n",
    ">>> \n",
    ">>> oneclasses = np.apply_along_axis(learner,1,test[0])\n",
    ">>> print \"1-NN classifier!\"\n",
    "1-NN classifier!\n",
    ">>> print \"%s / %s \" % (sum(oneclasses == test[1]),len(test[1]))\n",
    "198 / 200 \n",
    ">>> print theautil.classifications(oneclasses,test[1])\n",
    "[('tp', 91), ('tn', 107), ('fp', 2), ('fn', 0)]\n",
    "\n",
    "```\n",
    "\n",
    "1-NN has great performance in this example, but it uses Euclidean\n",
    "distance and the dataset is really quite biased to the positive\n",
    "classes.\n",
    "\n",
    "Thus we showed a simple learner that classifies data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Intro\n",
    "\n",
    "* That's really interesting performance and it worked but will it\n",
    "  scale and continue to work?\n",
    "\n",
    "* 1-NN doesn't work for all problems. And it is dependent on linear\n",
    "  relationships.\n",
    "\n",
    "* What if our problem is non-linear?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 2 Valid: 2\n",
      "Train X: (1620, 2) Y^:(1620, 1)\n",
      "Valid X: (180, 2)  Y^:(180, 1)\n"
     ]
    }
   ],
   "source": [
    "#\n",
    "# The MIT License (MIT)\n",
    "# \n",
    "# Copyright (c) 2016 Abram Hindle <hindle1@ualberta.ca>, Leif Johnson <leif@lmjohns3.com>\n",
    "# \n",
    "# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
    "# of this software and associated documentation files (the \"Software\"), to deal\n",
    "# in the Software without restriction, including without limitation the rights\n",
    "# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
    "# copies of the Software, and to permit persons to whom the Software is\n",
    "# furnished to do so, subject to the following conditions:\n",
    "# \n",
    "# The above copyright notice and this permission notice shall be included in all\n",
    "# copies or substantial portions of the Software.\n",
    "# \n",
    "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
    "# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
    "# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
    "# SOFTWARE.\n",
    "\n",
    "# first off we load up some modules we want to use\n",
    "# import keras\n",
    "import tensorflow.keras as keras\n",
    "import scipy\n",
    "import math\n",
    "import numpy as np\n",
    "import numpy.random as rnd\n",
    "import logging\n",
    "import sys\n",
    "import collections\n",
    "import theautil\n",
    "\n",
    "# setup logging\n",
    "logging.basicConfig(stream = sys.stderr, level=logging.INFO)\n",
    "\n",
    "mupdates = 1000\n",
    "data = np.loadtxt(\"small-slice.csv\", delimiter=\",\")\n",
    "inputs  = data[0:,0:2].astype(np.float32)\n",
    "outputs = data[0:,2:3].astype(np.int32)\n",
    "\n",
    "theautil.joint_shuffle(inputs,outputs)\n",
    "\n",
    "train_and_valid, test = theautil.split_validation(90, inputs, outputs)\n",
    "train, valid = theautil.split_validation(90, train_and_valid[0], train_and_valid[1])\n",
    "print(\"Train: %s Valid: %s\"%(len(train),len(valid)))\n",
    "print(\"Train X: %s Y^:%s\"%(train[0].shape,train[1].shape))\n",
    "print(\"Valid X: %s  Y^:%s\"%(valid[0].shape,valid[1].shape))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My classifier!\n",
      "176 / 200 \n",
      "[('tp', 94), ('tn', 82), ('fp', 24), ('fn', 0)]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def linit(x):\n",
    "    return x.reshape((len(x),))\n",
    "\n",
    "mltrain = (train[0],linit(train[1]))\n",
    "mlvalid = (valid[0],linit(valid[1]))\n",
    "mltest  = (test[0] ,linit(test[1]))\n",
    "\n",
    "# my solution\n",
    "def in_circle(x,y,cx,cy,radius):\n",
    "    return (x - float(cx)) ** 2 + (y - float(cy)) ** 2 < radius**2\n",
    "\n",
    "def mysolution(pt,outer=0.3):\n",
    "    return in_circle(pt[0],pt[1],0.5,0.5,outer) and \\\n",
    "           not in_circle(pt[0],pt[1],0.5,0.5,0.1)\n",
    "\n",
    "# apply my classifier\n",
    "myclasses = np.apply_along_axis(mysolution,1,mltest[0])\n",
    "print(\"My classifier!\")\n",
    "print(\"%s / %s \" % (sum(myclasses == mltest[1]),len(mltest[1])))\n",
    "print(theautil.classifications(myclasses,mltest[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1-NN classifier!\n",
      "198 / 200 \n",
      "[('tp', 94), ('tn', 104), ('fp', 2), ('fn', 0)]\n",
      "1-NN classifier! ON TRAIN\n",
      "1620 / 1620 \n",
      "[('tp', 816), ('tn', 804), ('fp', 0), ('fn', 0)]\n"
     ]
    }
   ],
   "source": [
    "def euclid(pt1,pt2):\n",
    "    return sum([ (pt1[i] - pt2[i])**2 for i in range(0,len(pt1)) ])\n",
    "\n",
    "def oneNN(data,labels):\n",
    "    def func(input):\n",
    "        distance = None\n",
    "        label = None\n",
    "        for i in range(0,len(data)):\n",
    "            d = euclid(input,data[i])\n",
    "            if distance == None or d < distance:\n",
    "                distance = d\n",
    "                label = labels[i]\n",
    "        return label\n",
    "    return func\n",
    "\n",
    "learner = oneNN(mltrain[0],mltrain[1])\n",
    "\n",
    "oneclasses = np.apply_along_axis(learner,1,mltest[0])\n",
    "print(\"1-NN classifier!\")\n",
    "print(\"%s / %s \" % (sum(oneclasses == mltest[1]),len(mltest[1])))\n",
    "print(theautil.classifications(oneclasses,mltest[1]))\n",
    "\n",
    "toneclasses = np.apply_along_axis(learner,1,mltrain[0])\n",
    "print(\"1-NN classifier! ON TRAIN\")\n",
    "print(\"%s / %s \" % (sum(toneclasses == mltrain[1]),len(mltrain[1])))\n",
    "print(theautil.classifications(toneclasses,mltrain[1]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Intro\n",
    "\n",
    "* Neural networks are popular\n",
    "   * Creating AI for Go\n",
    "   * Labeling Images with cats and dogs\n",
    "   * Speech Recognition\n",
    "   * Text summarization\n",
    "   * [Guitar Transcription](https://peerj.com/preprints/1193.pdf)\n",
    "   * Learn audio from video[1](https://archive.org/details/DeepLearningBitmaptoPCM/)[2](http://softwareprocess.es/blog/blog/2015/08/10/deep-learning-bitmaps-to-pcm/)\n",
    "\n",
    "* Neural networks can not only classify, but they can create content,\n",
    "  they can have complicated outputs.\n",
    "\n",
    "* Neural networks are generative!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "### Machine Learning: Neural Networks\n",
    "\n",
    "Neural networks or \"Artificial Neural Networks\" are a flexible class\n",
    "of non-linear machine learners. They have been found to be quite\n",
    "effective as of late.\n",
    "\n",
    "Neural networks are composed of neurons. These neurons try to emulate\n",
    "biological neurons in the most metaphorical of senses. Given a set of\n",
    "inputs they produce an output.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Neurons\n",
    "\n",
    "Neurons have functions.\n",
    "\n",
    "* Rectified Linear Units have been shown to train quite well and\n",
    "  achieve good results. By they aren't easier to differentiate.\n",
    "  f(x) = max(0,x)\n",
    "* Sigmoid functions are slow and were the classical neural network\n",
    "  neuron, but have fallen out of favour. They will work when nothing\n",
    "  else will. f(x) = 1/(1 + e^-x)\n",
    "* Softplus is a RELU that is slower to compute but differentiable.\n",
    "  f(x) = ln(1 + e^x)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Neurons\n",
    "\n",
    "![Rectifier and Sigmoid and Softplus](images/Rectifier_and_softplus_functions.svg)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Neurons\n",
    "\n",
    "The inputs to a neural network? The outputs of connected nodes times\n",
    "their weight + a bias.\n",
    "\n",
    "neuron(inputs) = neuron_f( sum(weights * inputs) + bias  )\n",
    "\n",
    "![Neuron example](images/neuron.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.5404427087207618, [0.5370495669980353, 0.7162978701990245, 0.7615941559557649, 0.8617231593133063], array([0.4, 0.3, 0.6]))\n",
      "Better 0 (1.4866137432899655, [-0.6647188082502699, -0.9918813653217701, -0.8539141268422795, -0.9968157451060176], array([-0.46920893, -1.95011547, -0.80122097]))\n",
      "Better 2 (1.4838992342049964, [-0.5421077685734538, -0.9073291027427597, -0.9711837558377038, -0.9952272157156234], array([-1.50558934, -0.90506655, -0.60713579]))\n",
      "Better 3 (1.342187408032217, [-0.36881990441710016, -0.9462765269984749, -0.5336489328059542, -0.9642451257769672], array([-0.20817668, -1.40785515, -0.38705651]))\n",
      "Better 4 (0.980351397497156, [-0.8888400983919069, -0.35220470123465963, 0.3338958373396271, 0.8844017301173606], array([ 1.76358079,  1.0484158 , -1.41637425]))\n",
      "Better 7 (0.7086348569632173, [-0.7922954726760598, 0.5944221722158192, 0.49450272250389854, 0.9802553004416915], array([ 1.61957104,  1.76204491, -1.07756797]))\n",
      "Better 13 (0.5189404356173829, [-0.08211296642629633, 0.7016273449425803, 0.9167940232146844, 0.9871700399429157], array([ 1.65084332,  0.95279682, -0.08229827]))\n",
      "Better 71 (0.5033474380649556, [-0.04446997286101252, 0.9577107208828372, 0.8784693907489365, 0.9974462005676167], array([ 1.41352253,  1.96199805, -0.04449932]))\n",
      "Better 457 (0.5009683726187336, [-0.04229894054240323, 0.9601229760583423, 0.9514625731532883, 0.9990705591912395], array([ 1.8893238 ,  1.98980528, -0.04232419]))\n",
      "Better 3669 (0.5005799740654677, [0.024026552232291033, 0.9576098830958973, 0.9547257383518329, 0.9989480896174916], array([1.85860206, 1.89225098, 0.02403118]))\n",
      "Best result!\n",
      "(0.5005799740654677, [0.024026552232291033, 0.9576098830958973, 0.9547257383518329, 0.9989480896174916], array([1.85860206, 1.89225098, 0.02403118]))\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "# this is a neuron\n",
    "def tanh(weights,inputs,bias=0):\n",
    "    return numpy.tanh(numpy.sum(weights * inputs) + bias)\n",
    "# XOR\n",
    "train_x = numpy.array([[0,0],[0,1],[1,0],[1,1]])\n",
    "train_y = numpy.array([0,1,1,0])\n",
    "def rms(train_y_hat, train_y):\n",
    "    return numpy.sqrt(numpy.mean((train_y_hat - train_y)**2))\n",
    "best = None\n",
    "weights = numpy.array([0.4,0.3,0.6])\n",
    "train_y_hat = [tanh(weights[0:2],t,weights[2]) for t in train_x]\n",
    "res = (rms(train_y_hat, train_y), train_y_hat, weights)\n",
    "print(res)\n",
    "\n",
    "for i in range(0,10000):\n",
    "    weights = 4*numpy.random.rand(3)-2\n",
    "    train_y_hat = [tanh(weights[0:2],t,weights[2]) for t in train_x]\n",
    "    res = (rms(train_y_hat, train_y), train_y_hat, weights)\n",
    "    #print(res)\n",
    "    if best is None or res[0] < best[0]:\n",
    "        print(f\"Better {i}\",res)\n",
    "        best = res\n",
    "print(\"Best result!\")\n",
    "print(best)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better 0 (1.0392997571367157, [-0.9885546885269915, -0.37812464265386203, 0.1748236970017731, 0.8736086691473437], array([-1.80513234, -1.25921525,  1.54565005, -1.14505175,  0.99035643,\n",
      "        1.03340689, -0.07836691,  1.77347142, -1.39458737]))\n",
      "Better 1 (1.0346776683852625, [0.1125591802577348, -0.5852577386616211, -0.1746527036830924, -0.6137677643081333], array([ 1.69555135,  1.77740615, -1.7353888 , -0.3835196 ,  0.88772935,\n",
      "       -1.44837477,  1.13666868,  0.36490135, -0.54412153]))\n",
      "Better 3 (0.5366106228234248, [0.6423068618632775, 0.819742337575398, 0.47827851567542007, 0.6592113509660609], array([-1.94740825,  1.21970186,  0.04489612,  0.3554685 , -0.2692329 ,\n",
      "       -0.2536154 ,  0.85163114, -0.68409661,  1.21948298]))\n",
      "Better 40 (0.5128192232498399, [-0.07388123393620917, 0.7816640898528522, 0.9981015976723836, 0.9994006266853941], array([-1.96901213,  0.22350755,  0.05445328, -1.94101921,  0.92330994,\n",
      "        0.88761934, -0.24228396,  1.91535551,  0.48676225]))\n",
      "Better 138 (0.49253392181860345, [0.019065442846213504, 0.54641289765083, 0.3675177722405955, 0.6035065348512135], array([ 0.0660339 , -1.11385004,  0.53555428, -0.69063896,  1.1127412 ,\n",
      "       -1.04146894, -0.90869045,  0.43397264,  0.66989384]))\n",
      "Better 246 (0.45333823317333316, [0.29362439800824286, 0.8721017576830323, 0.30457995098951185, 0.4856747085937818], array([ 1.48077885, -1.49870627,  1.46635495, -0.9110039 , -0.32804726,\n",
      "       -0.80418769, -0.2971426 , -0.37268243,  1.01381931]))\n",
      "Better 271 (0.44303729448585966, [0.1251727742102912, 0.5253153156202076, 0.5481248198758248, 0.5830465039950947], array([ 1.44421137,  0.39831121, -0.25490329,  0.49754898, -1.96647111,\n",
      "        1.0422884 , -1.64902895,  0.67926133,  0.88082875]))\n",
      "Better 1543 (0.4243668460267164, [-0.11579769199382711, 0.64429514559836, 0.64828703571835, 0.6758045737244436], array([ 1.59586655,  1.92422288,  0.51453013,  1.54504068, -1.48932235,\n",
      "       -0.67609806, -1.12889146, -0.67049745, -1.3913775 ]))\n",
      "Better 1697 (0.419417633367401, [0.2467133305959696, 0.5675799501548612, 0.765771036158875, 0.6331878317009344], array([ 1.69769073,  1.17412653,  0.36728877,  1.02611507, -0.34211705,\n",
      "        1.35296281,  0.62315608, -0.44830723,  0.1391677 ]))\n",
      "Better 2563 (0.3283221155466335, [0.18786941002735658, 0.714016615997109, 0.5296695689133887, 0.30477782181808794], array([-1.54397623,  1.45260262, -1.39502182,  0.80538383,  1.89287524,\n",
      "       -1.16199151, -1.16815698,  0.33825075,  1.18091568]))\n",
      "Better 3383 (0.27238237464215953, [-0.09547879290117499, 0.6812429323342409, 0.5703569686523091, -0.038121248621147935], array([-1.99014109,  0.80326938, -0.22102315,  1.61601935, -1.88169414,\n",
      "        0.81856735,  0.95260289, -1.3391241 ,  1.2479387 ]))\n",
      "Better 7288 (0.17746819481709566, [0.028440503991254595, 0.7035745351012105, 0.8471669758275592, 0.11808890253161194], array([ 1.57707367, -1.62814002, -0.56462667,  1.19815764,  0.93433469,\n",
      "       -1.10997375,  1.01170204, -1.6216985 ,  1.88416385]))\n",
      "Best result!\n",
      "(0.17746819481709566, [0.028440503991254595, 0.7035745351012105, 0.8471669758275592, 0.11808890253161194], array([ 1.57707367, -1.62814002, -0.56462667,  1.19815764,  0.93433469,\n",
      "       -1.10997375,  1.01170204, -1.6216985 ,  1.88416385]))\n"
     ]
    }
   ],
   "source": [
    "# now with 3 neurons\n",
    "\n",
    "#  input1--+->neuron1\\\n",
    "#        \\ /         \\+neuron3--->result\n",
    "#        /\\          /\n",
    "#  input2-+->neuron2/\n",
    "\n",
    "import numpy\n",
    "# this is a neuron\n",
    "def tanh(weights,inputs,bias=0):\n",
    "    return numpy.tanh(numpy.sum(weights * inputs) + bias)\n",
    "# XOR\n",
    "train_x = numpy.array([[0,0],[0,1],[1,0],[1,1]])\n",
    "train_y = numpy.array([0,1,1,0])\n",
    "def rms(train_y_hat, train_y):\n",
    "    return numpy.sqrt(numpy.mean((train_y_hat - train_y)**2))\n",
    "best = None\n",
    "\n",
    "def random_weights():\n",
    "    return 4*numpy.random.rand(9)-2\n",
    "#  input1--+->neuron1\\\n",
    "#        \\ /         \\+neuron3--->result\n",
    "#        /\\          /\n",
    "#  input2-+->neuron2/\n",
    "def network(weights,t):\n",
    "    inputs = [tanh(weights[0:2],t,weights[2]), #neuron1\n",
    "              tanh(weights[4:6],t,weights[6])] #neuron2\n",
    "    w = numpy.array([weights[3],weights[7]]) #weights for neuron3\n",
    "    return tanh(w,inputs,weights[8]) #neuron3\n",
    "    \n",
    "\n",
    "for i in range(0,100000):\n",
    "    weights = random_weights()\n",
    "    train_y_hat = [network(weights,t) for t in train_x]\n",
    "    res = (rms(train_y_hat, train_y), train_y_hat, weights)\n",
    "    #print(res)\n",
    "    if best is None or res[0] < best[0]:\n",
    "        print(f\"Better {i}\",res)\n",
    "        best = res\n",
    "print(\"Best result!\")\n",
    "print(best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Multi-layer perceptron\n",
    "\n",
    "Single hidden layer neural network.\n",
    "\n",
    "![Multi-layer perceptron](images/20160208141015.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Deep Learning\n",
    "\n",
    "There's nothing particularly crazy about deep learning other than it has more hidden layers.\n",
    "\n",
    "These hidden layers allow it to compute state and address the intricacies of complex functions. But each hidden layer adds a lot of search space.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Deep Learning\n",
    "\n",
    "![Deep network, multiple layers](images/20160208141143.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Search\n",
    "\n",
    "How do we find the different weights?\n",
    "\n",
    "Well we need to search a large space. A 2x3x2 network will have 2*3*2\n",
    "weights + 5 biases (3 hidden, 2 output) resulting in 17\n",
    "parameters. That's already a large search space.\n",
    "\n",
    "Most search algorithms measure their error at a certain point\n",
    "(difference between prediction and actual) and then choose a direction\n",
    "in their search space to travel. They do this different ways.\n",
    "One way is by sampling points\n",
    "around themselves in order to compute a gradient or slope and then\n",
    "follow the slope around. The most common way is to calculate the gradient symbolicly, compute the derivatives, and avoid sampling altogether (like stochastic gradient descent).\n",
    "\n",
    "Here's a 3D demo of different search algorithms.\n",
    "\n",
    "[Different Search Parameters](http://www.robertsdionne.com/bouncingball/)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Let's deep learn on our problem\n",
    "\n",
    "![2 crescent slices](images/slice.png \"A function we want to learn\n",
    " f(x,y) -> z where z is red\")\n",
    "\n",
    "Please open [slice-classifier](./src/slice-classifier.py) and a python\n",
    "interpreter such as bpython. Search for Part 3 around line 100.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "########################################################################\n",
      "# Part 3. Let's start using neural networks!\n",
      "########################################################################\n",
      "\n",
      "(1620, 2)\n",
      "(1620, 1)\n",
      "(1620, 2)\n",
      "[[0.867 0.463]\n",
      " [0.65  0.046]\n",
      " [0.384 0.736]\n",
      " [0.379 0.392]\n",
      " [0.562 0.657]\n",
      " [0.358 0.443]\n",
      " [0.034 0.993]\n",
      " [0.464 0.663]\n",
      " [0.386 0.573]\n",
      " [0.485 0.144]]\n",
      "[[0]\n",
      " [0]\n",
      " [0]\n",
      " [1]\n",
      " [1]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [0]\n",
      " [0]]\n",
      "[[1. 0.]\n",
      " [1. 0.]\n",
      " [1. 0.]\n",
      " [0. 1.]\n",
      " [0. 1.]\n",
      " [0. 1.]\n",
      " [1. 0.]\n",
      " [0. 1.]\n",
      " [1. 0.]\n",
      " [1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print('''\n",
    "########################################################################\n",
    "# Part 3. Let's start using neural networks!\n",
    "########################################################################\n",
    "''')\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "from tensorflow import convert_to_tensor as tft\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "enc = OneHotEncoder(handle_unknown='ignore',sparse=False)\n",
    "enc.fit(train[1])\n",
    "train_y = enc.transform(train[1]).astype('float64')\n",
    "valid_y = enc.transform(valid[1]).astype('float64')\n",
    "test_y = enc.transform(test[1]).astype('float64')\n",
    "\n",
    "print(train[0].shape)\n",
    "print(train[1].shape)\n",
    "print(train_y.shape)\n",
    "\n",
    "print(train[0][0:10])\n",
    "print(train[1][0:10])\n",
    "print(train_y[0:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense (Dense)               (None, 16)                48        \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 32)                544       \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 2)                 66        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 658\n",
      "Trainable params: 658\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/100\n",
      "405/405 [==============================] - 2s 1ms/step - loss: 0.6975 - accuracy: 0.4895 - val_loss: 0.6935 - val_accuracy: 0.5000\n",
      "Epoch 2/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6966 - accuracy: 0.5012 - val_loss: 0.6937 - val_accuracy: 0.5000\n",
      "Epoch 3/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6968 - accuracy: 0.5019 - val_loss: 0.6958 - val_accuracy: 0.5000\n",
      "Epoch 4/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6991 - accuracy: 0.4815 - val_loss: 0.7025 - val_accuracy: 0.5000\n",
      "Epoch 5/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6946 - accuracy: 0.5179 - val_loss: 0.6934 - val_accuracy: 0.5000\n",
      "Epoch 6/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6951 - accuracy: 0.4932 - val_loss: 0.6959 - val_accuracy: 0.5000\n",
      "Epoch 7/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6963 - accuracy: 0.4870 - val_loss: 0.6930 - val_accuracy: 0.5000\n",
      "Epoch 8/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6962 - accuracy: 0.5031 - val_loss: 0.6957 - val_accuracy: 0.5000\n",
      "Epoch 9/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6966 - accuracy: 0.4926 - val_loss: 0.6934 - val_accuracy: 0.5000\n",
      "Epoch 10/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6944 - accuracy: 0.5241 - val_loss: 0.6928 - val_accuracy: 0.4944\n",
      "Epoch 11/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6958 - accuracy: 0.4833 - val_loss: 0.6928 - val_accuracy: 0.5000\n",
      "Epoch 12/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6971 - accuracy: 0.4981 - val_loss: 0.6979 - val_accuracy: 0.5000\n",
      "Epoch 13/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6974 - accuracy: 0.4858 - val_loss: 0.6923 - val_accuracy: 0.4222\n",
      "Epoch 14/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6957 - accuracy: 0.4994 - val_loss: 0.6934 - val_accuracy: 0.5000\n",
      "Epoch 15/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6968 - accuracy: 0.4988 - val_loss: 0.6939 - val_accuracy: 0.5000\n",
      "Epoch 16/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6978 - accuracy: 0.4877 - val_loss: 0.6919 - val_accuracy: 0.5278\n",
      "Epoch 17/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6963 - accuracy: 0.4914 - val_loss: 0.6918 - val_accuracy: 0.6444\n",
      "Epoch 18/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6941 - accuracy: 0.5142 - val_loss: 0.6941 - val_accuracy: 0.5000\n",
      "Epoch 19/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6972 - accuracy: 0.5000 - val_loss: 0.6934 - val_accuracy: 0.5111\n",
      "Epoch 20/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6955 - accuracy: 0.4920 - val_loss: 0.6975 - val_accuracy: 0.5000\n",
      "Epoch 21/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6945 - accuracy: 0.5099 - val_loss: 0.6934 - val_accuracy: 0.5167\n",
      "Epoch 22/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6944 - accuracy: 0.4951 - val_loss: 0.6982 - val_accuracy: 0.5000\n",
      "Epoch 23/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6932 - accuracy: 0.5167 - val_loss: 0.6907 - val_accuracy: 0.5111\n",
      "Epoch 24/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6937 - accuracy: 0.5123 - val_loss: 0.6909 - val_accuracy: 0.6222\n",
      "Epoch 25/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6936 - accuracy: 0.5080 - val_loss: 0.6914 - val_accuracy: 0.4167\n",
      "Epoch 26/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6942 - accuracy: 0.5037 - val_loss: 0.6896 - val_accuracy: 0.4778\n",
      "Epoch 27/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6929 - accuracy: 0.5432 - val_loss: 0.6927 - val_accuracy: 0.4944\n",
      "Epoch 28/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6929 - accuracy: 0.4938 - val_loss: 0.6900 - val_accuracy: 0.4000\n",
      "Epoch 29/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6931 - accuracy: 0.5043 - val_loss: 0.6885 - val_accuracy: 0.4778\n",
      "Epoch 30/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6925 - accuracy: 0.5204 - val_loss: 0.6885 - val_accuracy: 0.6667\n",
      "Epoch 31/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6917 - accuracy: 0.5216 - val_loss: 0.6919 - val_accuracy: 0.4833\n",
      "Epoch 32/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6912 - accuracy: 0.5340 - val_loss: 0.6898 - val_accuracy: 0.6000\n",
      "Epoch 33/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6914 - accuracy: 0.5222 - val_loss: 0.6888 - val_accuracy: 0.6111\n",
      "Epoch 34/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6911 - accuracy: 0.5272 - val_loss: 0.6868 - val_accuracy: 0.5000\n",
      "Epoch 35/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6905 - accuracy: 0.5364 - val_loss: 0.6875 - val_accuracy: 0.4000\n",
      "Epoch 36/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6888 - accuracy: 0.5167 - val_loss: 0.6847 - val_accuracy: 0.5278\n",
      "Epoch 37/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6881 - accuracy: 0.5191 - val_loss: 0.6869 - val_accuracy: 0.6222\n",
      "Epoch 38/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6883 - accuracy: 0.5494 - val_loss: 0.6863 - val_accuracy: 0.6222\n",
      "Epoch 39/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6853 - accuracy: 0.5469 - val_loss: 0.6821 - val_accuracy: 0.4889\n",
      "Epoch 40/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6868 - accuracy: 0.5185 - val_loss: 0.6812 - val_accuracy: 0.4833\n",
      "Epoch 41/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6853 - accuracy: 0.5074 - val_loss: 0.6912 - val_accuracy: 0.5833\n",
      "Epoch 42/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6868 - accuracy: 0.5444 - val_loss: 0.6855 - val_accuracy: 0.4167\n",
      "Epoch 43/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6857 - accuracy: 0.5136 - val_loss: 0.6784 - val_accuracy: 0.5444\n",
      "Epoch 44/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6859 - accuracy: 0.5302 - val_loss: 0.6907 - val_accuracy: 0.5000\n",
      "Epoch 45/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6852 - accuracy: 0.5142 - val_loss: 0.6756 - val_accuracy: 0.4833\n",
      "Epoch 46/100\n",
      "405/405 [==============================] - 0s 995us/step - loss: 0.6798 - accuracy: 0.5432 - val_loss: 0.6743 - val_accuracy: 0.5278\n",
      "Epoch 47/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6810 - accuracy: 0.5407 - val_loss: 0.6730 - val_accuracy: 0.5056\n",
      "Epoch 48/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6777 - accuracy: 0.5346 - val_loss: 0.6727 - val_accuracy: 0.5444\n",
      "Epoch 49/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6787 - accuracy: 0.5278 - val_loss: 0.6694 - val_accuracy: 0.4833\n",
      "Epoch 50/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6743 - accuracy: 0.5543 - val_loss: 0.6673 - val_accuracy: 0.4889\n",
      "Epoch 51/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6749 - accuracy: 0.5426 - val_loss: 0.6651 - val_accuracy: 0.5278\n",
      "Epoch 52/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6730 - accuracy: 0.5407 - val_loss: 0.6663 - val_accuracy: 0.6611\n",
      "Epoch 53/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6699 - accuracy: 0.5525 - val_loss: 0.6639 - val_accuracy: 0.5500\n",
      "Epoch 54/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6672 - accuracy: 0.5654 - val_loss: 0.6619 - val_accuracy: 0.5333\n",
      "Epoch 55/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6632 - accuracy: 0.5537 - val_loss: 0.6638 - val_accuracy: 0.6222\n",
      "Epoch 56/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6613 - accuracy: 0.5722 - val_loss: 0.6584 - val_accuracy: 0.6222\n",
      "Epoch 57/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6595 - accuracy: 0.5685 - val_loss: 0.6470 - val_accuracy: 0.6611\n",
      "Epoch 58/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6571 - accuracy: 0.5741 - val_loss: 0.6431 - val_accuracy: 0.6722\n",
      "Epoch 59/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6475 - accuracy: 0.5895 - val_loss: 0.6392 - val_accuracy: 0.6667\n",
      "Epoch 60/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6434 - accuracy: 0.6043 - val_loss: 0.6313 - val_accuracy: 0.5722\n",
      "Epoch 61/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6367 - accuracy: 0.6364 - val_loss: 0.6243 - val_accuracy: 0.6222\n",
      "Epoch 62/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6306 - accuracy: 0.6364 - val_loss: 0.6192 - val_accuracy: 0.5889\n",
      "Epoch 63/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6196 - accuracy: 0.6426 - val_loss: 0.6077 - val_accuracy: 0.7389\n",
      "Epoch 64/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.6106 - accuracy: 0.6883 - val_loss: 0.5982 - val_accuracy: 0.7556\n",
      "Epoch 65/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5999 - accuracy: 0.7037 - val_loss: 0.5886 - val_accuracy: 0.7500\n",
      "Epoch 66/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5866 - accuracy: 0.7179 - val_loss: 0.5787 - val_accuracy: 0.6944\n",
      "Epoch 67/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5732 - accuracy: 0.7605 - val_loss: 0.5641 - val_accuracy: 0.8667\n",
      "Epoch 68/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5566 - accuracy: 0.7932 - val_loss: 0.5486 - val_accuracy: 0.8611\n",
      "Epoch 69/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5375 - accuracy: 0.7963 - val_loss: 0.5328 - val_accuracy: 0.8611\n",
      "Epoch 70/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5231 - accuracy: 0.8216 - val_loss: 0.5156 - val_accuracy: 0.8167\n",
      "Epoch 71/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.5055 - accuracy: 0.8278 - val_loss: 0.4995 - val_accuracy: 0.8389\n",
      "Epoch 72/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4893 - accuracy: 0.8346 - val_loss: 0.4848 - val_accuracy: 0.8333\n",
      "Epoch 73/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4736 - accuracy: 0.8444 - val_loss: 0.4773 - val_accuracy: 0.8111\n",
      "Epoch 74/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4568 - accuracy: 0.8562 - val_loss: 0.4585 - val_accuracy: 0.8500\n",
      "Epoch 75/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4440 - accuracy: 0.8574 - val_loss: 0.4457 - val_accuracy: 0.8333\n",
      "Epoch 76/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4292 - accuracy: 0.8667 - val_loss: 0.4350 - val_accuracy: 0.8333\n",
      "Epoch 77/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4157 - accuracy: 0.8691 - val_loss: 0.4332 - val_accuracy: 0.8167\n",
      "Epoch 78/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.4040 - accuracy: 0.8630 - val_loss: 0.4162 - val_accuracy: 0.8611\n",
      "Epoch 79/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3942 - accuracy: 0.8747 - val_loss: 0.4095 - val_accuracy: 0.8333\n",
      "Epoch 80/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3804 - accuracy: 0.8790 - val_loss: 0.3994 - val_accuracy: 0.8500\n",
      "Epoch 81/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3744 - accuracy: 0.8796 - val_loss: 0.3917 - val_accuracy: 0.8556\n",
      "Epoch 82/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3658 - accuracy: 0.8815 - val_loss: 0.3936 - val_accuracy: 0.8722\n",
      "Epoch 83/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3581 - accuracy: 0.8858 - val_loss: 0.3914 - val_accuracy: 0.8278\n",
      "Epoch 84/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3508 - accuracy: 0.8895 - val_loss: 0.3773 - val_accuracy: 0.8556\n",
      "Epoch 85/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3445 - accuracy: 0.8889 - val_loss: 0.3736 - val_accuracy: 0.8556\n",
      "Epoch 86/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3384 - accuracy: 0.8877 - val_loss: 0.3664 - val_accuracy: 0.8611\n",
      "Epoch 87/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3338 - accuracy: 0.8926 - val_loss: 0.3668 - val_accuracy: 0.8722\n",
      "Epoch 88/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3278 - accuracy: 0.8938 - val_loss: 0.3643 - val_accuracy: 0.8556\n",
      "Epoch 89/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3241 - accuracy: 0.8963 - val_loss: 0.3606 - val_accuracy: 0.8556\n",
      "Epoch 90/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3196 - accuracy: 0.8969 - val_loss: 0.3544 - val_accuracy: 0.8667\n",
      "Epoch 91/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3160 - accuracy: 0.9012 - val_loss: 0.3609 - val_accuracy: 0.8500\n",
      "Epoch 92/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3126 - accuracy: 0.8988 - val_loss: 0.3491 - val_accuracy: 0.8611\n",
      "Epoch 93/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3104 - accuracy: 0.9025 - val_loss: 0.3468 - val_accuracy: 0.8611\n",
      "Epoch 94/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3062 - accuracy: 0.9012 - val_loss: 0.3561 - val_accuracy: 0.8556\n",
      "Epoch 95/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3032 - accuracy: 0.9012 - val_loss: 0.3449 - val_accuracy: 0.8611\n",
      "Epoch 96/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3022 - accuracy: 0.9006 - val_loss: 0.3399 - val_accuracy: 0.8611\n",
      "Epoch 97/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.3022 - accuracy: 0.9037 - val_loss: 0.3386 - val_accuracy: 0.8611\n",
      "Epoch 98/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.2973 - accuracy: 0.9012 - val_loss: 0.3368 - val_accuracy: 0.8611\n",
      "Epoch 99/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.2917 - accuracy: 0.9105 - val_loss: 0.3532 - val_accuracy: 0.8500\n",
      "Epoch 100/100\n",
      "405/405 [==============================] - 0s 1ms/step - loss: 0.2928 - accuracy: 0.9037 - val_loss: 0.3485 - val_accuracy: 0.8556\n",
      "CPU times: user 54.5 s, sys: 5.2 s, total: 59.7 s\n",
      "Wall time: 46.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# rerunning this will produce different results\n",
    "# try different combos here\n",
    "net = Sequential([\n",
    "    Dense(16,input_shape=(2,),activation=\"sigmoid\"),\n",
    "    Dense(32,activation=\"sigmoid\"),\n",
    "    Dense(2,activation=\"softmax\")\n",
    "])\n",
    "\n",
    "# opt = SGD()#\n",
    "opt = Adam() # lr=0.1)\n",
    "net.compile(loss=\"categorical_crossentropy\", optimizer=opt, metrics=[\"accuracy\"])\n",
    "print(net.summary())\n",
    "# net.fit(x=train[0],y=train_y, epochs=100)\n",
    "history = net.fit(x=train[0], y=train_y, \n",
    "                validation_data=(valid[0], valid_y),\n",
    "                epochs=100, batch_size=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learner on the test set\n",
      "7/7 [==============================] - 0s 1ms/step - loss: 0.2566 - accuracy: 0.9050\n",
      "Scores: [0.2566322684288025, 0.9049999713897705]\n",
      "181 / 200 \n",
      "Counter({1: 113, 0: 87})\n",
      "[('tp', 94), ('tn', 87), ('fp', 19), ('fn', 0)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Learner on the test set\")\n",
    "score = net.evaluate(test[0], test_y)\n",
    "print(\"Scores: %s\" % score)\n",
    "predictit = net.predict(test[0])\n",
    "#print(predictit.shape)\n",
    "#print(predictit[0:10,])\n",
    "#print(net.__class__)\n",
    "#classify = net.predict_classes(test[0])\n",
    "#predict_x=net.predict(test[0]) \n",
    "#classify=np.argmax(predict_x,axis=1)\n",
    "\n",
    "def predict_classes(net,test):\n",
    "    predict_x=net.predict(test) \n",
    "    classify=np.argmax(predict_x,axis=1)\n",
    "    return classify\n",
    "    \n",
    "classify = predict_classes(net, test[0])\n",
    "print(\"%s / %s \" % (np.sum(classify == mltest[1]),len(mltest[1])))\n",
    "print(collections.Counter(classify))\n",
    "print(theautil.classifications(classify,mltest[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try this on unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "And now on more unseen data that isn't 50/50\n",
      "2378 / 3000 \n",
      "Counter({0: 2343, 1: 657})\n",
      "[('tp', 35), ('tn', 2343), ('fp', 622), ('fn', 0)]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def real_function(pt):\n",
    "    rad = 0.1643167672515498\n",
    "    in1 = in_circle(pt[0],pt[1],0.5,0.5,rad)\n",
    "    in2 = in_circle(pt[0],pt[1],0.51,0.51,rad)\n",
    "    return in1 ^ in2\n",
    "\n",
    "print(\"And now on more unseen data that isn't 50/50\")\n",
    "\n",
    "bigtest = np.random.uniform(size=(3000,2)).astype(np.float32)\n",
    "biglab = np.apply_along_axis(real_function,1,bigtest).astype(np.int32)\n",
    "\n",
    "classify = predict_classes(net,bigtest)\n",
    "print(\"%s / %s \" % (sum(classify == biglab),len(biglab)))\n",
    "print(collections.Counter(classify))\n",
    "print(theautil.classifications(classify,biglab))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Now let's discuss posing problems for neural networks\n",
    "\n",
    "* Scaling inputs: Scaling can sometimes help, so can\n",
    "  standardization. This means constraining values or re-centering\n",
    "  them. It depends on your problem and it is worth trying.\n",
    "\n",
    "* E.g. min max scaling:\n",
    "\n",
    "``` python\n",
    "def min_max_scale(data):\n",
    "    '''scales data by minimum and maximum values between 0 and 1'''\n",
    "    dmin = np.min(data)\n",
    "    return (data - dmin)/(np.max(data) - dmin)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The problem\n",
    "\n",
    "* [posing.py](src/posing.py) tries to show the problem of taking\n",
    "  random input data and determine what distribution it comes from.\n",
    "  That is what function can produce these random values.\n",
    "\n",
    "* Let's open up [posing.py](src/posing.py) and get an interpreter\n",
    "  going.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demonstration of how to pose the problem and how different formulations\n",
    "# lead to different results!\n",
    "#\n",
    "# The MIT License (MIT)\n",
    "# \n",
    "# Copyright (c) 2016 Abram Hindle <hindle1@ualberta.ca>, Leif Johnson <leif@lmjohns3.com>\n",
    "# \n",
    "# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
    "# of this software and associated documentation files (the \"Software\"), to deal\n",
    "# in the Software without restriction, including without limitation the rights\n",
    "# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
    "# copies of the Software, and to permit persons to whom the Software is\n",
    "# furnished to do so, subject to the following conditions:\n",
    "# \n",
    "# The above copyright notice and this permission notice shall be included in all\n",
    "# copies or substantial portions of the Software.\n",
    "# \n",
    "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
    "# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
    "# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
    "# SOFTWARE.\n",
    "\n",
    "# first off we load up some modules we want to use\n",
    "import keras\n",
    "import scipy\n",
    "import math\n",
    "import numpy as np\n",
    "import numpy.random as rnd\n",
    "import logging\n",
    "import sys\n",
    "from numpy.random import power, normal, lognormal, uniform\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "import theautil\n",
    "import tensorflow as tf\n",
    "\n",
    "# What are we going to do?\n",
    "# - we're going to generate data derived from 4 different distributions\n",
    "# - we're going to scale that data\n",
    "# - we're going to create a RBM (1 hidden layer neural network)\n",
    "# - we're going to train it to classify data as belonging to one of these distributions\n",
    "\n",
    "# maximum number of iterations before we bail\n",
    "mupdates = 1000\n",
    "\n",
    "# setup logging\n",
    "logging.basicConfig(stream = sys.stderr, level=logging.INFO)\n",
    "\n",
    "# how we pose our problem to the deep belief network matters.\n",
    "\n",
    "# lets make the task easier by scaling all values between 0 and 1\n",
    "def min_max_scale(data):\n",
    "    '''scales data by minimum and maximum values between 0 and 1'''\n",
    "    dmin = np.min(data)\n",
    "    return (data - dmin)/(np.max(data) - dmin)\n",
    "\n",
    "# how many samples per each distribution\n",
    "bsize    = 100 \n",
    "\n",
    "# poor man's enum\n",
    "LOGNORMAL=0\n",
    "POWER=1\n",
    "NORM=2\n",
    "UNIFORM=3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Experiment 1\n",
    "\n",
    "* Given 1 single sample what distribution does it come from?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "########################################################################\n",
      "# Experiment 1: can we classify single samples?\n",
      "#\n",
      "#\n",
      "#########################################################################\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('''\n",
    "########################################################################\n",
    "# Experiment 1: can we classify single samples?\n",
    "#\n",
    "#\n",
    "#########################################################################\n",
    "''')\n",
    "\n",
    "def make_dataset1():\n",
    "    '''Make a dataset of single samples with labels from which distribution they come from'''\n",
    "    # now lets make some samples \n",
    "    lns      = min_max_scale(lognormal(size=bsize)) #log normal\n",
    "    powers   = min_max_scale(power(0.1,size=bsize)) #power law\n",
    "    norms    = min_max_scale(normal(size=bsize))    #normal\n",
    "    uniforms = min_max_scale(uniform(size=bsize))    #uniform\n",
    "    # add our data together\n",
    "    data = np.concatenate((lns,powers,norms,uniforms))\n",
    "    \n",
    "    # concatenate our labels\n",
    "    labels = np.concatenate((\n",
    "        (np.repeat(LOGNORMAL,bsize)),\n",
    "        (np.repeat(POWER,bsize)),\n",
    "        (np.repeat(NORM,bsize)),\n",
    "        (np.repeat(UNIFORM,bsize))))\n",
    "    tsize = len(labels)\n",
    "    \n",
    "    # make sure dimensionality and types are right\n",
    "    data = data.reshape((len(data),1))\n",
    "    data = data.astype(np.float32)\n",
    "    labels = labels.astype(np.int32)\n",
    "    labels = labels.reshape((len(data),))\n",
    "    \n",
    "    return data, labels, tsize\n",
    "\n",
    "# this will be the training data and validation data\n",
    "data, labels, tsize = make_dataset1()\n",
    "\n",
    "\n",
    "# this is the test data, this is kept separate to prove we can\n",
    "# actually work on the data we claim we can.\n",
    "#\n",
    "# Without test data, you might just have great performance on the\n",
    "# train set.\n",
    "test_data, test_labels, _ = make_dataset1()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(360, 1)\n",
      "(360,)\n",
      "(360, 4)\n",
      "(40, 4)\n",
      "(400, 4)\n"
     ]
    }
   ],
   "source": [
    "# utilities\n",
    "\n",
    "# now lets shuffle\n",
    "# If we're going to select a validation set we probably want to shuffle\n",
    "def joint_shuffle(arr1,arr2):\n",
    "    assert len(arr1) == len(arr2)\n",
    "    indices = np.arange(len(arr1))\n",
    "    np.random.shuffle(indices)\n",
    "    arr1[0:len(arr1)] = arr1[indices]\n",
    "    arr2[0:len(arr2)] = arr2[indices]\n",
    "\n",
    "# our data and labels are shuffled together\n",
    "joint_shuffle(data,labels)\n",
    "\n",
    "def split_validation(percent, data, labels):\n",
    "    ''' \n",
    "    split_validation splits a dataset of data and labels into\n",
    "    2 partitions at the percent mark\n",
    "    percent should be an int between 1 and 99\n",
    "    '''\n",
    "    s = int(percent * len(data) / 100)\n",
    "    tdata = data[0:s]\n",
    "    vdata = data[s:]\n",
    "    tlabels = labels[0:s]\n",
    "    vlabels = labels[s:]\n",
    "    return ((tdata,tlabels),(vdata,vlabels))\n",
    "\n",
    "# make a validation set from the train set\n",
    "train1, valid1 = split_validation(90, data, labels)\n",
    "\n",
    "print(train1[0].shape)\n",
    "print(train1[1].shape)\n",
    "\n",
    "enc1 = OneHotEncoder(handle_unknown='ignore',sparse=False)\n",
    "enc1.fit(train1[1].reshape(len(train1[1]),1))\n",
    "train1_y = enc1.transform(train1[1].reshape(len(train1[1]),1))\n",
    "print(train1_y.shape)\n",
    "valid1_y = enc1.transform(valid1[1].reshape(len(valid1[1]),1))\n",
    "print(valid1_y.shape)\n",
    "test1_y = enc1.transform(test_labels.reshape(len(test_labels),1))\n",
    "print(test1_y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We're building a MLP of 1 input layer node, 4 hidden layer nodes, and an output layer of 4 nodes. The output layer has 4 nodes because we have 4 classes that the neural network will output.\n",
      "Epoch 1/100\n",
      " 1/23 [>.............................] - ETA: 3s - loss: 1.3987 - accuracy: 0.3125"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/hindle1/.local/lib/python3.8/site-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(SGD, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23/23 [==============================] - 0s 5ms/step - loss: 1.4330 - accuracy: 0.2861 - val_loss: 1.3622 - val_accuracy: 0.2750\n",
      "Epoch 2/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3780 - accuracy: 0.2861 - val_loss: 1.3659 - val_accuracy: 0.3000\n",
      "Epoch 3/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3698 - accuracy: 0.2917 - val_loss: 1.3607 - val_accuracy: 0.4000\n",
      "Epoch 4/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3637 - accuracy: 0.3222 - val_loss: 1.3618 - val_accuracy: 0.4500\n",
      "Epoch 5/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3581 - accuracy: 0.4306 - val_loss: 1.3551 - val_accuracy: 0.3500\n",
      "Epoch 6/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3538 - accuracy: 0.4167 - val_loss: 1.3472 - val_accuracy: 0.4250\n",
      "Epoch 7/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3474 - accuracy: 0.4250 - val_loss: 1.3424 - val_accuracy: 0.4250\n",
      "Epoch 8/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3425 - accuracy: 0.4056 - val_loss: 1.3430 - val_accuracy: 0.3500\n",
      "Epoch 9/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3359 - accuracy: 0.4528 - val_loss: 1.3299 - val_accuracy: 0.3750\n",
      "Epoch 10/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3307 - accuracy: 0.4250 - val_loss: 1.3271 - val_accuracy: 0.3500\n",
      "Epoch 11/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3228 - accuracy: 0.4389 - val_loss: 1.3217 - val_accuracy: 0.3500\n",
      "Epoch 12/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3174 - accuracy: 0.4111 - val_loss: 1.3096 - val_accuracy: 0.4250\n",
      "Epoch 13/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3118 - accuracy: 0.4139 - val_loss: 1.3041 - val_accuracy: 0.4250\n",
      "Epoch 14/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3039 - accuracy: 0.4444 - val_loss: 1.2926 - val_accuracy: 0.4250\n",
      "Epoch 15/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2960 - accuracy: 0.4139 - val_loss: 1.2890 - val_accuracy: 0.3500\n",
      "Epoch 16/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2868 - accuracy: 0.4556 - val_loss: 1.2834 - val_accuracy: 0.3500\n",
      "Epoch 17/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2792 - accuracy: 0.4333 - val_loss: 1.2748 - val_accuracy: 0.3500\n",
      "Epoch 18/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2704 - accuracy: 0.4361 - val_loss: 1.2711 - val_accuracy: 0.3500\n",
      "Epoch 19/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2629 - accuracy: 0.4639 - val_loss: 1.2566 - val_accuracy: 0.3750\n",
      "Epoch 20/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2560 - accuracy: 0.3889 - val_loss: 1.2510 - val_accuracy: 0.3500\n",
      "Epoch 21/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2451 - accuracy: 0.4472 - val_loss: 1.2390 - val_accuracy: 0.4000\n",
      "Epoch 22/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2377 - accuracy: 0.4472 - val_loss: 1.2326 - val_accuracy: 0.3750\n",
      "Epoch 23/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2292 - accuracy: 0.4389 - val_loss: 1.2236 - val_accuracy: 0.4500\n",
      "Epoch 24/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2205 - accuracy: 0.4417 - val_loss: 1.2155 - val_accuracy: 0.4250\n",
      "Epoch 25/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2117 - accuracy: 0.4306 - val_loss: 1.2094 - val_accuracy: 0.4500\n",
      "Epoch 26/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2038 - accuracy: 0.4556 - val_loss: 1.2001 - val_accuracy: 0.4000\n",
      "Epoch 27/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1962 - accuracy: 0.4556 - val_loss: 1.1929 - val_accuracy: 0.4250\n",
      "Epoch 28/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1884 - accuracy: 0.4444 - val_loss: 1.1909 - val_accuracy: 0.3750\n",
      "Epoch 29/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1825 - accuracy: 0.4306 - val_loss: 1.1876 - val_accuracy: 0.3250\n",
      "Epoch 30/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1735 - accuracy: 0.4361 - val_loss: 1.1769 - val_accuracy: 0.4250\n",
      "Epoch 31/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1682 - accuracy: 0.4583 - val_loss: 1.1724 - val_accuracy: 0.4250\n",
      "Epoch 32/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1621 - accuracy: 0.4389 - val_loss: 1.1733 - val_accuracy: 0.3500\n",
      "Epoch 33/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1552 - accuracy: 0.4056 - val_loss: 1.1704 - val_accuracy: 0.4250\n",
      "Epoch 34/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1511 - accuracy: 0.4556 - val_loss: 1.1625 - val_accuracy: 0.4250\n",
      "Epoch 35/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1457 - accuracy: 0.4306 - val_loss: 1.1543 - val_accuracy: 0.4250\n",
      "Epoch 36/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1393 - accuracy: 0.4389 - val_loss: 1.1498 - val_accuracy: 0.4250\n",
      "Epoch 37/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1339 - accuracy: 0.4167 - val_loss: 1.1531 - val_accuracy: 0.3500\n",
      "Epoch 38/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1284 - accuracy: 0.4333 - val_loss: 1.1461 - val_accuracy: 0.4000\n",
      "Epoch 39/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1255 - accuracy: 0.4556 - val_loss: 1.1408 - val_accuracy: 0.4250\n",
      "Epoch 40/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1195 - accuracy: 0.4472 - val_loss: 1.1385 - val_accuracy: 0.4250\n",
      "Epoch 41/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1182 - accuracy: 0.4417 - val_loss: 1.1338 - val_accuracy: 0.4000\n",
      "Epoch 42/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1142 - accuracy: 0.4389 - val_loss: 1.1320 - val_accuracy: 0.3500\n",
      "Epoch 43/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1098 - accuracy: 0.4333 - val_loss: 1.1331 - val_accuracy: 0.4250\n",
      "Epoch 44/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1079 - accuracy: 0.4417 - val_loss: 1.1302 - val_accuracy: 0.4500\n",
      "Epoch 45/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1043 - accuracy: 0.4306 - val_loss: 1.1296 - val_accuracy: 0.3750\n",
      "Epoch 46/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1012 - accuracy: 0.4222 - val_loss: 1.1289 - val_accuracy: 0.4000\n",
      "Epoch 47/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0998 - accuracy: 0.4306 - val_loss: 1.1302 - val_accuracy: 0.4250\n",
      "Epoch 48/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0960 - accuracy: 0.4583 - val_loss: 1.1280 - val_accuracy: 0.4250\n",
      "Epoch 49/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0935 - accuracy: 0.4778 - val_loss: 1.1261 - val_accuracy: 0.4000\n",
      "Epoch 50/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0922 - accuracy: 0.4389 - val_loss: 1.1263 - val_accuracy: 0.4000\n",
      "Epoch 51/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0913 - accuracy: 0.4306 - val_loss: 1.1222 - val_accuracy: 0.4000\n",
      "Epoch 52/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0882 - accuracy: 0.4028 - val_loss: 1.1266 - val_accuracy: 0.4250\n",
      "Epoch 53/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0870 - accuracy: 0.4222 - val_loss: 1.1241 - val_accuracy: 0.3750\n",
      "Epoch 54/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0838 - accuracy: 0.4167 - val_loss: 1.1262 - val_accuracy: 0.4000\n",
      "Epoch 55/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0842 - accuracy: 0.4472 - val_loss: 1.1267 - val_accuracy: 0.4000\n",
      "Epoch 56/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0819 - accuracy: 0.4667 - val_loss: 1.1258 - val_accuracy: 0.4000\n",
      "Epoch 57/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0821 - accuracy: 0.4500 - val_loss: 1.1267 - val_accuracy: 0.4750\n",
      "Epoch 58/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0799 - accuracy: 0.4472 - val_loss: 1.1242 - val_accuracy: 0.4250\n",
      "Epoch 59/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0784 - accuracy: 0.4222 - val_loss: 1.1288 - val_accuracy: 0.4500\n",
      "Epoch 60/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0771 - accuracy: 0.4750 - val_loss: 1.1305 - val_accuracy: 0.3500\n",
      "Epoch 61/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0771 - accuracy: 0.4889 - val_loss: 1.1273 - val_accuracy: 0.3500\n",
      "Epoch 62/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0756 - accuracy: 0.4361 - val_loss: 1.1353 - val_accuracy: 0.3500\n",
      "Epoch 63/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0756 - accuracy: 0.4194 - val_loss: 1.1310 - val_accuracy: 0.4500\n",
      "Epoch 64/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0748 - accuracy: 0.4528 - val_loss: 1.1288 - val_accuracy: 0.4750\n",
      "Epoch 65/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0743 - accuracy: 0.4806 - val_loss: 1.1263 - val_accuracy: 0.4500\n",
      "Epoch 66/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0724 - accuracy: 0.4278 - val_loss: 1.1351 - val_accuracy: 0.3500\n",
      "Epoch 67/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0728 - accuracy: 0.4444 - val_loss: 1.1246 - val_accuracy: 0.3750\n",
      "Epoch 68/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0716 - accuracy: 0.4667 - val_loss: 1.1311 - val_accuracy: 0.4000\n",
      "Epoch 69/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0708 - accuracy: 0.4889 - val_loss: 1.1280 - val_accuracy: 0.4250\n",
      "Epoch 70/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0709 - accuracy: 0.4528 - val_loss: 1.1223 - val_accuracy: 0.3750\n",
      "Epoch 71/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0695 - accuracy: 0.4083 - val_loss: 1.1240 - val_accuracy: 0.3750\n",
      "Epoch 72/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0685 - accuracy: 0.4806 - val_loss: 1.1251 - val_accuracy: 0.4250\n",
      "Epoch 73/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0675 - accuracy: 0.4861 - val_loss: 1.1296 - val_accuracy: 0.3250\n",
      "Epoch 74/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0698 - accuracy: 0.4528 - val_loss: 1.1242 - val_accuracy: 0.4000\n",
      "Epoch 75/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0670 - accuracy: 0.4806 - val_loss: 1.1264 - val_accuracy: 0.4000\n",
      "Epoch 76/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0673 - accuracy: 0.4944 - val_loss: 1.1274 - val_accuracy: 0.4250\n",
      "Epoch 77/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0665 - accuracy: 0.4611 - val_loss: 1.1299 - val_accuracy: 0.4250\n",
      "Epoch 78/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0661 - accuracy: 0.4944 - val_loss: 1.1299 - val_accuracy: 0.4000\n",
      "Epoch 79/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0675 - accuracy: 0.4917 - val_loss: 1.1282 - val_accuracy: 0.4250\n",
      "Epoch 80/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0665 - accuracy: 0.4889 - val_loss: 1.1306 - val_accuracy: 0.4250\n",
      "Epoch 81/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0656 - accuracy: 0.4972 - val_loss: 1.1346 - val_accuracy: 0.4750\n",
      "Epoch 82/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0661 - accuracy: 0.5000 - val_loss: 1.1266 - val_accuracy: 0.3750\n",
      "Epoch 83/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0645 - accuracy: 0.4556 - val_loss: 1.1268 - val_accuracy: 0.4000\n",
      "Epoch 84/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0644 - accuracy: 0.4583 - val_loss: 1.1245 - val_accuracy: 0.4500\n",
      "Epoch 85/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0634 - accuracy: 0.5056 - val_loss: 1.1261 - val_accuracy: 0.4250\n",
      "Epoch 86/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0622 - accuracy: 0.4556 - val_loss: 1.1280 - val_accuracy: 0.4250\n",
      "Epoch 87/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0637 - accuracy: 0.4778 - val_loss: 1.1257 - val_accuracy: 0.4500\n",
      "Epoch 88/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0633 - accuracy: 0.4222 - val_loss: 1.1269 - val_accuracy: 0.4000\n",
      "Epoch 89/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0637 - accuracy: 0.4750 - val_loss: 1.1303 - val_accuracy: 0.4000\n",
      "Epoch 90/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0622 - accuracy: 0.5139 - val_loss: 1.1267 - val_accuracy: 0.4500\n",
      "Epoch 91/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0631 - accuracy: 0.4944 - val_loss: 1.1249 - val_accuracy: 0.4000\n",
      "Epoch 92/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0627 - accuracy: 0.4528 - val_loss: 1.1264 - val_accuracy: 0.4250\n",
      "Epoch 93/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0623 - accuracy: 0.5111 - val_loss: 1.1247 - val_accuracy: 0.4250\n",
      "Epoch 94/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0622 - accuracy: 0.5028 - val_loss: 1.1276 - val_accuracy: 0.3750\n",
      "Epoch 95/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0599 - accuracy: 0.5000 - val_loss: 1.1300 - val_accuracy: 0.4000\n",
      "Epoch 96/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0604 - accuracy: 0.5000 - val_loss: 1.1282 - val_accuracy: 0.4000\n",
      "Epoch 97/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0609 - accuracy: 0.4806 - val_loss: 1.1273 - val_accuracy: 0.4250\n",
      "Epoch 98/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0600 - accuracy: 0.4694 - val_loss: 1.1289 - val_accuracy: 0.4000\n",
      "Epoch 99/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0600 - accuracy: 0.4889 - val_loss: 1.1347 - val_accuracy: 0.4250\n",
      "Epoch 100/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0601 - accuracy: 0.5333 - val_loss: 1.1294 - val_accuracy: 0.4000\n",
      "[('tp', 87), ('tn', 38), ('fp', 47), ('fn', 5)]\n",
      "13/13 [==============================] - 0s 900us/step - loss: 1.1083 - accuracy: 0.5200\n",
      "Scores: [1.1082683801651, 0.5199999809265137]\n"
     ]
    }
   ],
   "source": [
    "# build our classifier\n",
    "\n",
    "print(\"We're building a MLP of 1 input layer node, 4 hidden layer nodes, and an output layer of 4 nodes. The output layer has 4 nodes because we have 4 classes that the neural network will output.\")\n",
    "cnet = Sequential()\n",
    "cnet.add(Dense(4,input_shape=(1,),activation=\"sigmoid\"))\n",
    "cnet.add(Dense(4,activation=\"softmax\"))\n",
    "copt = SGD(lr=0.1)\n",
    "# opt = Adam(lr=0.1)\n",
    "cnet.compile(loss=\"categorical_crossentropy\", optimizer=copt, metrics=[\"accuracy\"])\n",
    "history = cnet.fit(train1[0], train1_y, validation_data=(valid1[0], valid1_y),\n",
    "\t            epochs=100, batch_size=16)\n",
    "\n",
    "#score = cnet.evaluate(test_data, test_labels)\n",
    "#print(\"Scores: %s\" % score)\n",
    "classify = predict_classes(cnet,test_data)\n",
    "print(theautil.classifications(classify,test_labels))\n",
    "score = cnet.evaluate(test_data, test1_y)\n",
    "print(\"Scores: %s\" % score)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Experiment 2\n",
    "\n",
    "* Given 40 samples what distribution does it come from?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "########################################################################\n",
      "# Experiment 2: can we classify a sample of data?\n",
      "#\n",
      "#\n",
      "#########################################################################\n",
      "\n",
      "In this example we're going to input 40 values from a single distribution, and we'll see if we can classify the distribution.\n",
      "At this point we have a weird decision to make, how many neurons in the hidden layer?\n",
      "Epoch 1/100\n",
      " 1/23 [>.............................] - ETA: 3s - loss: 1.5122 - accuracy: 0.2500"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/hindle1/.local/lib/python3.8/site-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(SGD, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23/23 [==============================] - 0s 10ms/step - loss: 1.4236 - accuracy: 0.2444 - val_loss: 1.3809 - val_accuracy: 0.3250\n",
      "Epoch 2/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3939 - accuracy: 0.2167 - val_loss: 1.3812 - val_accuracy: 0.3000\n",
      "Epoch 3/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3805 - accuracy: 0.2972 - val_loss: 1.3877 - val_accuracy: 0.2250\n",
      "Epoch 4/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3718 - accuracy: 0.3278 - val_loss: 1.3802 - val_accuracy: 0.4250\n",
      "Epoch 5/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3480 - accuracy: 0.3056 - val_loss: 1.3292 - val_accuracy: 0.4000\n",
      "Epoch 6/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3253 - accuracy: 0.4083 - val_loss: 1.3208 - val_accuracy: 0.4750\n",
      "Epoch 7/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3008 - accuracy: 0.4361 - val_loss: 1.2810 - val_accuracy: 0.4750\n",
      "Epoch 8/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2641 - accuracy: 0.4139 - val_loss: 1.2445 - val_accuracy: 0.5000\n",
      "Epoch 9/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2195 - accuracy: 0.4611 - val_loss: 1.2019 - val_accuracy: 0.4000\n",
      "Epoch 10/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1636 - accuracy: 0.4694 - val_loss: 1.1196 - val_accuracy: 0.5250\n",
      "Epoch 11/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1039 - accuracy: 0.4833 - val_loss: 1.0601 - val_accuracy: 0.5000\n",
      "Epoch 12/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0381 - accuracy: 0.5000 - val_loss: 1.0050 - val_accuracy: 0.5000\n",
      "Epoch 13/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9795 - accuracy: 0.4861 - val_loss: 0.9549 - val_accuracy: 0.5000\n",
      "Epoch 14/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9297 - accuracy: 0.5111 - val_loss: 0.9083 - val_accuracy: 0.4750\n",
      "Epoch 15/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8853 - accuracy: 0.5306 - val_loss: 0.8688 - val_accuracy: 0.5000\n",
      "Epoch 16/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8516 - accuracy: 0.5167 - val_loss: 0.8378 - val_accuracy: 0.7000\n",
      "Epoch 17/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8236 - accuracy: 0.6167 - val_loss: 0.8102 - val_accuracy: 0.5500\n",
      "Epoch 18/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8034 - accuracy: 0.5222 - val_loss: 0.7916 - val_accuracy: 0.7000\n",
      "Epoch 19/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7842 - accuracy: 0.5694 - val_loss: 0.7810 - val_accuracy: 0.5750\n",
      "Epoch 20/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7679 - accuracy: 0.5833 - val_loss: 0.7652 - val_accuracy: 0.6000\n",
      "Epoch 21/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7569 - accuracy: 0.6306 - val_loss: 0.7493 - val_accuracy: 0.6250\n",
      "Epoch 22/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7455 - accuracy: 0.6083 - val_loss: 0.7552 - val_accuracy: 0.6250\n",
      "Epoch 23/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7393 - accuracy: 0.6333 - val_loss: 0.7329 - val_accuracy: 0.5500\n",
      "Epoch 24/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7283 - accuracy: 0.5889 - val_loss: 0.7254 - val_accuracy: 0.7000\n",
      "Epoch 25/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7225 - accuracy: 0.6750 - val_loss: 0.7225 - val_accuracy: 0.6000\n",
      "Epoch 26/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7156 - accuracy: 0.6194 - val_loss: 0.7171 - val_accuracy: 0.6500\n",
      "Epoch 27/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7080 - accuracy: 0.6194 - val_loss: 0.7138 - val_accuracy: 0.5000\n",
      "Epoch 28/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7052 - accuracy: 0.6028 - val_loss: 0.7068 - val_accuracy: 0.6250\n",
      "Epoch 29/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7003 - accuracy: 0.6250 - val_loss: 0.6983 - val_accuracy: 0.6000\n",
      "Epoch 30/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6925 - accuracy: 0.6833 - val_loss: 0.7021 - val_accuracy: 0.6000\n",
      "Epoch 31/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6911 - accuracy: 0.6194 - val_loss: 0.6887 - val_accuracy: 0.7250\n",
      "Epoch 32/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6833 - accuracy: 0.6972 - val_loss: 0.6853 - val_accuracy: 0.8250\n",
      "Epoch 33/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6787 - accuracy: 0.6861 - val_loss: 0.6788 - val_accuracy: 0.6750\n",
      "Epoch 34/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6738 - accuracy: 0.6861 - val_loss: 0.6741 - val_accuracy: 0.7000\n",
      "Epoch 35/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6688 - accuracy: 0.6611 - val_loss: 0.6731 - val_accuracy: 0.7250\n",
      "Epoch 36/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6651 - accuracy: 0.6806 - val_loss: 0.6659 - val_accuracy: 0.7000\n",
      "Epoch 37/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6627 - accuracy: 0.6806 - val_loss: 0.6649 - val_accuracy: 0.6750\n",
      "Epoch 38/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6571 - accuracy: 0.6833 - val_loss: 0.6663 - val_accuracy: 0.6500\n",
      "Epoch 39/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6552 - accuracy: 0.6917 - val_loss: 0.6716 - val_accuracy: 0.6750\n",
      "Epoch 40/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6487 - accuracy: 0.6806 - val_loss: 0.6598 - val_accuracy: 0.6750\n",
      "Epoch 41/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6406 - accuracy: 0.7139 - val_loss: 0.6455 - val_accuracy: 0.7000\n",
      "Epoch 42/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6383 - accuracy: 0.7000 - val_loss: 0.6523 - val_accuracy: 0.6750\n",
      "Epoch 43/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6315 - accuracy: 0.7194 - val_loss: 0.6603 - val_accuracy: 0.6500\n",
      "Epoch 44/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6295 - accuracy: 0.6750 - val_loss: 0.6407 - val_accuracy: 0.7000\n",
      "Epoch 45/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6237 - accuracy: 0.6889 - val_loss: 0.6343 - val_accuracy: 0.7000\n",
      "Epoch 46/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6199 - accuracy: 0.6917 - val_loss: 0.6356 - val_accuracy: 0.6500\n",
      "Epoch 47/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6162 - accuracy: 0.6806 - val_loss: 0.6323 - val_accuracy: 0.7750\n",
      "Epoch 48/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6093 - accuracy: 0.6972 - val_loss: 0.6252 - val_accuracy: 0.7000\n",
      "Epoch 49/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6049 - accuracy: 0.6944 - val_loss: 0.6257 - val_accuracy: 0.7000\n",
      "Epoch 50/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5980 - accuracy: 0.6889 - val_loss: 0.6181 - val_accuracy: 0.8000\n",
      "Epoch 51/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5975 - accuracy: 0.6861 - val_loss: 0.6293 - val_accuracy: 0.6750\n",
      "Epoch 52/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5905 - accuracy: 0.6917 - val_loss: 0.6171 - val_accuracy: 0.6500\n",
      "Epoch 53/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5865 - accuracy: 0.7000 - val_loss: 0.6088 - val_accuracy: 0.6500\n",
      "Epoch 54/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5811 - accuracy: 0.7056 - val_loss: 0.6188 - val_accuracy: 0.8000\n",
      "Epoch 55/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5804 - accuracy: 0.7000 - val_loss: 0.6072 - val_accuracy: 0.6500\n",
      "Epoch 56/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5741 - accuracy: 0.7083 - val_loss: 0.6030 - val_accuracy: 0.7000\n",
      "Epoch 57/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5695 - accuracy: 0.6833 - val_loss: 0.6216 - val_accuracy: 0.6500\n",
      "Epoch 58/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5675 - accuracy: 0.6861 - val_loss: 0.6075 - val_accuracy: 0.6500\n",
      "Epoch 59/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5608 - accuracy: 0.7028 - val_loss: 0.6038 - val_accuracy: 0.6500\n",
      "Epoch 60/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5598 - accuracy: 0.7000 - val_loss: 0.5937 - val_accuracy: 0.6500\n",
      "Epoch 61/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5523 - accuracy: 0.6972 - val_loss: 0.5884 - val_accuracy: 0.6500\n",
      "Epoch 62/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5494 - accuracy: 0.6917 - val_loss: 0.5896 - val_accuracy: 0.6750\n",
      "Epoch 63/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5440 - accuracy: 0.7028 - val_loss: 0.5810 - val_accuracy: 0.7500\n",
      "Epoch 64/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5423 - accuracy: 0.6972 - val_loss: 0.5933 - val_accuracy: 0.6500\n",
      "Epoch 65/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5409 - accuracy: 0.6944 - val_loss: 0.5982 - val_accuracy: 0.6500\n",
      "Epoch 66/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5344 - accuracy: 0.7250 - val_loss: 0.5774 - val_accuracy: 0.6500\n",
      "Epoch 67/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5334 - accuracy: 0.7028 - val_loss: 0.5864 - val_accuracy: 0.6750\n",
      "Epoch 68/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5295 - accuracy: 0.6889 - val_loss: 0.5829 - val_accuracy: 0.6500\n",
      "Epoch 69/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5261 - accuracy: 0.7000 - val_loss: 0.5674 - val_accuracy: 0.7250\n",
      "Epoch 70/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5231 - accuracy: 0.7083 - val_loss: 0.5605 - val_accuracy: 0.7000\n",
      "Epoch 71/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5202 - accuracy: 0.7028 - val_loss: 0.5695 - val_accuracy: 0.7000\n",
      "Epoch 72/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5155 - accuracy: 0.7111 - val_loss: 0.5580 - val_accuracy: 0.6750\n",
      "Epoch 73/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5129 - accuracy: 0.7194 - val_loss: 0.5574 - val_accuracy: 0.7000\n",
      "Epoch 74/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5077 - accuracy: 0.7222 - val_loss: 0.5683 - val_accuracy: 0.6750\n",
      "Epoch 75/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5067 - accuracy: 0.7139 - val_loss: 0.5496 - val_accuracy: 0.7000\n",
      "Epoch 76/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5024 - accuracy: 0.7167 - val_loss: 0.5661 - val_accuracy: 0.6500\n",
      "Epoch 77/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4997 - accuracy: 0.7222 - val_loss: 0.5555 - val_accuracy: 0.7000\n",
      "Epoch 78/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4993 - accuracy: 0.7028 - val_loss: 0.5461 - val_accuracy: 0.7250\n",
      "Epoch 79/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4934 - accuracy: 0.7361 - val_loss: 0.5515 - val_accuracy: 0.7000\n",
      "Epoch 80/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4917 - accuracy: 0.7000 - val_loss: 0.5437 - val_accuracy: 0.6500\n",
      "Epoch 81/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4910 - accuracy: 0.7194 - val_loss: 0.5472 - val_accuracy: 0.7250\n",
      "Epoch 82/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4898 - accuracy: 0.7028 - val_loss: 0.5460 - val_accuracy: 0.6750\n",
      "Epoch 83/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4859 - accuracy: 0.7167 - val_loss: 0.5457 - val_accuracy: 0.6500\n",
      "Epoch 84/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4822 - accuracy: 0.7278 - val_loss: 0.5337 - val_accuracy: 0.7000\n",
      "Epoch 85/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4836 - accuracy: 0.7028 - val_loss: 0.5594 - val_accuracy: 0.6500\n",
      "Epoch 86/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4755 - accuracy: 0.7556 - val_loss: 0.5340 - val_accuracy: 0.7250\n",
      "Epoch 87/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4759 - accuracy: 0.7222 - val_loss: 0.5589 - val_accuracy: 0.6500\n",
      "Epoch 88/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4750 - accuracy: 0.7111 - val_loss: 0.5294 - val_accuracy: 0.7250\n",
      "Epoch 89/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4742 - accuracy: 0.7417 - val_loss: 0.5611 - val_accuracy: 0.6500\n",
      "Epoch 90/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4707 - accuracy: 0.7222 - val_loss: 0.5270 - val_accuracy: 0.7250\n",
      "Epoch 91/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4710 - accuracy: 0.7056 - val_loss: 0.5370 - val_accuracy: 0.6500\n",
      "Epoch 92/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4652 - accuracy: 0.7444 - val_loss: 0.5329 - val_accuracy: 0.6500\n",
      "Epoch 93/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4619 - accuracy: 0.7250 - val_loss: 0.5279 - val_accuracy: 0.7000\n",
      "Epoch 94/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4626 - accuracy: 0.7417 - val_loss: 0.5456 - val_accuracy: 0.6500\n",
      "Epoch 95/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4607 - accuracy: 0.7250 - val_loss: 0.5247 - val_accuracy: 0.6750\n",
      "Epoch 96/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4586 - accuracy: 0.7278 - val_loss: 0.5272 - val_accuracy: 0.7000\n",
      "Epoch 97/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4584 - accuracy: 0.7111 - val_loss: 0.5236 - val_accuracy: 0.7000\n",
      "Epoch 98/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4591 - accuracy: 0.7167 - val_loss: 0.5176 - val_accuracy: 0.7250\n",
      "Epoch 99/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4530 - accuracy: 0.7444 - val_loss: 0.5325 - val_accuracy: 0.6500\n",
      "Epoch 100/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4548 - accuracy: 0.7250 - val_loss: 0.5227 - val_accuracy: 0.7000\n",
      "13/13 [==============================] - 0s 957us/step - loss: 0.4696 - accuracy: 0.7275\n",
      "['loss', 'accuracy']\n",
      "Scores: [0.46964675188064575, 0.7275000214576721]\n"
     ]
    }
   ],
   "source": [
    "print('''\n",
    "########################################################################\n",
    "# Experiment 2: can we classify a sample of data?\n",
    "#\n",
    "#\n",
    "#########################################################################\n",
    "''')\n",
    "print(\"In this example we're going to input 40 values from a single distribution, and we'll see if we can classify the distribution.\")\n",
    "\n",
    "width=40\n",
    "\n",
    "def make_widedataset(width=width):\n",
    "    # we're going to make rows of 40 features unsorted\n",
    "    wlns      = min_max_scale(lognormal(size=(bsize,width))) #log normal\n",
    "    wpowers   = min_max_scale(power(0.1,size=(bsize,width))) #power law\n",
    "    wnorms    = min_max_scale(normal(size=(bsize,width)))    #normal\n",
    "    wuniforms = min_max_scale(uniform(size=(bsize,width)))    #uniform\n",
    "    \n",
    "    wdata = np.concatenate((wlns,wpowers,wnorms,wuniforms))\n",
    "    \n",
    "    # concatenate our labels\n",
    "    wlabels = np.concatenate((\n",
    "        (np.repeat(LOGNORMAL,bsize)),\n",
    "        (np.repeat(POWER,bsize)),\n",
    "        (np.repeat(NORM,bsize)),\n",
    "        (np.repeat(UNIFORM,bsize))))\n",
    "    \n",
    "    joint_shuffle(wdata,wlabels)\n",
    "    wdata = wdata.astype(np.float32)\n",
    "    wlabels = wlabels.astype(np.int32)\n",
    "    wlabels = wlabels.reshape((len(data),))\n",
    "    return wdata, wlabels\n",
    "\n",
    "# make our train sets\n",
    "wdata, wlabels = make_widedataset()\n",
    "# make our test sets\n",
    "test_wdata, test_wlabels = make_widedataset()\n",
    "\n",
    "# split out our validation set\n",
    "wtrain, wvalid = split_validation(90, wdata, wlabels)\n",
    "print(\"At this point we have a weird decision to make, how many neurons in the hidden layer?\")\n",
    "\n",
    "encwc = OneHotEncoder(handle_unknown='ignore', sparse=False)\n",
    "encwc.fit(wtrain[1].reshape(len(wtrain[1]),1))\n",
    "wtrain_y = encwc.transform(wtrain[1].reshape(len(wtrain[1]),1))\n",
    "wvalid_y = encwc.transform(wvalid[1].reshape(len(wvalid[1]),1))\n",
    "wtest_y  = encwc.transform(test_wlabels.reshape(len(test_wlabels),1))\n",
    "\n",
    "# wcnet = theanets.Classifier([width,width/4,4]) #267\n",
    "wcnet = Sequential()\n",
    "wcnet.add(Dense(width,input_shape=(width,),activation=\"sigmoid\"))\n",
    "wcnet.add(Dense(int(width/4),activation=\"sigmoid\"))\n",
    "wcnet.add(Dense(4,activation=\"softmax\"))\n",
    "wcnet.compile(loss=\"categorical_crossentropy\", optimizer=SGD(lr=0.1), metrics=[\"accuracy\"])\n",
    "history = wcnet.fit(wtrain[0], wtrain_y, validation_data=(wvalid[0], wvalid_y),\n",
    "\t            epochs=100, batch_size=16)\n",
    "score = wcnet.evaluate(test_wdata, wtest_y)\n",
    "print(wcnet.metrics_names)\n",
    "print(\"Scores: %s\" % score)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('tp', 95), ('tn', 96), ('fp', 4), ('fn', 5)]\n",
      "13/13 [==============================] - 0s 1ms/step - loss: 0.4696 - accuracy: 0.7275\n",
      "Scores: [0.46964675188064575, 0.7275000214576721]\n",
      "Ok that was neat, it definitely worked better, it had more data though.\n",
      "But what if we help it out, and we sort the values so that the first and last bins are always the min and max values?\n"
     ]
    }
   ],
   "source": [
    "\n",
    "classify = predict_classes(wcnet,test_wdata)\n",
    "print(theautil.classifications(classify,test_wlabels))\n",
    "score = wcnet.evaluate(test_wdata, wtest_y)\n",
    "print(\"Scores: %s\" % score)\n",
    "\n",
    "# # You could try some of these alternative setups\n",
    "# \n",
    "# [width,4]) #248\n",
    "# [width,width/2,4]) #271\n",
    "# [width,width,4]) #289\n",
    "# [width,width*2,4]) #292\n",
    "# [width,width/2,width/4,4]) #270\n",
    "# [width,width/2,width/4,width/8,width/16,4]) #232\n",
    "# [width,width*8,4]) #304\n",
    "\n",
    "print(\"Ok that was neat, it definitely worked better, it had more data though.\")\n",
    "\n",
    "print(\"But what if we help it out, and we sort the values so that the first and last bins are always the min and max values?\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Experiment 3\n",
    "\n",
    "* Given 40 sorted samples what distribution does it come from?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "########################################################################\n",
      "# Experiment 3: can we classify a SORTED sample of data?\n",
      "#\n",
      "#\n",
      "#########################################################################\n",
      "\n",
      "Sorting the data\n",
      "Epoch 1/100\n",
      "23/23 [==============================] - 0s 5ms/step - loss: 1.4488 - accuracy: 0.2139 - val_loss: 1.4085 - val_accuracy: 0.2000\n",
      "Epoch 2/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3945 - accuracy: 0.2556 - val_loss: 1.3938 - val_accuracy: 0.2000\n",
      "Epoch 3/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3804 - accuracy: 0.2944 - val_loss: 1.3680 - val_accuracy: 0.5250\n",
      "Epoch 4/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3706 - accuracy: 0.3833 - val_loss: 1.3501 - val_accuracy: 0.2750\n",
      "Epoch 5/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3574 - accuracy: 0.3000 - val_loss: 1.3602 - val_accuracy: 0.2000\n",
      "Epoch 6/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3407 - accuracy: 0.3667 - val_loss: 1.3349 - val_accuracy: 0.4250\n",
      "Epoch 7/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3215 - accuracy: 0.4667 - val_loss: 1.3234 - val_accuracy: 0.4750\n",
      "Epoch 8/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2902 - accuracy: 0.5222 - val_loss: 1.2780 - val_accuracy: 0.5250\n",
      "Epoch 9/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2587 - accuracy: 0.5417 - val_loss: 1.2490 - val_accuracy: 0.5750\n",
      "Epoch 10/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2247 - accuracy: 0.4917 - val_loss: 1.1940 - val_accuracy: 0.4750\n",
      "Epoch 11/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1688 - accuracy: 0.5056 - val_loss: 1.1491 - val_accuracy: 0.5000\n",
      "Epoch 12/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1139 - accuracy: 0.5167 - val_loss: 1.0911 - val_accuracy: 0.5750\n",
      "Epoch 13/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0524 - accuracy: 0.5833 - val_loss: 1.0389 - val_accuracy: 0.5250\n",
      "Epoch 14/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9965 - accuracy: 0.5750 - val_loss: 0.9742 - val_accuracy: 0.5750\n",
      "Epoch 15/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9401 - accuracy: 0.5833 - val_loss: 0.9232 - val_accuracy: 0.6000\n",
      "Epoch 16/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8913 - accuracy: 0.5889 - val_loss: 0.8810 - val_accuracy: 0.5750\n",
      "Epoch 17/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8471 - accuracy: 0.6028 - val_loss: 0.8442 - val_accuracy: 0.6500\n",
      "Epoch 18/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8095 - accuracy: 0.6528 - val_loss: 0.8059 - val_accuracy: 0.7000\n",
      "Epoch 19/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7789 - accuracy: 0.6833 - val_loss: 0.7794 - val_accuracy: 0.6500\n",
      "Epoch 20/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7535 - accuracy: 0.6694 - val_loss: 0.7550 - val_accuracy: 0.9250\n",
      "Epoch 21/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7315 - accuracy: 0.7139 - val_loss: 0.7418 - val_accuracy: 0.6500\n",
      "Epoch 22/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7130 - accuracy: 0.6583 - val_loss: 0.7187 - val_accuracy: 0.7250\n",
      "Epoch 23/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6917 - accuracy: 0.7250 - val_loss: 0.7010 - val_accuracy: 0.7000\n",
      "Epoch 24/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6760 - accuracy: 0.7056 - val_loss: 0.6887 - val_accuracy: 0.9500\n",
      "Epoch 25/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6608 - accuracy: 0.7389 - val_loss: 0.6902 - val_accuracy: 0.6500\n",
      "Epoch 26/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6513 - accuracy: 0.7222 - val_loss: 0.6644 - val_accuracy: 0.6750\n",
      "Epoch 27/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6359 - accuracy: 0.7333 - val_loss: 0.6529 - val_accuracy: 0.7250\n",
      "Epoch 28/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6240 - accuracy: 0.7361 - val_loss: 0.6375 - val_accuracy: 0.7500\n",
      "Epoch 29/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6118 - accuracy: 0.7444 - val_loss: 0.6305 - val_accuracy: 0.8250\n",
      "Epoch 30/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5973 - accuracy: 0.7778 - val_loss: 0.6310 - val_accuracy: 0.6750\n",
      "Epoch 31/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5873 - accuracy: 0.7722 - val_loss: 0.6227 - val_accuracy: 0.6750\n",
      "Epoch 32/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5762 - accuracy: 0.7583 - val_loss: 0.6095 - val_accuracy: 0.6750\n",
      "Epoch 33/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5669 - accuracy: 0.7556 - val_loss: 0.5896 - val_accuracy: 0.7500\n",
      "Epoch 34/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5577 - accuracy: 0.7472 - val_loss: 0.5930 - val_accuracy: 0.6750\n",
      "Epoch 35/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5456 - accuracy: 0.7667 - val_loss: 0.5797 - val_accuracy: 0.9500\n",
      "Epoch 36/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5392 - accuracy: 0.7611 - val_loss: 0.5659 - val_accuracy: 0.7250\n",
      "Epoch 37/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5309 - accuracy: 0.7583 - val_loss: 0.5661 - val_accuracy: 0.9500\n",
      "Epoch 38/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5211 - accuracy: 0.7889 - val_loss: 0.5577 - val_accuracy: 0.9500\n",
      "Epoch 39/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5119 - accuracy: 0.7917 - val_loss: 0.5512 - val_accuracy: 0.9000\n",
      "Epoch 40/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.5063 - accuracy: 0.7972 - val_loss: 0.5402 - val_accuracy: 0.7250\n",
      "Epoch 41/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4969 - accuracy: 0.7639 - val_loss: 0.5410 - val_accuracy: 0.9250\n",
      "Epoch 42/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4890 - accuracy: 0.8139 - val_loss: 0.5306 - val_accuracy: 0.8250\n",
      "Epoch 43/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4856 - accuracy: 0.7694 - val_loss: 0.5243 - val_accuracy: 0.7250\n",
      "Epoch 44/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4796 - accuracy: 0.7694 - val_loss: 0.5265 - val_accuracy: 0.7000\n",
      "Epoch 45/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4726 - accuracy: 0.7694 - val_loss: 0.5213 - val_accuracy: 0.8000\n",
      "Epoch 46/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4657 - accuracy: 0.7722 - val_loss: 0.5325 - val_accuracy: 0.7250\n",
      "Epoch 47/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4595 - accuracy: 0.8111 - val_loss: 0.5341 - val_accuracy: 0.7000\n",
      "Epoch 48/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4550 - accuracy: 0.8500 - val_loss: 0.5383 - val_accuracy: 0.6500\n",
      "Epoch 49/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4548 - accuracy: 0.7611 - val_loss: 0.5062 - val_accuracy: 0.8500\n",
      "Epoch 50/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4473 - accuracy: 0.7889 - val_loss: 0.4991 - val_accuracy: 0.8000\n",
      "Epoch 51/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4438 - accuracy: 0.8056 - val_loss: 0.4956 - val_accuracy: 0.9500\n",
      "Epoch 52/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4391 - accuracy: 0.8333 - val_loss: 0.4932 - val_accuracy: 0.9250\n",
      "Epoch 53/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4339 - accuracy: 0.8444 - val_loss: 0.4924 - val_accuracy: 0.7250\n",
      "Epoch 54/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4292 - accuracy: 0.8361 - val_loss: 0.4919 - val_accuracy: 0.8250\n",
      "Epoch 55/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4244 - accuracy: 0.8333 - val_loss: 0.4997 - val_accuracy: 0.6500\n",
      "Epoch 56/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4222 - accuracy: 0.8500 - val_loss: 0.5049 - val_accuracy: 0.9000\n",
      "Epoch 57/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4204 - accuracy: 0.8472 - val_loss: 0.4858 - val_accuracy: 0.8500\n",
      "Epoch 58/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4155 - accuracy: 0.8278 - val_loss: 0.4820 - val_accuracy: 0.8000\n",
      "Epoch 59/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4132 - accuracy: 0.8139 - val_loss: 0.4839 - val_accuracy: 0.7250\n",
      "Epoch 60/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4116 - accuracy: 0.8389 - val_loss: 0.4749 - val_accuracy: 0.9500\n",
      "Epoch 61/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4065 - accuracy: 0.8417 - val_loss: 0.4675 - val_accuracy: 0.9500\n",
      "Epoch 62/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4030 - accuracy: 0.8556 - val_loss: 0.4729 - val_accuracy: 0.7250\n",
      "Epoch 63/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.4011 - accuracy: 0.8417 - val_loss: 0.4662 - val_accuracy: 0.9500\n",
      "Epoch 64/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3964 - accuracy: 0.8972 - val_loss: 0.4667 - val_accuracy: 0.8000\n",
      "Epoch 65/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3943 - accuracy: 0.8889 - val_loss: 0.4623 - val_accuracy: 0.7250\n",
      "Epoch 66/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3906 - accuracy: 0.8611 - val_loss: 0.4587 - val_accuracy: 0.9500\n",
      "Epoch 67/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3878 - accuracy: 0.8889 - val_loss: 0.4587 - val_accuracy: 0.7500\n",
      "Epoch 68/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3852 - accuracy: 0.8806 - val_loss: 0.4511 - val_accuracy: 0.9500\n",
      "Epoch 69/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3810 - accuracy: 0.8694 - val_loss: 0.4490 - val_accuracy: 0.9250\n",
      "Epoch 70/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3784 - accuracy: 0.9139 - val_loss: 0.4518 - val_accuracy: 0.9500\n",
      "Epoch 71/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3755 - accuracy: 0.9306 - val_loss: 0.4477 - val_accuracy: 0.8250\n",
      "Epoch 72/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3710 - accuracy: 0.9056 - val_loss: 0.4368 - val_accuracy: 0.9250\n",
      "Epoch 73/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3676 - accuracy: 0.9194 - val_loss: 0.4348 - val_accuracy: 0.7250\n",
      "Epoch 74/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3634 - accuracy: 0.9278 - val_loss: 0.4256 - val_accuracy: 0.9250\n",
      "Epoch 75/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3570 - accuracy: 0.9167 - val_loss: 0.4213 - val_accuracy: 0.9250\n",
      "Epoch 76/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3544 - accuracy: 0.9333 - val_loss: 0.4148 - val_accuracy: 0.9750\n",
      "Epoch 77/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3495 - accuracy: 0.9583 - val_loss: 0.4119 - val_accuracy: 0.9750\n",
      "Epoch 78/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3407 - accuracy: 0.9389 - val_loss: 0.4078 - val_accuracy: 0.9250\n",
      "Epoch 79/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3371 - accuracy: 0.9583 - val_loss: 0.4057 - val_accuracy: 0.8250\n",
      "Epoch 80/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3323 - accuracy: 0.9472 - val_loss: 0.3909 - val_accuracy: 0.9500\n",
      "Epoch 81/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3247 - accuracy: 0.9528 - val_loss: 0.3900 - val_accuracy: 0.9750\n",
      "Epoch 82/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3188 - accuracy: 0.9694 - val_loss: 0.3785 - val_accuracy: 0.9750\n",
      "Epoch 83/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3129 - accuracy: 0.9722 - val_loss: 0.3785 - val_accuracy: 0.9750\n",
      "Epoch 84/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.3047 - accuracy: 0.9694 - val_loss: 0.3634 - val_accuracy: 0.9750\n",
      "Epoch 85/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2992 - accuracy: 0.9639 - val_loss: 0.3569 - val_accuracy: 0.9750\n",
      "Epoch 86/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2913 - accuracy: 0.9667 - val_loss: 0.3672 - val_accuracy: 0.8750\n",
      "Epoch 87/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2818 - accuracy: 0.9778 - val_loss: 0.3391 - val_accuracy: 0.9750\n",
      "Epoch 88/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2743 - accuracy: 0.9806 - val_loss: 0.3332 - val_accuracy: 0.9750\n",
      "Epoch 89/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2660 - accuracy: 0.9806 - val_loss: 0.3178 - val_accuracy: 0.9750\n",
      "Epoch 90/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2579 - accuracy: 0.9750 - val_loss: 0.3123 - val_accuracy: 0.9750\n",
      "Epoch 91/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2505 - accuracy: 0.9833 - val_loss: 0.3069 - val_accuracy: 0.9750\n",
      "Epoch 92/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2412 - accuracy: 0.9750 - val_loss: 0.3018 - val_accuracy: 0.9750\n",
      "Epoch 93/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2328 - accuracy: 0.9889 - val_loss: 0.2916 - val_accuracy: 0.9750\n",
      "Epoch 94/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2279 - accuracy: 0.9861 - val_loss: 0.2849 - val_accuracy: 0.9750\n",
      "Epoch 95/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2187 - accuracy: 0.9889 - val_loss: 0.2772 - val_accuracy: 0.9750\n",
      "Epoch 96/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2132 - accuracy: 0.9833 - val_loss: 0.2664 - val_accuracy: 0.9750\n",
      "Epoch 97/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.2025 - accuracy: 0.9861 - val_loss: 0.2584 - val_accuracy: 0.9750\n",
      "Epoch 98/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1970 - accuracy: 0.9889 - val_loss: 0.2445 - val_accuracy: 0.9750\n",
      "Epoch 99/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1904 - accuracy: 0.9889 - val_loss: 0.2461 - val_accuracy: 0.9750\n",
      "Epoch 100/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1833 - accuracy: 0.9917 - val_loss: 0.2367 - val_accuracy: 0.9500\n",
      "13/13 [==============================] - 0s 1ms/step - loss: 0.1766 - accuracy: 0.9825\n",
      "Scores: [0.17662428319454193, 0.9825000166893005]\n"
     ]
    }
   ],
   "source": [
    "print('''\n",
    "########################################################################\n",
    "# Experiment 3: can we classify a SORTED sample of data?\n",
    "#\n",
    "#\n",
    "#########################################################################\n",
    "''')\n",
    "\n",
    "\n",
    "print(\"Sorting the data\")\n",
    "wdata.sort(axis=1)\n",
    "test_wdata.sort(axis=1)\n",
    "\n",
    "\n",
    "swcnet = Sequential()\n",
    "swcnet.add(Dense(width,input_shape=(width,),activation=\"sigmoid\"))\n",
    "swcnet.add(Dense(int(width/4),activation=\"sigmoid\"))\n",
    "swcnet.add(Dense(4,activation=\"softmax\"))\n",
    "swcnet.compile(loss=\"categorical_crossentropy\", optimizer=SGD(lr=0.1), metrics=[\"accuracy\"])\n",
    "history = swcnet.fit(wtrain[0], wtrain_y, validation_data=(wvalid[0], wvalid_y),\n",
    "\t            epochs=100, batch_size=16)\n",
    "score = swcnet.evaluate(test_wdata, wtest_y)\n",
    "print(\"Scores: %s\" % score)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('tp', 97), ('tn', 96), ('fp', 4), ('fn', 3)]\n",
      "13/13 [==============================] - 0s 1ms/step - loss: 0.1766 - accuracy: 0.9825\n",
      "Scores: [0.17662428319454193, 0.9825000166893005]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "classify = predict_classes(swcnet,test_wdata)\n",
    "print(theautil.classifications(classify,test_wlabels))\n",
    "score = swcnet.evaluate(test_wdata, wtest_y)\n",
    "print(\"Scores: %s\" % score)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was an improvement. What if we do binning instead?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1752 - accuracy: 0.9861 - val_loss: 0.2354 - val_accuracy: 0.9500\n",
      "Epoch 2/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1695 - accuracy: 0.9917 - val_loss: 0.2266 - val_accuracy: 0.9500\n",
      "Epoch 3/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1632 - accuracy: 0.9889 - val_loss: 0.2327 - val_accuracy: 0.9500\n",
      "Epoch 4/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1593 - accuracy: 0.9861 - val_loss: 0.2274 - val_accuracy: 0.9500\n",
      "Epoch 5/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1539 - accuracy: 0.9917 - val_loss: 0.2112 - val_accuracy: 0.9500\n",
      "Epoch 6/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1479 - accuracy: 0.9917 - val_loss: 0.2052 - val_accuracy: 0.9500\n",
      "Epoch 7/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1438 - accuracy: 0.9889 - val_loss: 0.2116 - val_accuracy: 0.9500\n",
      "Epoch 8/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1390 - accuracy: 0.9917 - val_loss: 0.2035 - val_accuracy: 0.9500\n",
      "Epoch 9/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1346 - accuracy: 0.9917 - val_loss: 0.1971 - val_accuracy: 0.9500\n",
      "Epoch 10/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1304 - accuracy: 0.9917 - val_loss: 0.1938 - val_accuracy: 0.9500\n",
      "Epoch 11/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1273 - accuracy: 0.9917 - val_loss: 0.1901 - val_accuracy: 0.9500\n",
      "Epoch 12/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1224 - accuracy: 0.9917 - val_loss: 0.1879 - val_accuracy: 0.9500\n",
      "Epoch 13/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1183 - accuracy: 0.9917 - val_loss: 0.1804 - val_accuracy: 0.9500\n",
      "Epoch 14/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1153 - accuracy: 0.9917 - val_loss: 0.1799 - val_accuracy: 0.9500\n",
      "Epoch 15/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1115 - accuracy: 0.9917 - val_loss: 0.1735 - val_accuracy: 0.9500\n",
      "Epoch 16/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1090 - accuracy: 0.9917 - val_loss: 0.1744 - val_accuracy: 0.9500\n",
      "Epoch 17/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1056 - accuracy: 0.9917 - val_loss: 0.1691 - val_accuracy: 0.9500\n",
      "Epoch 18/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1030 - accuracy: 0.9917 - val_loss: 0.1661 - val_accuracy: 0.9500\n",
      "Epoch 19/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.1000 - accuracy: 0.9917 - val_loss: 0.1670 - val_accuracy: 0.9500\n",
      "Epoch 20/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0968 - accuracy: 0.9917 - val_loss: 0.1720 - val_accuracy: 0.9500\n",
      "Epoch 21/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0949 - accuracy: 0.9917 - val_loss: 0.1688 - val_accuracy: 0.9500\n",
      "Epoch 22/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0928 - accuracy: 0.9889 - val_loss: 0.1583 - val_accuracy: 0.9500\n",
      "Epoch 23/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0897 - accuracy: 0.9917 - val_loss: 0.1569 - val_accuracy: 0.9500\n",
      "Epoch 24/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0879 - accuracy: 0.9917 - val_loss: 0.1548 - val_accuracy: 0.9500\n",
      "Epoch 25/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0855 - accuracy: 0.9917 - val_loss: 0.1549 - val_accuracy: 0.9500\n",
      "Epoch 26/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0838 - accuracy: 0.9917 - val_loss: 0.1593 - val_accuracy: 0.9500\n",
      "Epoch 27/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0817 - accuracy: 0.9889 - val_loss: 0.1553 - val_accuracy: 0.9500\n",
      "Epoch 28/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0789 - accuracy: 0.9917 - val_loss: 0.1642 - val_accuracy: 0.9500\n",
      "Epoch 29/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0777 - accuracy: 0.9917 - val_loss: 0.1498 - val_accuracy: 0.9750\n",
      "Epoch 30/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0760 - accuracy: 0.9917 - val_loss: 0.1487 - val_accuracy: 0.9750\n",
      "Epoch 31/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0744 - accuracy: 0.9917 - val_loss: 0.1526 - val_accuracy: 0.9500\n",
      "Epoch 32/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0728 - accuracy: 0.9917 - val_loss: 0.1464 - val_accuracy: 0.9500\n",
      "Epoch 33/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0715 - accuracy: 0.9917 - val_loss: 0.1448 - val_accuracy: 0.9750\n",
      "Epoch 34/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0695 - accuracy: 0.9917 - val_loss: 0.1538 - val_accuracy: 0.9750\n",
      "Epoch 35/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0688 - accuracy: 0.9917 - val_loss: 0.1517 - val_accuracy: 0.9500\n",
      "Epoch 36/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0662 - accuracy: 0.9917 - val_loss: 0.1468 - val_accuracy: 0.9750\n",
      "Epoch 37/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0655 - accuracy: 0.9889 - val_loss: 0.1405 - val_accuracy: 0.9750\n",
      "Epoch 38/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0642 - accuracy: 0.9917 - val_loss: 0.1418 - val_accuracy: 0.9750\n",
      "Epoch 39/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0627 - accuracy: 0.9917 - val_loss: 0.1430 - val_accuracy: 0.9750\n",
      "Epoch 40/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0618 - accuracy: 0.9889 - val_loss: 0.1436 - val_accuracy: 0.9750\n",
      "Epoch 41/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0610 - accuracy: 0.9917 - val_loss: 0.1530 - val_accuracy: 0.9750\n",
      "Epoch 42/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0599 - accuracy: 0.9944 - val_loss: 0.1433 - val_accuracy: 0.9750\n",
      "Epoch 43/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0585 - accuracy: 0.9917 - val_loss: 0.1422 - val_accuracy: 0.9750\n",
      "Epoch 44/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0573 - accuracy: 0.9944 - val_loss: 0.1395 - val_accuracy: 0.9750\n",
      "Epoch 45/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0563 - accuracy: 0.9917 - val_loss: 0.1492 - val_accuracy: 0.9750\n",
      "Epoch 46/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0554 - accuracy: 0.9944 - val_loss: 0.1341 - val_accuracy: 0.9750\n",
      "Epoch 47/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0546 - accuracy: 0.9917 - val_loss: 0.1371 - val_accuracy: 0.9750\n",
      "Epoch 48/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0536 - accuracy: 0.9944 - val_loss: 0.1336 - val_accuracy: 0.9750\n",
      "Epoch 49/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0532 - accuracy: 0.9917 - val_loss: 0.1333 - val_accuracy: 0.9750\n",
      "Epoch 50/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0517 - accuracy: 0.9889 - val_loss: 0.1370 - val_accuracy: 0.9750\n",
      "Epoch 51/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0509 - accuracy: 0.9944 - val_loss: 0.1311 - val_accuracy: 0.9750\n",
      "Epoch 52/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0503 - accuracy: 0.9889 - val_loss: 0.1349 - val_accuracy: 0.9750\n",
      "Epoch 53/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0495 - accuracy: 0.9944 - val_loss: 0.1311 - val_accuracy: 0.9750\n",
      "Epoch 54/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0490 - accuracy: 0.9889 - val_loss: 0.1295 - val_accuracy: 0.9750\n",
      "Epoch 55/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0484 - accuracy: 0.9917 - val_loss: 0.1359 - val_accuracy: 0.9750\n",
      "Epoch 56/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0477 - accuracy: 0.9917 - val_loss: 0.1371 - val_accuracy: 0.9750\n",
      "Epoch 57/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0464 - accuracy: 0.9944 - val_loss: 0.1352 - val_accuracy: 0.9750\n",
      "Epoch 58/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0459 - accuracy: 0.9917 - val_loss: 0.1320 - val_accuracy: 0.9750\n",
      "Epoch 59/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0455 - accuracy: 0.9889 - val_loss: 0.1281 - val_accuracy: 0.9750\n",
      "Epoch 60/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0453 - accuracy: 0.9917 - val_loss: 0.1291 - val_accuracy: 0.9750\n",
      "Epoch 61/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0446 - accuracy: 0.9917 - val_loss: 0.1311 - val_accuracy: 0.9750\n",
      "Epoch 62/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0437 - accuracy: 0.9917 - val_loss: 0.1306 - val_accuracy: 0.9750\n",
      "Epoch 63/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0442 - accuracy: 0.9917 - val_loss: 0.1317 - val_accuracy: 0.9750\n",
      "Epoch 64/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0425 - accuracy: 0.9917 - val_loss: 0.1341 - val_accuracy: 0.9750\n",
      "Epoch 65/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0423 - accuracy: 0.9917 - val_loss: 0.1346 - val_accuracy: 0.9750\n",
      "Epoch 66/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0416 - accuracy: 0.9917 - val_loss: 0.1319 - val_accuracy: 0.9750\n",
      "Epoch 67/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0412 - accuracy: 0.9889 - val_loss: 0.1346 - val_accuracy: 0.9750\n",
      "Epoch 68/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0407 - accuracy: 0.9917 - val_loss: 0.1343 - val_accuracy: 0.9750\n",
      "Epoch 69/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0404 - accuracy: 0.9917 - val_loss: 0.1389 - val_accuracy: 0.9750\n",
      "Epoch 70/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0398 - accuracy: 0.9917 - val_loss: 0.1358 - val_accuracy: 0.9750\n",
      "Epoch 71/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0398 - accuracy: 0.9917 - val_loss: 0.1384 - val_accuracy: 0.9750\n",
      "Epoch 72/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0391 - accuracy: 0.9944 - val_loss: 0.1340 - val_accuracy: 0.9750\n",
      "Epoch 73/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0390 - accuracy: 0.9917 - val_loss: 0.1396 - val_accuracy: 0.9750\n",
      "Epoch 74/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0379 - accuracy: 0.9917 - val_loss: 0.1410 - val_accuracy: 0.9750\n",
      "Epoch 75/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0377 - accuracy: 0.9917 - val_loss: 0.1346 - val_accuracy: 0.9750\n",
      "Epoch 76/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0370 - accuracy: 0.9944 - val_loss: 0.1318 - val_accuracy: 0.9750\n",
      "Epoch 77/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0368 - accuracy: 0.9917 - val_loss: 0.1324 - val_accuracy: 0.9750\n",
      "Epoch 78/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0366 - accuracy: 0.9917 - val_loss: 0.1333 - val_accuracy: 0.9750\n",
      "Epoch 79/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0359 - accuracy: 0.9917 - val_loss: 0.1375 - val_accuracy: 0.9750\n",
      "Epoch 80/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0355 - accuracy: 0.9944 - val_loss: 0.1407 - val_accuracy: 0.9750\n",
      "Epoch 81/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0357 - accuracy: 0.9944 - val_loss: 0.1339 - val_accuracy: 0.9750\n",
      "Epoch 82/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0352 - accuracy: 0.9917 - val_loss: 0.1346 - val_accuracy: 0.9750\n",
      "Epoch 83/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0349 - accuracy: 0.9917 - val_loss: 0.1341 - val_accuracy: 0.9750\n",
      "Epoch 84/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0345 - accuracy: 0.9917 - val_loss: 0.1355 - val_accuracy: 0.9750\n",
      "Epoch 85/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0342 - accuracy: 0.9917 - val_loss: 0.1318 - val_accuracy: 0.9750\n",
      "Epoch 86/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0341 - accuracy: 0.9917 - val_loss: 0.1354 - val_accuracy: 0.9750\n",
      "Epoch 87/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0343 - accuracy: 0.9917 - val_loss: 0.1388 - val_accuracy: 0.9750\n",
      "Epoch 88/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0327 - accuracy: 0.9944 - val_loss: 0.1344 - val_accuracy: 0.9750\n",
      "Epoch 89/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0337 - accuracy: 0.9917 - val_loss: 0.1345 - val_accuracy: 0.9750\n",
      "Epoch 90/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0327 - accuracy: 0.9944 - val_loss: 0.1258 - val_accuracy: 0.9750\n",
      "Epoch 91/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0325 - accuracy: 0.9944 - val_loss: 0.1381 - val_accuracy: 0.9750\n",
      "Epoch 92/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0326 - accuracy: 0.9917 - val_loss: 0.1345 - val_accuracy: 0.9750\n",
      "Epoch 93/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0316 - accuracy: 0.9917 - val_loss: 0.1470 - val_accuracy: 0.9750\n",
      "Epoch 94/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0317 - accuracy: 0.9944 - val_loss: 0.1379 - val_accuracy: 0.9750\n",
      "Epoch 95/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0310 - accuracy: 0.9944 - val_loss: 0.1346 - val_accuracy: 0.9750\n",
      "Epoch 96/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0312 - accuracy: 0.9944 - val_loss: 0.1310 - val_accuracy: 0.9750\n",
      "Epoch 97/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0308 - accuracy: 0.9944 - val_loss: 0.1377 - val_accuracy: 0.9750\n",
      "Epoch 98/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0307 - accuracy: 0.9944 - val_loss: 0.1354 - val_accuracy: 0.9750\n",
      "Epoch 99/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0305 - accuracy: 0.9944 - val_loss: 0.1341 - val_accuracy: 0.9750\n",
      "Epoch 100/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.0306 - accuracy: 0.9917 - val_loss: 0.1339 - val_accuracy: 0.9750\n",
      "13/13 [==============================] - 0s 935us/step - loss: 0.0654 - accuracy: 0.9825\n",
      "Scores: [0.0653819590806961, 0.9825000166893005]\n"
     ]
    }
   ],
   "source": [
    "history = swcnet.fit(wtrain[0], wtrain_y, validation_data=(wvalid[0], wvalid_y),\n",
    "\t            epochs=100, batch_size=16)\n",
    "score = swcnet.evaluate(test_wdata, wtest_y)\n",
    "print(\"Scores: %s\" % score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('tp', 98), ('tn', 95), ('fp', 5), ('fn', 2)]\n",
      "13/13 [==============================] - 0s 927us/step - loss: 0.0654 - accuracy: 0.9825\n",
      "Scores: [0.0653819590806961, 0.9825000166893005]\n"
     ]
    }
   ],
   "source": [
    "classify = predict_classes(swcnet, test_wdata)\n",
    "print(theautil.classifications(classify,test_wlabels))\n",
    "score = swcnet.evaluate(test_wdata, wtest_y)\n",
    "print(\"Scores: %s\" % score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 4\n",
    "\n",
    "* Given 40 histogrammed samples what distribution does it come from?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "########################################################################\n",
      "# Experiment 4: can we classify a discretized histogram of sample data?\n",
      "#\n",
      "#\n",
      "#########################################################################\n",
      "\n",
      "Apply the histogram to all the data rows\n",
      "Epoch 1/100\n",
      " 1/23 [>.............................] - ETA: 3s - loss: 1.6236 - accuracy: 0.3125"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/hindle1/.local/lib/python3.8/site-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(SGD, self).__init__(name, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23/23 [==============================] - 0s 5ms/step - loss: 1.4524 - accuracy: 0.2417 - val_loss: 1.3854 - val_accuracy: 0.2000\n",
      "Epoch 2/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3951 - accuracy: 0.2222 - val_loss: 1.3839 - val_accuracy: 0.2750\n",
      "Epoch 3/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3952 - accuracy: 0.2306 - val_loss: 1.3872 - val_accuracy: 0.2000\n",
      "Epoch 4/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3946 - accuracy: 0.2250 - val_loss: 1.4017 - val_accuracy: 0.2000\n",
      "Epoch 5/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3916 - accuracy: 0.2528 - val_loss: 1.3872 - val_accuracy: 0.2750\n",
      "Epoch 6/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3951 - accuracy: 0.2361 - val_loss: 1.3836 - val_accuracy: 0.2750\n",
      "Epoch 7/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3922 - accuracy: 0.2250 - val_loss: 1.4020 - val_accuracy: 0.2000\n",
      "Epoch 8/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3917 - accuracy: 0.2306 - val_loss: 1.3997 - val_accuracy: 0.2000\n",
      "Epoch 9/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3869 - accuracy: 0.2722 - val_loss: 1.4103 - val_accuracy: 0.2000\n",
      "Epoch 10/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3951 - accuracy: 0.2278 - val_loss: 1.4012 - val_accuracy: 0.2000\n",
      "Epoch 11/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3859 - accuracy: 0.2639 - val_loss: 1.3933 - val_accuracy: 0.2000\n",
      "Epoch 12/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3859 - accuracy: 0.2917 - val_loss: 1.3705 - val_accuracy: 0.3250\n",
      "Epoch 13/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3923 - accuracy: 0.2222 - val_loss: 1.3738 - val_accuracy: 0.2750\n",
      "Epoch 14/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3878 - accuracy: 0.2556 - val_loss: 1.3809 - val_accuracy: 0.5250\n",
      "Epoch 15/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3882 - accuracy: 0.2833 - val_loss: 1.3874 - val_accuracy: 0.2000\n",
      "Epoch 16/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3856 - accuracy: 0.2361 - val_loss: 1.3838 - val_accuracy: 0.2000\n",
      "Epoch 17/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3829 - accuracy: 0.2722 - val_loss: 1.3932 - val_accuracy: 0.2000\n",
      "Epoch 18/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3818 - accuracy: 0.2889 - val_loss: 1.3770 - val_accuracy: 0.4500\n",
      "Epoch 19/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3831 - accuracy: 0.2694 - val_loss: 1.3765 - val_accuracy: 0.3500\n",
      "Epoch 20/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3822 - accuracy: 0.3111 - val_loss: 1.3756 - val_accuracy: 0.2750\n",
      "Epoch 21/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3808 - accuracy: 0.2500 - val_loss: 1.3784 - val_accuracy: 0.4750\n",
      "Epoch 22/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3780 - accuracy: 0.3333 - val_loss: 1.3723 - val_accuracy: 0.3750\n",
      "Epoch 23/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3780 - accuracy: 0.3056 - val_loss: 1.3628 - val_accuracy: 0.3250\n",
      "Epoch 24/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3777 - accuracy: 0.2556 - val_loss: 1.3765 - val_accuracy: 0.2000\n",
      "Epoch 25/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3747 - accuracy: 0.2806 - val_loss: 1.3708 - val_accuracy: 0.4750\n",
      "Epoch 26/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3686 - accuracy: 0.4083 - val_loss: 1.3581 - val_accuracy: 0.3250\n",
      "Epoch 27/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3708 - accuracy: 0.3028 - val_loss: 1.3696 - val_accuracy: 0.2000\n",
      "Epoch 28/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3635 - accuracy: 0.3333 - val_loss: 1.3622 - val_accuracy: 0.3250\n",
      "Epoch 29/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3602 - accuracy: 0.3139 - val_loss: 1.3705 - val_accuracy: 0.2000\n",
      "Epoch 30/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3579 - accuracy: 0.3528 - val_loss: 1.3633 - val_accuracy: 0.4750\n",
      "Epoch 31/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3549 - accuracy: 0.4056 - val_loss: 1.3515 - val_accuracy: 0.4750\n",
      "Epoch 32/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3468 - accuracy: 0.4167 - val_loss: 1.3458 - val_accuracy: 0.4750\n",
      "Epoch 33/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3454 - accuracy: 0.4389 - val_loss: 1.3322 - val_accuracy: 0.4750\n",
      "Epoch 34/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3372 - accuracy: 0.4806 - val_loss: 1.3289 - val_accuracy: 0.5250\n",
      "Epoch 35/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3312 - accuracy: 0.4528 - val_loss: 1.3430 - val_accuracy: 0.2000\n",
      "Epoch 36/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3251 - accuracy: 0.4111 - val_loss: 1.3285 - val_accuracy: 0.2250\n",
      "Epoch 37/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3108 - accuracy: 0.4111 - val_loss: 1.3116 - val_accuracy: 0.4750\n",
      "Epoch 38/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.3019 - accuracy: 0.4583 - val_loss: 1.2989 - val_accuracy: 0.4750\n",
      "Epoch 39/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2912 - accuracy: 0.4583 - val_loss: 1.2766 - val_accuracy: 0.5250\n",
      "Epoch 40/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2750 - accuracy: 0.4944 - val_loss: 1.2716 - val_accuracy: 0.4750\n",
      "Epoch 41/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2582 - accuracy: 0.5111 - val_loss: 1.2449 - val_accuracy: 0.5250\n",
      "Epoch 42/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2386 - accuracy: 0.4778 - val_loss: 1.2280 - val_accuracy: 0.5250\n",
      "Epoch 43/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.2162 - accuracy: 0.5278 - val_loss: 1.2146 - val_accuracy: 0.4750\n",
      "Epoch 44/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1857 - accuracy: 0.5167 - val_loss: 1.1744 - val_accuracy: 0.5250\n",
      "Epoch 45/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1618 - accuracy: 0.4972 - val_loss: 1.1314 - val_accuracy: 0.5250\n",
      "Epoch 46/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.1308 - accuracy: 0.5139 - val_loss: 1.1171 - val_accuracy: 0.4750\n",
      "Epoch 47/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0972 - accuracy: 0.4917 - val_loss: 1.0798 - val_accuracy: 0.5250\n",
      "Epoch 48/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0603 - accuracy: 0.5083 - val_loss: 1.0491 - val_accuracy: 0.4750\n",
      "Epoch 49/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 1.0267 - accuracy: 0.5028 - val_loss: 1.0058 - val_accuracy: 0.4750\n",
      "Epoch 50/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9892 - accuracy: 0.4917 - val_loss: 0.9929 - val_accuracy: 0.5500\n",
      "Epoch 51/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9603 - accuracy: 0.5167 - val_loss: 0.9488 - val_accuracy: 0.4750\n",
      "Epoch 52/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9308 - accuracy: 0.4944 - val_loss: 0.9155 - val_accuracy: 0.4750\n",
      "Epoch 53/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.9017 - accuracy: 0.5139 - val_loss: 0.8924 - val_accuracy: 0.4750\n",
      "Epoch 54/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8784 - accuracy: 0.5000 - val_loss: 0.8729 - val_accuracy: 0.4750\n",
      "Epoch 55/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8581 - accuracy: 0.5056 - val_loss: 0.8459 - val_accuracy: 0.6000\n",
      "Epoch 56/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8384 - accuracy: 0.5250 - val_loss: 0.8403 - val_accuracy: 0.4750\n",
      "Epoch 57/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8206 - accuracy: 0.5250 - val_loss: 0.8131 - val_accuracy: 0.5250\n",
      "Epoch 58/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.8074 - accuracy: 0.5306 - val_loss: 0.8025 - val_accuracy: 0.4750\n",
      "Epoch 59/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7966 - accuracy: 0.5028 - val_loss: 0.7908 - val_accuracy: 0.4750\n",
      "Epoch 60/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7869 - accuracy: 0.4972 - val_loss: 0.7810 - val_accuracy: 0.4750\n",
      "Epoch 61/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7794 - accuracy: 0.4750 - val_loss: 0.7752 - val_accuracy: 0.4750\n",
      "Epoch 62/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7705 - accuracy: 0.5056 - val_loss: 0.7685 - val_accuracy: 0.4750\n",
      "Epoch 63/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7610 - accuracy: 0.4889 - val_loss: 0.7693 - val_accuracy: 0.4750\n",
      "Epoch 64/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7575 - accuracy: 0.5194 - val_loss: 0.7602 - val_accuracy: 0.4750\n",
      "Epoch 65/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7529 - accuracy: 0.5139 - val_loss: 0.7566 - val_accuracy: 0.4750\n",
      "Epoch 66/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7486 - accuracy: 0.4889 - val_loss: 0.7436 - val_accuracy: 0.5250\n",
      "Epoch 67/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7439 - accuracy: 0.5306 - val_loss: 0.7474 - val_accuracy: 0.4750\n",
      "Epoch 68/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7387 - accuracy: 0.5167 - val_loss: 0.7425 - val_accuracy: 0.4750\n",
      "Epoch 69/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7333 - accuracy: 0.5361 - val_loss: 0.7325 - val_accuracy: 0.7250\n",
      "Epoch 70/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7338 - accuracy: 0.5000 - val_loss: 0.7318 - val_accuracy: 0.4750\n",
      "Epoch 71/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7314 - accuracy: 0.4917 - val_loss: 0.7253 - val_accuracy: 0.6500\n",
      "Epoch 72/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7265 - accuracy: 0.5389 - val_loss: 0.7225 - val_accuracy: 0.5250\n",
      "Epoch 73/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7257 - accuracy: 0.5222 - val_loss: 0.7322 - val_accuracy: 0.4750\n",
      "Epoch 74/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7254 - accuracy: 0.5528 - val_loss: 0.7304 - val_accuracy: 0.4750\n",
      "Epoch 75/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7240 - accuracy: 0.4833 - val_loss: 0.7153 - val_accuracy: 0.5250\n",
      "Epoch 76/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7197 - accuracy: 0.5194 - val_loss: 0.7136 - val_accuracy: 0.6000\n",
      "Epoch 77/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7189 - accuracy: 0.5278 - val_loss: 0.7141 - val_accuracy: 0.5750\n",
      "Epoch 78/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7133 - accuracy: 0.5750 - val_loss: 0.7170 - val_accuracy: 0.4750\n",
      "Epoch 79/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7141 - accuracy: 0.5417 - val_loss: 0.7093 - val_accuracy: 0.5250\n",
      "Epoch 80/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7127 - accuracy: 0.5583 - val_loss: 0.7128 - val_accuracy: 0.5250\n",
      "Epoch 81/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7139 - accuracy: 0.5500 - val_loss: 0.7114 - val_accuracy: 0.4750\n",
      "Epoch 82/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7106 - accuracy: 0.5111 - val_loss: 0.7104 - val_accuracy: 0.4750\n",
      "Epoch 83/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7085 - accuracy: 0.5389 - val_loss: 0.7071 - val_accuracy: 0.5500\n",
      "Epoch 84/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7077 - accuracy: 0.5667 - val_loss: 0.7025 - val_accuracy: 0.5250\n",
      "Epoch 85/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7069 - accuracy: 0.5778 - val_loss: 0.7024 - val_accuracy: 0.6250\n",
      "Epoch 86/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7058 - accuracy: 0.5778 - val_loss: 0.7012 - val_accuracy: 0.6000\n",
      "Epoch 87/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7039 - accuracy: 0.6139 - val_loss: 0.7082 - val_accuracy: 0.4750\n",
      "Epoch 88/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7014 - accuracy: 0.5750 - val_loss: 0.6970 - val_accuracy: 0.5500\n",
      "Epoch 89/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7022 - accuracy: 0.6278 - val_loss: 0.6985 - val_accuracy: 0.5750\n",
      "Epoch 90/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6990 - accuracy: 0.5361 - val_loss: 0.6973 - val_accuracy: 0.6750\n",
      "Epoch 91/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7033 - accuracy: 0.5917 - val_loss: 0.6947 - val_accuracy: 0.7750\n",
      "Epoch 92/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6994 - accuracy: 0.6250 - val_loss: 0.6923 - val_accuracy: 0.5250\n",
      "Epoch 93/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.7022 - accuracy: 0.5667 - val_loss: 0.6919 - val_accuracy: 0.7750\n",
      "Epoch 94/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6987 - accuracy: 0.5833 - val_loss: 0.6961 - val_accuracy: 0.5500\n",
      "Epoch 95/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6979 - accuracy: 0.5861 - val_loss: 0.6926 - val_accuracy: 0.7250\n",
      "Epoch 96/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6964 - accuracy: 0.5778 - val_loss: 0.6902 - val_accuracy: 0.7250\n",
      "Epoch 97/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6941 - accuracy: 0.5861 - val_loss: 0.6947 - val_accuracy: 0.4750\n",
      "Epoch 98/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6983 - accuracy: 0.5667 - val_loss: 0.6882 - val_accuracy: 0.7250\n",
      "Epoch 99/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6952 - accuracy: 0.6222 - val_loss: 0.6866 - val_accuracy: 0.7250\n",
      "Epoch 100/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6912 - accuracy: 0.6111 - val_loss: 0.6911 - val_accuracy: 0.5750\n",
      "13/13 [==============================] - 0s 939us/step - loss: 0.6953 - accuracy: 0.5500\n",
      "Scores: [0.6952625513076782, 0.550000011920929]\n"
     ]
    }
   ],
   "source": [
    "print('''\n",
    "########################################################################\n",
    "# Experiment 4: can we classify a discretized histogram of sample data?\n",
    "#\n",
    "#\n",
    "#########################################################################\n",
    "'''\n",
    ")\n",
    "# let's try actual binning\n",
    "import collections\n",
    "\n",
    "def bin(row):\n",
    "    return np.histogram(row,bins=len(row),range=(0.0,1.0))[0]/float(len(row))\n",
    "\n",
    "print(\"Apply the histogram to all the data rows\")\n",
    "bdata = np.apply_along_axis(bin,1,wdata).astype(np.float32)\n",
    "blabels = wlabels\n",
    "\n",
    "# ensure we have our test data\n",
    "test_bdata = np.apply_along_axis(bin,1,test_wdata).astype(np.float32)\n",
    "test_blabels = test_wlabels\n",
    "\n",
    "# helper data \n",
    "enum_funcs = [\n",
    "    (LOGNORMAL,\"log normal\",lambda size: lognormal(size=size)),\n",
    "    (POWER,\"power\",lambda size: power(0.1,size=size)),\n",
    "    (NORM,\"normal\",lambda size: normal(size=size)),\n",
    "    (UNIFORM,\"uniforms\",lambda size: uniform(size=size)),\n",
    "]\n",
    "\n",
    "# uses enum_funcs to evaluate PER CLASS how well our classify operates\n",
    "def classify_test(bnet,ntests=1000):\n",
    "    for tup in enum_funcs:\n",
    "        enum, name, func = tup\n",
    "        lns = min_max_scale(func(size=(ntests,width))) #log normal\n",
    "        blns = np.apply_along_axis(bin,1,lns).astype(np.float32)\n",
    "        blns_labels = np.repeat(enum,ntests)\n",
    "        blns_labels.astype(np.int32)\n",
    "        classification = predict_classes(bnet,blns)\n",
    "        classified = theautil.classifications(classification,blns_labels)\n",
    "        print(\"Name:%s Tests:[%s] Count:%s -- Res:%s\" % (name,ntests, collections.Counter(classification),classified ))\n",
    "\n",
    "# train & valid\n",
    "btrain, bvalid = split_validation(90, bdata, blabels)\n",
    "\n",
    "encb = OneHotEncoder(handle_unknown='ignore', sparse=False)\n",
    "encb.fit(btrain[1].reshape(len(btrain[1]),1))\n",
    "btrain_y = encb.transform(btrain[1].reshape(len(btrain[1]),1))\n",
    "bvalid_y = encb.transform(bvalid[1].reshape(len(bvalid[1]),1))\n",
    "btest_y  = encb.transform(test_blabels.reshape(len(test_blabels),1))\n",
    "\n",
    "\n",
    "\n",
    "# similar network structure\n",
    "# bnet = theanets.Classifier([width,width/2,4])\n",
    "\n",
    "bnet = Sequential()\n",
    "bnet.add(Dense(width,input_shape=(width,),activation=\"sigmoid\"))\n",
    "bnet.add(Dense(int(width/4),activation=\"sigmoid\"))\n",
    "bnet.add(Dense(4,activation=\"softmax\"))\n",
    "bnet.compile(loss=\"categorical_crossentropy\", optimizer=SGD(lr=0.1), metrics=[\"accuracy\"])\n",
    "history = bnet.fit(btrain[0], btrain_y, validation_data=(bvalid[0], bvalid_y),\n",
    "\t            epochs=100, batch_size=16)\n",
    "score = bnet.evaluate(test_bdata, btest_y)\n",
    "print(\"Scores: %s\" % score)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6939 - accuracy: 0.5972 - val_loss: 0.6836 - val_accuracy: 0.8500\n",
      "Epoch 2/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6925 - accuracy: 0.5889 - val_loss: 0.6839 - val_accuracy: 0.5500\n",
      "Epoch 3/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6931 - accuracy: 0.5889 - val_loss: 0.6806 - val_accuracy: 0.6000\n",
      "Epoch 4/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6860 - accuracy: 0.5861 - val_loss: 0.6893 - val_accuracy: 0.5000\n",
      "Epoch 5/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6859 - accuracy: 0.6083 - val_loss: 0.6796 - val_accuracy: 0.5250\n",
      "Epoch 6/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6898 - accuracy: 0.6194 - val_loss: 0.6820 - val_accuracy: 0.6500\n",
      "Epoch 7/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6865 - accuracy: 0.6333 - val_loss: 0.6768 - val_accuracy: 0.6250\n",
      "Epoch 8/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6864 - accuracy: 0.6056 - val_loss: 0.6865 - val_accuracy: 0.4750\n",
      "Epoch 9/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6849 - accuracy: 0.6056 - val_loss: 0.6839 - val_accuracy: 0.5000\n",
      "Epoch 10/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6819 - accuracy: 0.6194 - val_loss: 0.6763 - val_accuracy: 0.7500\n",
      "Epoch 11/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6832 - accuracy: 0.6000 - val_loss: 0.6755 - val_accuracy: 0.7500\n",
      "Epoch 12/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6847 - accuracy: 0.6167 - val_loss: 0.6712 - val_accuracy: 0.8000\n",
      "Epoch 13/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6805 - accuracy: 0.6556 - val_loss: 0.6692 - val_accuracy: 0.8000\n",
      "Epoch 14/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6782 - accuracy: 0.6694 - val_loss: 0.6768 - val_accuracy: 0.7750\n",
      "Epoch 15/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6804 - accuracy: 0.6667 - val_loss: 0.6751 - val_accuracy: 0.6750\n",
      "Epoch 16/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6764 - accuracy: 0.6944 - val_loss: 0.6735 - val_accuracy: 0.6500\n",
      "Epoch 17/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6759 - accuracy: 0.6583 - val_loss: 0.6660 - val_accuracy: 0.8000\n",
      "Epoch 18/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6764 - accuracy: 0.6750 - val_loss: 0.6760 - val_accuracy: 0.5750\n",
      "Epoch 19/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6742 - accuracy: 0.6278 - val_loss: 0.6646 - val_accuracy: 0.7250\n",
      "Epoch 20/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6714 - accuracy: 0.7056 - val_loss: 0.6731 - val_accuracy: 0.5750\n",
      "Epoch 21/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6722 - accuracy: 0.6778 - val_loss: 0.6609 - val_accuracy: 0.8000\n",
      "Epoch 22/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6714 - accuracy: 0.6472 - val_loss: 0.6594 - val_accuracy: 0.8000\n",
      "Epoch 23/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6663 - accuracy: 0.6528 - val_loss: 0.6608 - val_accuracy: 0.7250\n",
      "Epoch 24/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6683 - accuracy: 0.6833 - val_loss: 0.6623 - val_accuracy: 0.6000\n",
      "Epoch 25/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6665 - accuracy: 0.6389 - val_loss: 0.6549 - val_accuracy: 0.5250\n",
      "Epoch 26/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6697 - accuracy: 0.6583 - val_loss: 0.6564 - val_accuracy: 0.7250\n",
      "Epoch 27/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6657 - accuracy: 0.6778 - val_loss: 0.6562 - val_accuracy: 0.6500\n",
      "Epoch 28/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6646 - accuracy: 0.7139 - val_loss: 0.6469 - val_accuracy: 0.8250\n",
      "Epoch 29/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6662 - accuracy: 0.6972 - val_loss: 0.6459 - val_accuracy: 0.8500\n",
      "Epoch 30/100\n",
      "23/23 [==============================] - 0s 2ms/step - loss: 0.6578 - accuracy: 0.7222 - val_loss: 0.6434 - val_accuracy: 0.8500\n",
      "Epoch 31/100\n",
      " 1/23 [>.............................] - ETA: 0s - loss: 0.6539 - accuracy: 0.8750"
     ]
    }
   ],
   "source": [
    "history = bnet.fit(btrain[0], btrain_y, validation_data=(bvalid[0], bvalid_y),\n",
    "\t            epochs=100, batch_size=16)\n",
    "score = bnet.evaluate(test_bdata, btest_y)\n",
    "print(\"Scores: %s\" % score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "classify = predict_classes(bnet,test_bdata)\n",
    "print(theautil.classifications(classify,test_blabels))\n",
    "score = bnet.evaluate(test_bdata, btest_y)\n",
    "print(\"Scores: %s\" % score)\n",
    "\n",
    "classify_test(bnet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representation: Inputs\n",
    "\n",
    "* For discrete values consider discrete inputs neurons. E.g. if you have 3 letters are your input you should have 3 * 26 input neurons. \n",
    "* Each neuron is \"one-hot\" -- 1 neuron is set to 1 to indicate that 1 discerete value. \n",
    "* An input of AAA would be: \n",
    "  * 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
    "* ZZZ would be \n",
    "  * 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representation: Inputs\n",
    "\n",
    "* For groups of elements consider representing them as their counts.\n",
    "* E.g. 3 cats, 4 dogs, 1 car as: 3 4 1 on 3 input neurons.\n",
    "* Neural networks work well with distributions as inputs and distributions as outputs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Representation: Words\n",
    "\n",
    "* Words can be represented as word counts where by your vector is the count of each word per document -- you might have a large vocabulary so watch out!\n",
    "* n-grams are popular too with one-hot encoding\n",
    "* Embeddings (a dense vector representation) are popular too. Autoencoded words!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Representaiton: Images\n",
    "\n",
    "* Each neuron can represent a pixel represented from 0 to 1\n",
    "* You can have images as output too!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Representation: Outputs\n",
    "\n",
    "* Do not ask the neural network to distingush discrete values on 1 neuron. Don't expect 1 neuron to output 0.25 for A and 0.9 for B and 1.0 for C. Use 3 neurons!\n",
    "* Distribution outputs are good\n",
    "* Interpretting the output is fine for regression problems\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning\n",
    "\n",
    "* The parameters you chose were probably not correct!\n",
    "* You could grid search, that is try all combinations. But that takes a lot of time.\n",
    "* We want to get engage in hyper-parameter tuning.\n",
    "  * We want to find good parameters for our network that perform well.\n",
    "* Grid Search\n",
    "  * Step 1: choose the parameter space\n",
    "  * Step 2: choose a method of selecting parameters\n",
    "  * Step 3: get next combination of parameters\n",
    "  * Step 4: evaluate the parameters\n",
    "  * Step 5: If current performance is better than prior performances keep this set of parameters\n",
    "  * Step 6: goto step 3 until all parameter combinations are exhausted.\n",
    "  * Step 7: report results\n",
    "* Random Search\n",
    "  * Step 1: choose the parameter space\n",
    "  * Step 2: choose a method of selecting parameters\n",
    "  * Step 3: randomly choose parameters\n",
    "  * Step 4: Evaluate the parameters\n",
    "  * Step 5: If current performance is better than prior performances keep this set of parameters\n",
    "  * Step 6: goto step 3 until satisfied (N iterations or M seconds)\n",
    "  * Step 7: report results\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's tune\n",
    "print('''\n",
    "########################################################################\n",
    "# Experiment 5: Can we tune the binned data?\n",
    "#\n",
    "#\n",
    "#########################################################################\n",
    "'''\n",
    ")\n",
    "\n",
    "\n",
    "import Search\n",
    "\n",
    "# 1 repetition\n",
    "state = {\"reps\":1}\n",
    "params = {\"batch_size\":[1,4,8,16,32,64],\n",
    "          \"lr\":[1.0,0.1,0.01,0.001,0.0001],\n",
    "          \"activation\":[\"sigmoid\",\"tanh\",\"relu\"],\n",
    "          \"optimizer\":[\"SGD\",\"Adam\"],\n",
    "          \"epochs\":[25],\n",
    "          \"arch\":[\n",
    "              [width],\n",
    "              [width,width],\n",
    "              [width,int(width/4)],\n",
    "              [2*width,width],\n",
    "              [int(width/4),int(width/8)],\n",
    "              [int(width/4)]]\n",
    "          }\n",
    "def get_optimizer(x):\n",
    "    if x == \"Adam\":\n",
    "        return Adam\n",
    "    return SGD\n",
    "    \n",
    "def f(state,params):\n",
    "    bnet = Sequential()\n",
    "    arch = params[\"arch\"]\n",
    "    bnet.add(Dense(arch[0],input_shape=(width,),activation=params[\"activation\"]))\n",
    "    for layer in arch[1:]:\n",
    "        bnet.add(Dense(int(layer),activation=params[\"activation\"]))\n",
    "    bnet.add(Dense(4,activation=\"softmax\"))\n",
    "    optimizer = get_optimizer(params[\"optimizer\"])\n",
    "    bnet.compile(loss=\"categorical_crossentropy\",\n",
    "                 optimizer=optimizer(lr=params[\"lr\"]), metrics=[\"accuracy\"])\n",
    "    history = bnet.fit(btrain[0], btrain_y,\n",
    "                       validation_data=(bvalid[0], bvalid_y),\n",
    "\t               epochs=params[\"epochs\"], batch_size=params[\"batch_size\"])\n",
    "    classify = predict_classes(bnet, test_bdata)\n",
    "    print(theautil.classifications(classify,test_blabels))\n",
    "    score = bnet.evaluate(test_bdata, btest_y)\n",
    "    print(\"Scores: %s\" % score)\n",
    "    return score[1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the heuristic function to f\n",
    "state[\"f\"] = f\n",
    "# random search for 60 seconds\n",
    "random_results = Search.random_search(state,params,Search.heuristic_function,time=60)\n",
    "# get the random results\n",
    "random_results   = sorted(random_results, key=lambda x: x['Score'])\n",
    "print(random_results[-1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "pprint(random_results[-10:]) # print last 10 results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## References\n",
    "\n",
    "* [Theanets Documentation](https://theanets.readthedocs.org/en/stable/)\n",
    "* [A Practical Guide to TrainingRestricted Boltzmann Machines](https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf)\n",
    "* [MLP](http://deeplearning.net/tutorial/mlp.html#mlp)\n",
    "* [Deep Learning Tutorials](http://www.iro.umontreal.ca/~pift6266/H10/notes/deepintro.html)\n",
    "* [Deep Learning Tutorials](http://deeplearning.net/tutorial/)\n",
    "* [Coursera: Hinton's Neural Networks for Machine Learning](https://www.coursera.org/course/neuralnets)\n",
    "* [The Next Generation of Neural Networks](https://www.youtube.com/watch?v=AyzOUbkUf3M)\n",
    "* [Geoffrey Hinton: \"Introduction to Deep Learning & Deep Belief Nets\"](https://www.youtube.com/watch?v=GJdWESd543Y)\n",
    "* Bengio's Deep Learning\n",
    "  [(1)](https://www.youtube.com/watch?v=JuimBuvEWBg)[(2)](https://www.youtube.com/watch?v=Fl-W7_z3w3o)\n",
    "* [Nvidia's Deep Learning tutorials](https://developer.nvidia.com/deep-learning-courses\n",
    ")\n",
    "* [Udacity Deep Learning MOOC](https://www.udacity.com/course/deep-learning--ud730)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}