lab6 finished

Author: mstaczek

lab6.ipynb

@@ -640,7 +640,7 @@
     "\n",
     "### Additional question regarding ReLU gradient implementation\n",
     "\n",
-    "ReLU function may have differences in gradient implementation - some may set gradient to 0 for negative values, some may set it as 0.1 for example (called Leaky ReLu).\n",
+    "ReLU function may have differences in gradient implementation - some may set gradient to 0 for negative values, some may set it as 0.01x for example (called Leaky ReLu).\n",
     "\n",
     "### Comparison of speed and accuracy of networks with different number of hidden layers and activation functions\n",
     "\n",
@@ -650,10 +650,17 @@
     "\n",
     "Linear activation function is useless as no matter how many layers and neurons, the output will always be a linear function.\n",
     "\n",
-    "It's also worth noticing that ReLU and tanh activation functions are faster than the sigmoid activation function. The cause of this is that ReLU and tanh activation functions are implemented in numpy, which turns out to be much faster than a vectorized python function implemented for sigmoid.\n",
+    "It's also worth noticing that ReLU and tanh activation functions are faster than the sigmoid activation function. The cause of this is that ReLU and tanh activation functions are implemented in numpy, which \n",
+    "turns out to be much faster than a vectorized python function implemented for sigmoid.\n",
+    "\n",
+    "In terms of learning speed, the more layers and neurons, the more time it takes to train the network. It's especially visible for networks with tanh and sigmoid activation functions.\n",
     "\n",
     "### Performance on other datasets\n",
     "\n",
+    "For regression tasks, same architecture and activation functions were used as in the comparison above.\n",
+    "\n",
+    "For the classification tasks, all networks had the same architecture of 3 hidden layers with 20 neurons each. The output layer had a tanh activation function to map the output to values closer to 0. Hidden layers were using either ReLU or tanh activation functions as sigmoid has poor implementation and is therefore slower.\n",
+    "\n",
     "#### Regression - steps-large\n",
     "\n",
     "Model 1 - 1x30 neurons, tahn activation function (on output layer linear activation)\n",