2022-06-12-Information-Theory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Information \n",
    "> Inductive inference, is the problem of reasoning under conditions of incomplete information, or `uncertainty`. According to Shannon's theory, information and uncertainty are two sides of the same coin: the more uncertainty there is, the more information we gain by removing the uncertainty. \n",
    "\n",
    "* toc: true\n",
    "* badges: true\n",
    "* comments: true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logarithm in Information Theory\n",
    "\n",
    "It is important to know binary logarithm $log_2$ to understand mathematical calculation related to Information Throry.\n",
    "\n",
    "Logarithm is the another way to represent indices or power "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$log_x(y) = z \\ represent \\ x^z=y \\\\\n",
    "\\therefore log_2(4)=2 \\ as \\  2^2 = 4 \\ and \\ log_2(8)=3 \\ as \\ 2^3 = 8 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three basic laws of logarithms:\n",
    "$$\n",
    "log A + logB= logAB \\\\\n",
    "logA^n = nlogA \\\\\n",
    "logA-logB = log \\frac{A}{B}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Entropy is a measure of the uncertainty (randomeness) of a probability distribution.\n",
    "\n",
    "Mathematicaly is it defined as `The entropy of a random rariable $X$ with a probability mas function p(x) is defined by \n",
    "\n",
    "Shannon `entropy` formula:\n",
    "\n",
    "$$H(X) = -\\sum_{i=1}^n p_i \\log_2 p_i$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shannon found that entropy was the only funcion satisfying the three requirements \n",
    "\n",
    "1. $H(X)$ is always non-negtive, since information cann't be lost.\n",
    "2. The uniform distribution maximize $H(X)$, since it also maximizes uncertainty.\n",
    "3. The additivity property which relate the sum of entropies of two independent events. For instance, in thermodynamics, the total entropy of two isolated system which co-exist in equilibrium is the same as the sum of the entropies each system in isolation.m \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problems:\n",
    "1. For an event which is certain to happen, what is the entropy of the event?\n",
    "2. For an event which is uncertain, what is the entropy of the event?\n",
    "3. For n equiprobable event, what is the entropy of the event?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With respect to the notion of `surprise` in the context of information theory:\n",
    "\n",
    "1. Define what is actually mean by being surprised.\n",
    "2. Describe how it is related to the likelihood of an event happening.\n",
    "3. True or False: The less likely the occurrence of an event, the smaller information it conveys."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. The notion of surprise is directly related to the likelihood of an event happenning. Mathematically it is inversely propotional to the probability of that event. A high-probability event gives less information than los-probability event. \n",
    "2. Learning that a high-probability event has taken place, for instance the sun rising, is much less of a surprise than a low-probability event, for instance the moon landing.\n",
    "3. Therefore, the less likely the likely  the occurence of an event, the greater information it conveys. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reference: https://arxiv.org/ftp/arxiv/papers/2201/2201.00650.pdf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "4f946df053fbf2b937619d3c5458e7af74262f9a954d8797ba0b27400bcafe06"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}