1-map-filter-reduce-groupby.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Map, Filter, Reduce, and Groupby\n",
    "\n",
    "本部分展示高阶函数应用\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def square(x):\n",
    "    return x ** 2\n",
    "\n",
    "def iseven(n):\n",
    "    return n % 2 == 0\n",
    "\n",
    "def add(x, y):\n",
    "    return x + y\n",
    "\n",
    "def mul(x, y):\n",
    "    return x * y\n",
    "\n",
    "def lesser(x, y):\n",
    "    if x < y:\n",
    "        return x\n",
    "    else:\n",
    "        return y\n",
    "\n",
    "def greater(x, y):\n",
    "    if x > y:\n",
    "        return x\n",
    "    else:\n",
    "        return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# map works like this\n",
    "list(map(square, data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2,  4,  6,  8, 10, 12, 14, 16, 18, 20])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In this way it's like numpy's broadcasting operators\n",
    "import numpy as np\n",
    "X = np.arange(1, 11)\n",
    "X*2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是 `map` 是纯python函数，所以 \n",
    "\n",
    "*   很慢\n",
    "*   能够处理普通的函数比如fibonacci"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fib(i):\n",
    "    if i in (0, 1):\n",
    "        return i\n",
    "    else:\n",
    "        return fib(i - 1) + fib(i - 2)\n",
    "    \n",
    "list(map(fib, data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 通过情况下，我们将会这样应用他们\n",
    "result = []\n",
    "for item in data:\n",
    "    result.append(fib(item))\n",
    "    \n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 查看下上面给的好的范式说明了如何定义一个map方法 map looking at the function above gives us a good pattern for how to define `map`\n",
    "# 我们将自己抽象定义一个map函数  We just abstract out the function `fib` for a user input\n",
    "\n",
    "# `map` is easy to define\n",
    "def map(fn, sequence):\n",
    "    result = []\n",
    "    for item in sequence:\n",
    "        result.append(fn(item))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "鲜为人知的事实是，对象的方法也是完全有效的函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ALICE', 'BOB', 'CHARLIE']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(str.upper, ['Alice', 'Bob', 'Charlie'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "map函数是非常重要的，它有自己的语法，**列表理解**，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[fib(i) for i in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ALICE', 'BOB', 'CHARLIE']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name.upper() for name in ['Alice', 'Bob', 'Charlie']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter\n",
    "\n",
    "“过滤器” 高阶函数通过断言方式来过滤数据集。\n",
    "\n",
    "断言是一个返回“True”或“False”的函数。 `filter`函数返回一个新的只有断言为true的元素的列表。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 4, 6, 8, 10]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(filter(iseven, data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 3, 5, 7]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sympy import isprime  # Only works if you have the sympy math library installed\n",
    "list(filter(isprime, data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def filter(predicate, sequence):\n",
    "    result = []\n",
    "    for item in sequence:\n",
    "        if predicate(item):\n",
    "            result.append(item)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reduce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reduce是`map`和`filter`的小兄弟。 Reduce不太受欢迎，经常被责骂难以理解。\n",
    "\n",
    "尽管社会问题“Reduce”是相当强大的，一旦你写了“Reduce”，一旦你明白它是如何工作的。 更重要的是，您将学习如何识别约简操作以及如何将它们与二元运算符配对。 数据分析中的降维操作很常见的，特别是在将大型数据集缩减为概要时。\n",
    "\n",
    "为了显示“Reduce”，我们将首先实现两个常见的减少，“sum”和“min”。 我们已经用二元运算符`add`和`lessser`来暗示它们，以突出它们的类似结构。 选出以下两个互不相同功能的部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sum(sequence):\n",
    "    result = 0\n",
    "    for item in sequence:\n",
    "        # reult = result + item\n",
    "        result = add(result, item)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def min(sequence):\n",
    "    result = 99999999999999  # a really big number\n",
    "    for item in sequence:\n",
    "        # result = result if result < item else item\n",
    "        result = lesser(result, item)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "现在填写下面的空白来完成`product`的定义，这个函数将序列的元素放在一起。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-16-92db2dd2fc1e>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-16-92db2dd2fc1e>\"\u001b[1;36m, line \u001b[1;32m2\u001b[0m\n\u001b[1;33m    result = ?\u001b[0m\n\u001b[1;37m             ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "def product(sequence):\n",
    "    result = ?\n",
    "    for item in sequence:\n",
    "        result = ?(result, item)\n",
    "    return result\n",
    "\n",
    "assert product([2, 3, 10]) == 60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "实现 `reduce`.\n",
    "\n",
    "首先复制上述三个函数的模式。 三者之间的差异是你的输入。 传统上，reduce的论点是有序的，以便下面的例子运行良好。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-17-880dfea8dc75>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-17-880dfea8dc75>\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m    def reduce(...):\u001b[0m\n\u001b[1;37m               ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "def reduce(...):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from functools import reduce\n",
    "reduce(add, data, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3628800"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduce(mul, data, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduce(lesser, data, 10000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduce(greater, data, -100000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们这部分课程有很多像这样小的函数定义\n",
    "\n",
    "```\n",
    "def add(x, y):\n",
    "    return x + y\n",
    "```\n",
    "\n",
    "这些单线功能有时看起来有点愚蠢。 我们使用`lambda`关键字来即时创建小函数。 上述定义可以表述如下\n",
    "\n",
    "```\n",
    "add = lambda x, y: x + y\n",
    "```\n",
    "\n",
    "表达式`lambda x，y：x + y`是一个值，就像`3`或`Alice`一样。 就像文字整数和字符串一样，Lambda表达式可以在不用变量存储的情况下即时使用。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduce(add, data, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduce(lambda x, y: x + y, data, 0)  # Define `add` on the fly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外，我们可以使用`lambda`来快速指定函数作为更一般化的特化。 在下面我们快速定义总和，最小值和最大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sum = lambda data: reduce(add, data, 0)\n",
    "min = lambda data: reduce(lesser, data, 99999999999)\n",
    "max = lambda data: reduce(greater, data, -999999999999)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "作业练习，自己定制一个 `product` 使用下 `lambda`, `reduce`, 和 `mul`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-26-405b2d336b95>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-26-405b2d336b95>\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m    product = ...\u001b[0m\n\u001b[1;37m              ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "product = ...\n",
    "assert product([2, 3, 10]) == 60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Groupby可以被看作是“filter”的一个更强大的版本。 而不是给你一个数据的子集，它把数据分成所有相关的子集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 4, 6, 8, 10]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filter(iseven, data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{False: [1, 3, 5, 7, 9], True: [2, 4, 6, 8, 10]}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from toolz import groupby\n",
    "groupby(iseven, data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{False: [1, 4, 6, 8, 9, 10], True: [2, 3, 5, 7]}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groupby(isprime, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是 `groupby` 不能严格的断言 (传递函数返回 `True` 或 `False`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: [3, 6, 9], 1: [1, 4, 7, 10], 2: [2, 5, 8]}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groupby(lambda n: n % 3, data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{3: ['Bob', 'Dan'], 5: ['Alice', 'Edith', 'Frank'], 7: ['Charlie']}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groupby(len, ['Alice', 'Bob', 'Charlie', 'Dan', 'Edith', 'Frank'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "令人惊讶的是`groupby`在普通情况下并不比`filter` 消费更多资源。 它通过数据一次性计算这些数据组。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integrative example\n",
    "\n",
    "让我们把它们放在一个小数据集中展示。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "likes = \"\"\"Alice likes Chocolate\n",
    "Bob likes Chocolate\n",
    "Bob likes Apples\n",
    "Charlie likes Apples\n",
    "Alice likes Peanut Butter\n",
    "Charlie likes Peanut Butter\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<map at 0xe5f2c88>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuples = map(lambda s: s.split(' likes '), likes.split('\\n'))\n",
    "tuples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Alice': [['Alice', 'Chocolate'], ['Alice', 'Peanut Butter']],\n",
       " 'Bob': [['Bob', 'Chocolate'], ['Bob', 'Apples']],\n",
       " 'Charlie': [['Charlie', 'Apples'], ['Charlie', 'Peanut Butter']]}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groups = groupby(lambda x: x[0], tuples)\n",
    "groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Alice': ['Chocolate', 'Peanut Butter'],\n",
       " 'Bob': ['Chocolate', 'Apples'],\n",
       " 'Charlie': ['Apples', 'Peanut Butter']}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from toolz import valmap, first, second\n",
    "valmap(lambda L: list(map(second, L)), groups)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Apples': ['Bob', 'Charlie'],\n",
       " 'Chocolate': ['Alice', 'Bob'],\n",
       " 'Peanut Butter': ['Alice', 'Charlie']}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuples = map(lambda s: s.split(' likes '), likes.split('\\n'))\n",
    "valmap(lambda L: list(map(first, L)), groupby(lambda x: x[1], tuples))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Apples': ['Bob', 'Apples'],\n",
       " 'Chocolate': ['Alice', 'Chocolate'],\n",
       " 'Peanut Butter': ['Alice', 'Peanut Butter']}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuples = list(map(lambda s: s.split(' likes '), likes.split('\\n')))\n",
    "# first second 取数据\n",
    "# groupby 数据分组\n",
    "# compose 函数组合\n",
    "# valmap 字典元组计算\n",
    "from toolz.curried import map, valmap, groupby, first, second, get, curry, compose, pipe\n",
    "\n",
    "f = compose(valmap(first), groupby(second))\n",
    "\n",
    "f(tuples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}