C5Week3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week3\n",
    "\n",
    "- Sequence to Sequence Model\n",
    "- Beam Search\n",
    "- Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2Seq\n",
    "\n",
    "seq2seq 是一个Encoder–Decoder 结构的网络，它的输入是一个序列，输出也是一个序列， Encoder 中将一个可变长度的信号序列变为固定长度的向量表达，Decoder 将这个固定长度的向量变成可变长度的目标的信号序列。\n",
    "\n",
    "<img src=\"src/pictures/c5_w3_2.jpeg\">\n",
    "\n",
    "RNN中当前时间步的隐藏状态由上一个时间步的隐藏状态和输入决定\n",
    "$$h_t = f(h_{t-1}, x_t)$$\n",
    "获得各个时间步的隐藏层, 将隐藏层信息进行汇总生成语义向量\n",
    "$$C=q(h_1, h_2, h_3,...,h_{T_x})$$\n",
    "\n",
    "最简单的语义向量就是$T_x$时间步的隐藏层作为语义向量.\n",
    "\n",
    "解码阶段, 根据语义C和之前已经生成的输出序列来预测下一时间步的输出单词$y_t$, 有\n",
    "$$y_t=argmax_y P(y_t) \\\\ \n",
    "\\qquad = argmax \\prod_{t=1}^T p(y_t|y_1, y_2, \\dots, y_{t-1}, C) \\\\\n",
    "\\qquad = argmax \\sum_{t=1}^T log p(y_t|y_1, y_2, \\dots, y_{t-1}, C)$$\n",
    "\n",
    "对于机器翻译等问题, 输出的目标是获得最可能的句子, 即要求$P(y_1, y_2, \\dots, y_t|X)$整体最大, 如果使用Greedy Search, 在t时间步选择概率最大的单词$y_t$作为输出, 不能够得到整体最优, 因此提出了Beam Search(束搜索)来获得近似全局最优的句子.\n",
    "\n",
    "## Beam Search\n",
    "\n",
    "束搜索的直观理解是在t时间步选择概率最大的B个$\\{y_1, y_2, \\dots, y_t\\}$序列代替只选择概率当前概率最大的序列, 其中B称为束宽度. 如B=2的情形如下图:\n",
    "\n",
    "![B=2](src/pictures/c5_w3_4.png)\n",
    "\n",
    "基本的B结构偏向于短句, 因为连乘小于1的数值, 越长数值一定越小, 为解决这个问题, 提出长度正则化, 将序列的概率计算变为:\n",
    "$$\\frac{1}{T^\\alpha} argmax \\sum_{t=1}^T log p(y_t|y_1, y_2, \\dots, y_{t-1}, C)$$\n",
    "\n",
    "其中$\\alpha$控制正则化的柔和程度, 当$\\alpha=1$时最严苛, 当$\\alpha = 0$时没有正则化.\n",
    "\n",
    "B的参数选择也对搜索的性能具有影响, B越小,搜索越快搜索结果更差;B越大计算量更大搜索结果更好. 工业上常采用B=10, 20, 研究中最大常才用到1000.\n",
    "\n",
    "### 加入BS的误差分析\n",
    "\n",
    "如何判断使用BS机制和RNN结构进行训练时, 出现误差时是哪个部分出现了问题?\n",
    "\n",
    "可以对输入的目标结果(人工翻译的结果), 使用上述的RNN的权重和上述的评价式计算其概率$y^*$, 同时计算机器翻译的结果$\\hat{y}$, 若$y^* > \\hat{y}$, 说明BS不能够搜索到最优的结果;若$y^* \\leq \\hat{y}$, 说明RNN不能正确的估算正确结果的概率.\n",
    "\n",
    "## Attention\n",
    "\n",
    "Attention机制使神经网络在当前时间步只关注序列的一部分. 在Seq2Seq模型中加入Attention机制, 是利用Decoder上一个时间步t-1的隐藏层信息$S_{t-1}$和encoder的各时间步$t'$的输出$a_{t'}$通过函数g获得在encoder的各时间步的输出a对当前Decoder时间步t的影响的权重, 换句话说, 使用$S_{t-1}$作为查询在encoder输出$\\{a_1, a_2, \\dots, a_t\\}$为键构成的字典中查询, 获得字典值的加权输出.即\n",
    "$$\\alpha^{<t,t'>}=g(S_{t-1}, a_{t'}) \\\\\n",
    "Context_t = \\alpha_t \\cdot a $$\n",
    "其中$\\alpha_t = \\{\\alpha^{<t,1>}, \\alpha^{<t,2>}, \\dots, \\alpha^{<t,t'>}\\}$\n",
    "\n",
    "其中, g函数的实现有不同的版本, Andrew给出的是利用一个共享参数的全连接层处理,再接一个softmax层得到权重.\n",
    "\n",
    "<table>\n",
    "<td> \n",
    "<img src=\"src/pictures/c5_w3_5.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "<td> \n",
    "<img src=\"src/pictures/c5_w3_6.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "</table>\n",
    "\n",
    "注意力机制可以分为三步: 一是信息输入, 二是计算注意力分布$\\alpha$, 三是根据注意力机制$\\alpha$计算输入信息的加权平均."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class BaseAttention(nn.Module):\n",
    "    '''\n",
    "    Attention mechanism gets the output of encoder a(a_hidden_size, batch, str_len), \n",
    "    and hidden state s_t-1(s_hidden_size, batch) from the last time step in  decoder\n",
    "    '''\n",
    "    def __init__(self, a_dim, s_dim):\n",
    "        super(attention, self).__init__()\n",
    "        self.dense = nn.Sequential(\n",
    "            nn.Linear(a_dim + s_dim, 10),\n",
    "            nn.Tanh(),\n",
    "            nn.Linear(10, 1),\n",
    "            nn.RelU())\n",
    "        self.softmax = nn.Softmax(dim=2)\n",
    "    \n",
    "    def forward(self, a, st):\n",
    "        '''\n",
    "        a -- (batch, str_len, a_dim)\n",
    "        st -- (batch, s_dim)\n",
    "        '''\n",
    "        str_len = a.shape[1]\n",
    "        s = st.unsqueeze(1).repeat(1, str_len, 1)\n",
    "        concat = torch.cat((a, s), axis=2) # (batch, str_len, a_dim+s_dim)\n",
    "        a = self.dense(concat) # (batch, str_len, 1)\n",
    "        alpha = self.softmax(a2) # (batch, str_len, 1)\n",
    "        context = alpha.mul(a) # (batch, str_len, a_dim)\n",
    "        return context\n",
    "\n",
    "def test():\n",
    "    a_dim, s_dim = 10, 8\n",
    "    att = BaseAttention(a_dim, s_dim)\n",
    "    a = torch.rand(3, 5, a_dim)\n",
    "    st = torch.rand(3, s_dim)\n",
    "    context = att(a, st)\n",
    "    print(context)\n",
    "\n",
    "test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Attention机制的实质其实就是一个寻址（addressing）的过程**，这个过程实际上是Attention机制缓解神经网络模型复杂度的体现：不需要将所有的N个输入信息都输入到神经网络进行计算，只需要从X中选择一些和任务相关的信息输入给神经网络。\n",
    "\n",
    "g函数的计算有不同的版本:\n",
    "\n",
    "- 加性模型: $g(S_t, a) = v^Ttanh(WS_t,Ua)$\n",
    "- 点积模型: $g(S_t, a) = S_i^Ta$\n",
    "- 缩放点积模型: $g(S_t, a) = \\frac{S_i^Ta}{\\sqrt{d}}$\n",
    "- 双线性模型: $g(S_t, a) = S_i^TWa$\n",
    "\n",
    "使用$context_t = \\alpha_t \\cdot a$进行编码的称为软性注意力机制, 而选择概率最高的输入信息/根据概率对输入信息进行采样的编码方式称为硬性注意力机制.\n",
    "\n",
    "> 硬性注意力的一个缺点是基于最大采样或随机采样的方式来选择信息。因此最终的损失函数与注意力分布之间的函数关系不可导，因此无法使用在反向传播算法进行训练。为了使用反向传播算法，一般使用软性注意力来代替硬性注意力。硬性注意力需要通过强化学习来进行训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多头注意力机制\n",
    "多头注意力是将键和查询在hidden_size维拆分成m份, 平行的计算每部分查询对键的注意力, 然后将每段查询的结果拼接回来, 得到最终的结果."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multi Head Tensorflow\n",
    "\n",
    "# Linear projections\n",
    "Q = tf.layers.dense(inputs, num_units, activation=tf.nn.relu) # [batch_size, seq_length, n_hidden]\n",
    "K = tf.layers.dense(inputs, num_units, activation=tf.nn.relu) # [batch_size, seq_length, n_hidden]\n",
    "V = tf.layers.dense(inputs, num_units, activation=tf.nn.relu) # [batch_size, seq_length, n_hidden]\n",
    "\n",
    "# 在这里拆分成多头\n",
    "Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0) # [batch_size, seq_length, n_hidden/num_heads]\n",
    "K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0) # [batch_size, seq_length, n_hidden/num_heads]\n",
    "V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0) # [batch_size, seq_length, n_hidden/num_heads]\n",
    "\n",
    "# Multiplication\n",
    "outputs = tf.matmul(Q_, tf.transpose(K_, [0, 2, 1])) # num_heads*[batch_size, seq_length, seq_length]\n",
    "\n",
    "# Scale\n",
    "outputs = outputs / (K_.get_shape().as_list()[-1] ** 0.5)\n",
    "\n",
    "# Activation\n",
    "outputs = tf.nn.softmax(outputs) # num_heads*[batch_size, seq_length, seq_length]\n",
    "\n",
    "# Dropouts\n",
    "outputs = tf.layers.dropout(outputs, rate=dropout_rate, training=tf.convert_to_tensor(is_training))\n",
    "\n",
    "# Weighted sum\n",
    "outputs = tf.matmul(outputs, V_) # num_heads*[batch_size, seq_length, n_hidden/num_heads]\n",
    "\n",
    "# 在这里将多头求得的结果进行合并\n",
    "outputs = tf.concat(tf.split(outputs, num_heads, axis=0), axis=2 ) # [batch_size, seq_length, n_hidden]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 自注意力机制\n",
    "\n",
    "自注意力机制是键, 值, 查询均为input本身的Attention, 自注意力层的实质, 是可以处理边长输入序列的全连接层.\n",
    "\n",
    ">如果要建立输入序列之间的长距离依赖关系，可以使用以下两种方法：一 种方法是增加网络的层数，通过一个深层网络来获取远距离的信息交互，另一种方法是使用全连接网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Transformer（Attention Is All You Need）详解](https://zhuanlan.zhihu.com/p/53682800)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}