2148792.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "[我在AI Studio上获得黄金等级，点亮9个徽章，来互关呀~](http://aistudio.baidu.com/aistudio/personalcenter/thirdview/335435) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （1）赛题介绍\n",
    "\n",
    "图神经网络（Graph Neural Network）是一种专门处理图结构数据的神经网络，目前被广泛应用于推荐系统、金融风控、生物计算等领域。图神经网络的经典问题主要有三类，分别为节点分类、连接预测和图分类。本次比赛旨在让参赛同学了解并掌握如何使用图神经网络处理节点分类问题。\n",
    "\n",
    "在过去的一个世纪里，科学出版物的数量每12年增加近一倍，对每一种出版物的主题及领域进行自动分类已成为当下十分重要的工作。本次任务的目标是预测未知论文的主题类别，如软件工程，人工智能，语言计算和操作系统等。比赛所选35个领域标签已得到论文作者和arXiv版主确认并标记。\n",
    "\n",
    "本次比赛选用的数据集为arXiv论文引用网络——ogbn-arixv数据集的子集。ogbn-arixv数据集由大量的学术论文组成，论文之间的引用关系形成一张巨大的有向图，每一条有向边表示一篇论文引用另一篇论文，每一个节点提供100维简单的词向量作为节点特征。在论文引用网络中，我们已对训练集对应节点做了论文类别标注处理。本次任务希望参赛者通过已有的节点类别以及论文之间的引用关系，预测未知节点的论文类别。\n",
    "\n",
    "\n",
    "[赛题地址](https://aistudio.baidu.com/aistudio/competition/detail/59)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （2）重要参考\n",
    "###  1.UniMP算法\n",
    "[UniMP算法GitHub链接](https://github.com/PaddlePaddle/PGL/tree/main/ogb_examples/nodeproppred/unimp)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### 2.参考代码\n",
    "[https://aistudio.baidu.com/aistudio/projectdetail/1467127?channelType=0&channel=0](https://aistudio.baidu.com/aistudio/projectdetail/1467127?channelType=0&channel=0)\n",
    "\n",
    "[飞桨常规赛：图神经网络入门节点分类 5月第4名方案](https://aistudio.baidu.com/aistudio/projectdetail/1931047)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （3）具体方案分享"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 环境配置\n",
    "\n",
    "```\n",
    "\n",
    "#导入相关包\n",
    "!pip install --upgrade python-dateutil\n",
    "!pip install easydict\n",
    "!pip install pgl==1.2.0 \n",
    "!pip install pandas>=0.25\n",
    "!pip install pyarrow==0.13.0\n",
    "!pip install chardet==3.0.4\n",
    "\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "# 尝试使用标签进行训练：\n",
    "（试验结果：过拟合严重，但可以为最后投票集成提供数据，代码仅供参考）\n",
    "### ①从训练集中随机选择35个类别各1个\n",
    "### ②利用余弦相似度计算每个feat可能的类别\n",
    "### ③把每个feat中用余弦相似度计算出的标签类别的位置设置成1，其余位置设置为0\n",
    "### ④用新的feat进行训练\n",
    "### 代码如下：\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "edges = pd.read_csv(\"work/edges.csv\", header=None, names=[\"src\", \"dst\"]).values\n",
    "node_feat = np.load(\"work/feat.npy\")\n",
    "df = pd.read_csv(\"work/train.csv\")\n",
    "node_index = df[\"nid\"].values\n",
    "node_label = df[\"label\"].values\n",
    "feat_dict={}\n",
    "feat1=np.zeros([node_feat.shape[0],100],dtype='float32')\n",
    "for i in range(len(node_feat)):\n",
    "    feat_dict[i]=0\n",
    "for i in range(len(node_index)):\n",
    "    if i<int(0.8*len(node_index)):\n",
    "        feat_dict[node_index[i]]=1\n",
    "    else:\n",
    "        feat_dict[node_index[i]]=2\n",
    "node_labels={}\n",
    "for i in range(len(node_index)):\n",
    "    node_labels[node_index[i]]=node_label[i]\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "def cos_sim(vector_a, vector_b):\n",
    "    \"\"\"\n",
    "    计算两个向量之间的余弦相似度\n",
    "     :param vector_a: 向量 a\n",
    "     :param vector_b: 向量 b\n",
    "    :return: sim\n",
    "    \"\"\"\n",
    "    vector_a = np.mat(vector_a)\n",
    "    vector_b = np.mat(vector_b)\n",
    "    num = float(vector_a * vector_b.T)\n",
    "    denom = np.linalg.norm(vector_a) * np.linalg.norm(vector_b)\n",
    "    cos = num / denom\n",
    "    sim = 0.5 + 0.5 * cos\n",
    "    return sim\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "rnd_list=[]\n",
    "\n",
    "for j in range(35):\n",
    "    while True:\n",
    "        n=np.random.randint(0,len(node_index))\n",
    "        if node_label[n]==j:\n",
    "            rnd_list.append(n)      \n",
    "            break      \n",
    "\n",
    "\n",
    "for i in range(len(node_feat)):\n",
    "    \n",
    "    if feat_dict[i]!=1:\n",
    "        cos_sim_max=0\n",
    "        j_max=0\n",
    "        for j in range(len(rnd_list)):\n",
    "            sim=cos_sim(node_feat[i],node_feat[rnd_list[j]])\n",
    "            if sim>cos_sim_max:\n",
    "                com_sim_max=sim\n",
    "                j_max=j\n",
    "            if cos_sim_max>0.95:\n",
    "                break\n",
    "        feat1[i,node_label[rnd_list[j_max]]]=1.0\n",
    "    else:\n",
    "        feat1[i,node_labels[i]]=1.0\n",
    "    if i%10000==0:\n",
    "        print(i)\n",
    "node_feat=feat1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （4）代码实现\n",
    "\n",
    "###  1.模型构建（mode.py）\n",
    "Res_Unimp_Large代码，也可参见model_modified.py。\n",
    "\n",
    "```\n",
    "class res_unimp_large(object):\n",
    "    def __init__(self, config, num_class):\n",
    "        self.num_class = num_class\n",
    "        self.num_layers = config.get(\"num_layers\", 2)\n",
    "        self.hidden_size = config.get(\"hidden_size\", 128)\n",
    "        self.out_size=config.get(\"out_size\", 40)\n",
    "        self.embed_size=config.get(\"embed_size\", 100)\n",
    "        self.heads = config.get(\"heads\", 8) \n",
    "        self.dropout = config.get(\"dropout\", 0.3)\n",
    "        self.edge_dropout = config.get(\"edge_dropout\", 0.0)\n",
    "        self.use_label_e = config.get(\"use_label_e\", False)\n",
    "    \n",
    "    # 编码输入        \n",
    "    def embed_input(self, feature):   \n",
    "        lay_norm_attr=F.ParamAttr(initializer=F.initializer.ConstantInitializer(value=1))\n",
    "        lay_norm_bias=F.ParamAttr(initializer=F.initializer.ConstantInitializer(value=0))\n",
    "        feature=L.layer_norm(feature, name='layer_norm_feature_input', \n",
    "                                      param_attr=lay_norm_attr, \n",
    "                                      bias_attr=lay_norm_bias)\n",
    "        return feature\n",
    "    \n",
    "    # 连同部分已知的标签编码输入（MaskLabel）\n",
    "    def label_embed_input(self, feature):\n",
    "        label = F.data(name=\"label\", shape=[None, 1], dtype=\"int64\")\n",
    "        label_idx = F.data(name='label_idx', shape=[None, 1], dtype=\"int64\")\n",
    "\n",
    "        label = L.reshape(label, shape=[-1])\n",
    "        label_idx = L.reshape(label_idx, shape=[-1])\n",
    "\n",
    "        embed_attr = F.ParamAttr(initializer=F.initializer.NormalInitializer(loc=0.0, scale=1.0))\n",
    "        embed = F.embedding(input=label, size=(self.out_size, self.embed_size), param_attr=embed_attr )\n",
    "\n",
    "        feature_label = L.gather(feature, label_idx, overwrite=False)\n",
    "        feature_label = feature_label + embed\n",
    "        feature = L.scatter(feature, label_idx, feature_label, overwrite=True)\n",
    "     \n",
    "        lay_norm_attr = F.ParamAttr(initializer=F.initializer.ConstantInitializer(value=1))\n",
    "        lay_norm_bias = F.ParamAttr(initializer=F.initializer.ConstantInitializer(value=0))\n",
    "        feature = L.layer_norm(feature, name='layer_norm_feature_input', \n",
    "                                      param_attr=lay_norm_attr, \n",
    "                                      bias_attr=lay_norm_bias)\n",
    "        return feature\n",
    "        \n",
    "    def forward(self, graph_wrapper, feature, phase):\n",
    "        if phase == \"train\": \n",
    "            edge_dropout = self.edge_dropout\n",
    "            dropout = self.dropout\n",
    "        else:\n",
    "            edge_dropout = 0\n",
    "            dropout = 0\n",
    "\n",
    "        if self.use_label_e:\n",
    "            feature = self.label_embed_input(feature)\n",
    "        else:\n",
    "            feature = self.embed_input(feature)\n",
    "        if dropout > 0:\n",
    "            feature = L.dropout(feature, dropout_prob=dropout, \n",
    "                                    dropout_implementation='upscale_in_train')\n",
    "        \n",
    "        #改变输入特征维度是为了Res连接可以直接相加\n",
    "        feature = L.fc(feature, size=self.hidden_size * self.heads, name=\"init_feature\")\n",
    "\n",
    "\n",
    "        for i in range(self.num_layers - 1):\n",
    "            ngw = pgl.sample.edge_drop(graph_wrapper, edge_dropout) \n",
    "            from model_unimp_large import graph_transformer, attn_appnp\n",
    "\n",
    "            res_feature = feature\n",
    "\n",
    "            feature, _, cks = graph_transformer(str(i), ngw, feature, \n",
    "                                             hidden_size=self.hidden_size,\n",
    "                                             num_heads=self.heads, \n",
    "                                             concat=True, skip_feat=True,\n",
    "                                             layer_norm=True, relu=True, gate=True)\n",
    "            if dropout > 0:\n",
    "                feature = L.dropout(feature, dropout_prob=dropout, \n",
    "                                     dropout_implementation='upscale_in_train') \n",
    "            \n",
    "            # 下面这行便是Res连接了\n",
    "            feature = res_feature + feature \n",
    "        \n",
    "        feature, attn, cks = graph_transformer(str(self.num_layers - 1), ngw, feature, \n",
    "                                             hidden_size=self.out_size,\n",
    "                                             num_heads=self.heads, \n",
    "                                             concat=False, skip_feat=True,\n",
    "                                             layer_norm=False, relu=False, gate=True)\n",
    "\n",
    "        feature = attn_appnp(ngw, feature, attn, alpha=0.2, k_hop=10)\n",
    "\n",
    "        pred = L.fc(\n",
    "            feature, self.num_class, act=None, name=\"pred_output\")\n",
    "        return pred\n",
    "```\n",
    "        \n",
    "### 2.模型配置（Notebook）\n",
    "最优策略：3层res_unimp_large，隐层神经元128个，配置两种dropout，使用MaskLabel，且label_rate = 0.66（在模型训练中设置）。\n",
    "\n",
    "```\n",
    "config = {\n",
    "    \"model_name\": \"res_unimp_large\",\n",
    "    \"num_layers\": 3,\n",
    "    \"hidden_size\": 128,\n",
    "    \"heads\": 2,\n",
    "    \"learning_rate\": 0.001,\n",
    "    \"dropout\": 0.33,\n",
    "    \"weight_decay\": 0.0005,\n",
    "    \"edge_dropout\": 0.32,\n",
    "    \"use_label_e\": True\n",
    "}\n",
    "\n",
    "```\n",
    "###  3.模型训练（Notebook）\n",
    "\n",
    "```\n",
    "import os\n",
    "use_label_e = True\n",
    "label_rate = 0.66\n",
    "epoch = 4000\n",
    "exe.run(startup_program)\n",
    "max_val_acc = 0\n",
    "\n",
    "# 这里可以恢复训练\n",
    "pretrained = False\n",
    "if pretrained:\n",
    "    def name_filter(var):\n",
    "        res = var.name in os.listdir('./output')\n",
    "        return res\n",
    "    fluid.io.load_vars(exe, './output',predicate=name_filter)\n",
    "    max_val_acc = 0.756\n",
    "\n",
    "earlystop = 0\n",
    "# 将图数据变成 feed_dict 用于传入Paddle Excecutor\n",
    "feed_dict = gw.to_feed(dataset.graph)\n",
    "for epoch in range(epoch):\n",
    "    # Full Batch 训练\n",
    "    # 设定图上面那些节点要获取\n",
    "    # node_index: 未知label节点的nid    \n",
    "    # node_label: 未知label\n",
    "    # label_idx: 已知label节点的nid    \n",
    "    # label: 已知label\n",
    "    \n",
    "    if use_label_e:\n",
    "        # 在训练集中抽取部分数据，其Label已知，并可以输入网络训练\n",
    "        train_idx_temp = np.array(train_index, dtype=\"int64\")\n",
    "        train_lab_temp = np.array(train_label, dtype=\"int64\")\n",
    "        state = np.random.get_state()\n",
    "        np.random.shuffle(train_idx_temp)\n",
    "        np.random.set_state(state)\n",
    "        np.random.shuffle(train_lab_temp)\n",
    "\n",
    "        label_idx=train_idx_temp[:int(label_rate*len(train_idx_temp))]\n",
    "        unlabel_idx=train_idx_temp[int(label_rate*len(train_idx_temp)):]\n",
    "        label=train_lab_temp[:int(label_rate*len(train_idx_temp))]\n",
    "        unlabel=train_lab_temp[int(label_rate*len(train_idx_temp)):]\n",
    "\n",
    "        feed_dict[\"node_index\"] = unlabel_idx\n",
    "        feed_dict[\"node_label\"] = unlabel\n",
    "        feed_dict['label_idx']= label_idx\n",
    "        feed_dict['label']= label\n",
    "    else:\n",
    "        feed_dict[\"node_label\"] = np.array(train_label, dtype=\"int64\")\n",
    "        feed_dict[\"node_index\"] = np.array(train_index, dtype=\"int64\")\n",
    "        \n",
    "\n",
    "    train_loss, train_acc = exe.run(train_program,\n",
    "                                feed=feed_dict,\n",
    "                                fetch_list=[loss, acc],\n",
    "                                return_numpy=True)\n",
    "\n",
    "    # Full Batch 验证\n",
    "    # 设定图上面那些节点要获取\n",
    "    # node_index: 未知label节点的nid    \n",
    "    # node_label: 未知label\n",
    "    # label_idx: 已知label节点的nid    \n",
    "    # label: 已知label\n",
    "    \n",
    "    feed_dict[\"node_index\"] = np.array(val_index, dtype=\"int64\")\n",
    "    feed_dict[\"node_label\"] = np.array(val_label, dtype=\"int64\")\n",
    "    if use_label_e:\n",
    "        feed_dict['label_idx'] = np.array(train_index, dtype=\"int64\")\n",
    "        feed_dict['label'] = np.array(train_label, dtype=\"int64\")\n",
    "    val_loss, val_acc = exe.run(test_program,\n",
    "                            feed=feed_dict,\n",
    "                            fetch_list=[v_loss, v_acc],\n",
    "                            return_numpy=True)\n",
    "    print(\"Epoch\", epoch, \"Train Acc\", train_acc[0], \"Valid Acc\", val_acc[0])\n",
    "    \n",
    "    # 保存历史最优验证精度对应的模型\n",
    "    if val_acc[0] > max_val_acc:\n",
    "        max_val_acc = val_acc[0]\n",
    "        fluid.io.save_persistables(exe, './output', train_program)\n",
    "    \n",
    "    # 训练精度持续大于验证精度，结束训练\n",
    "    if train_acc[0] > val_acc[0]:\n",
    "        earlystop += 1\n",
    "        if earlystop == 40:\n",
    "            break\n",
    "    else:\n",
    "        earlystop = 0\n",
    "```\n",
    "\n",
    "### 4.简单投票\n",
    "```\n",
    "def publicnum(num, d = 0):\n",
    "    dictnum = {}\n",
    "    for i in range(len(num)):\n",
    "        if num[i] in dictnum.keys():\n",
    "            dictnum[num[i]] += 1\n",
    "        else:\n",
    "            dictnum.setdefault(num[i], 1)\n",
    "    maxnum = 0\n",
    "    maxkey = 0\n",
    "    for k, v in dictnum.items():\n",
    "        if v > maxnum:\n",
    "            maxnum = v\n",
    "            maxkey = k\n",
    "    return maxkey\n",
    "\n",
    "df0=pd.read_csv(\"submission0.76136.csv\")\n",
    "df1=pd.read_csv(\"submission0.757822.csv\")\n",
    "df2=pd.read_csv(\"submission0.7583.csv\")\n",
    "df3=pd.read_csv(\"submission0.75758.csv\")\n",
    "df4=pd.read_csv(\"submission0.75921.csv\")\n",
    "df5=pd.read_csv(\"submission0.75782.csv\")\n",
    "df6=pd.read_csv(\"submission0.75956.csv\")\n",
    "df7=pd.read_csv(\"submission0.75801.csv\")\n",
    "df8=pd.read_csv(\"submission0.75884.csv\")\n",
    "#df9=pd.read_csv(\"submission9.csv\")\n",
    "#df10=pd.read_csv(\"submission10.csv\")\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "nids=[]\n",
    "labels=[]\n",
    "\n",
    "for i in range(df4.shape[0]):\n",
    "    label_zs=[]\n",
    "    label_zs.append(df0.label[i])\n",
    "    label_zs.append(df1.label[i])\n",
    "    label_zs.append(df2.label[i])\n",
    "    label_zs.append(df3.label[i])\n",
    "    label_zs.append(df4.label[i])\n",
    "    label_zs.append(df5.label[i])\n",
    "    label_zs.append(df6.label[i])\n",
    "    label_zs.append(df7.label[i])\n",
    "    label_zs.append(df8.label[i])\n",
    "    #label_zs.append(df9.label[i])\n",
    "    #label_zs.append(df10.label[i])\n",
    "    lab=publicnum(label_zs, d = 0)\n",
    "    labels.append(lab)\n",
    "    nids.append(df4.nid[i])\n",
    "\n",
    "\n",
    "submission = pd.DataFrame(data={\n",
    "                            \"nid\": nids,\n",
    "                            \"label\": labels\n",
    "                        })\n",
    "submission.to_csv(\"submissiona.csv\", index=False)\n",
    "```\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （5）总结及改善方向\r\n",
    "1. 使用UniMP算法可以提高成绩。\r\n",
    "2. 提前中止有利于减少过拟合提高成绩。\r\n",
    "3. 投票方法能提高成绩，但是存在天花板。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （6）给其他选手学习飞桨的建议\n",
    "\n",
    "####  建议大家多参加百度AI Studio课程，多看别人写的AI Studio项目，也许会有灵感迸发，在比赛中取得更好的成绩。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# （7）One More Thing\n",
    "如果大家还想要别的奇思妙想，可以参考以下论文，他们都在节点分类上有很大提升。\n",
    "\n",
    "[Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://arxiv.org/abs/1810.05997)\n",
    "\n",
    "[Simple and Deep Graph Convolutional Networks](https://arxiv.org/abs/2007.02133)\n",
    "\n",
    "[Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://arxiv.org/abs/2009.03509)\n",
    "\n",
    "[Combining Label Propagation and Simple Models Out-performs Graph Neural Networks](https://arxiv.org/abs/2010.13993)\n",
    "\n",
    "大家也可以看看github的[ UniMP](https://github.com/PaddlePaddle/PGL/tree/main/ogb_examples/nodeproppred/unimp)算法 这个例子，里面有相似的数据集，并且最近也是SOTA效果，有帮助👏欢迎点Star\n",
    "\n",
    "相关课程连接：[图神经网络7日打卡营](http://aistudio.baidu.com/aistudio/education/group/info/1956)\n",
    "\n",
    "代码参考：\n",
    "\n",
    "[论文引用网络节点分类—炼丹经验总结](https://aistudio.baidu.com/aistudio/projectdetail/1642136)\n",
    "\n",
    "[飞桨常规赛：图神经网络入门节点分类 5月第4名方案](https://aistudio.baidu.com/aistudio/projectdetail/1931047)\n",
    "\n",
    "[我在AI Studio上获得黄金等级，点亮9个徽章，来互关呀~](http://aistudio.baidu.com/aistudio/personalcenter/thirdview/335435) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 代码整体逻辑\n",
    "\n",
    "1. 读取提供的数据集，包含构图以及读取节点特征（用户可自己改动边的构造方式）\n",
    "\n",
    "2. 配置化生成模型，用户也可以根据教程进行图神经网络的实现。\n",
    "\n",
    "3. 开始训练\n",
    "\n",
    "4. 执行预测并产生结果文件\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 环境配置\n",
    "\n",
    "该项目依赖飞桨paddlepaddle==1.8.4, 以及pgl==1.2.0。请按照版本号下载对应版本就可运行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!unzip -oq /home/aistudio/data/data93851/graph.zip -d work/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#导入相关包\r\n",
    "!pip install --upgrade python-dateutil\r\n",
    "!pip install easydict\r\n",
    "!pip install pgl==1.2.0 \r\n",
    "!pip install pandas>=0.25\r\n",
    "!pip install pyarrow==0.13.0\r\n",
    "!pip install chardet==3.0.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pgl\n",
    "import paddle.fluid as fluid\n",
    "import numpy as np\n",
    "import time\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 图网络配置\n",
    "\n",
    "这里已经有很多强大的模型配置了，你可以尝试简单的改一下config的字段。\n",
    "例如，换成GAT的配置\n",
    "```\n",
    "config = {\n",
    "    \"model_name\": \"GAT\",\n",
    "    \"num_layers\":  1,\n",
    "    \"dropout\": 0.5,\n",
    "    \"learning_rate\": 0.01,\n",
    "    \"weight_decay\": 0.0005,\n",
    "    \"edge_dropout\": 0.00,\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from easydict import EasyDict as edict\n",
    "config = {\n",
    "    \"model_name\": \"res_unimp_large\",\n",
    "    \"num_layers\": 3,\n",
    "    \"hidden_size\": 128,\n",
    "    \"heads\": 2,\n",
    "    \"learning_rate\": 0.001,\n",
    "    \"dropout\": 0.33,\n",
    "    \"weight_decay\": 0.0005,\n",
    "    \"edge_dropout\": 0.32,\n",
    "    \"use_label_e\": True\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "config = edict(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 数据加载模块\n",
    "\n",
    "这里主要是用于读取数据集，包括读取图数据构图，以及训练集的划分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "\n",
    "Dataset = namedtuple(\"Dataset\", \n",
    "               [\"graph\", \"num_classes\", \"train_index\",\n",
    "                \"train_label\", \"valid_index\", \"valid_label\", \"test_index\"])\n",
    "\n",
    "def load_edges(num_nodes, self_loop=True, add_inverse_edge=True):\n",
    "    # 从数据中读取边\n",
    "    edges = pd.read_csv(\"work/edges.csv\", header=None, names=[\"src\", \"dst\"]).values\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "    if add_inverse_edge:\n",
    "        edges = np.vstack([edges, edges[:, ::-1]])\n",
    "\n",
    "    if self_loop:\n",
    "        src = np.arange(0, num_nodes)\n",
    "        dst = np.arange(0, num_nodes)\n",
    "        self_loop = np.vstack([src, dst]).T\n",
    "        edges = np.vstack([edges, self_loop])\n",
    "    \n",
    "    return edges\n",
    "\n",
    "def load():\n",
    "    # 从数据中读取点特征和边，以及数据划分\n",
    "    node_feat = np.load(\"work/feat.npy\")\n",
    "    #node_feat=feat1\n",
    "    num_nodes = node_feat.shape[0]\n",
    "    edges = load_edges(num_nodes=num_nodes, self_loop=True, add_inverse_edge=True)\n",
    "    graph = pgl.graph.Graph(num_nodes=num_nodes, edges=edges, node_feat={\"feat\": node_feat})\n",
    "    \n",
    "    indegree = graph.indegree()\n",
    "    norm = np.maximum(indegree.astype(\"float32\"), 1)\n",
    "    norm = np.power(norm, -0.5)\n",
    "    graph.node_feat[\"norm\"] = np.expand_dims(norm, -1)\n",
    "    \n",
    "    df = pd.read_csv(\"work/train.csv\")\n",
    "    # 打乱顺序\n",
    "    df.sample(frac=1.0) \n",
    "    node_index = df[\"nid\"].values\n",
    "    node_label = df[\"label\"].values\n",
    "    train_part = int(len(node_index) * 0.8)\n",
    "    train_index = node_index[:train_part]\n",
    "    train_label = node_label[:train_part]\n",
    "    valid_index = node_index[train_part:]\n",
    "    valid_label = node_label[train_part:]\n",
    "    test_index = pd.read_csv(\"work/test.csv\")[\"nid\"].values\n",
    "    dataset = Dataset(graph=graph, \n",
    "                    train_label=train_label,\n",
    "                    train_index=train_index,\n",
    "                    valid_index=valid_index,\n",
    "                    valid_label=valid_label,\n",
    "                    test_index=test_index, num_classes=35)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dataset = load()\n",
    "\n",
    "train_index = dataset.train_index\n",
    "train_label = np.reshape(dataset.train_label, [-1 , 1])\n",
    "train_index = np.expand_dims(train_index, -1)\n",
    "\n",
    "val_index = dataset.valid_index\n",
    "val_label = np.reshape(dataset.valid_label, [-1, 1])\n",
    "val_index = np.expand_dims(val_index, -1)\n",
    "\n",
    "test_index = dataset.test_index\n",
    "test_index = np.expand_dims(test_index, -1)\n",
    "test_label = np.zeros((len(test_index), 1), dtype=\"int64\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 组网模块\n",
    "\n",
    "这里是组网模块，目前已经提供了一些预定义的模型，包括**GCN**, **GAT**, **APPNP**等。可以通过简单的配置，设定模型的层数，hidden_size等。你也可以深入到model.py里面，去奇思妙想，写自己的图神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pgl\n",
    "import model\n",
    "import paddle.fluid as fluid\n",
    "import numpy as np\n",
    "import time\n",
    "from build_model import build_model\n",
    "\n",
    "# # 使用CPU\n",
    "#place = fluid.CPUPlace()\n",
    "\n",
    "# 使用GPU\n",
    "place = fluid.CUDAPlace(0)\n",
    "\n",
    "train_program = fluid.default_main_program()\n",
    "startup_program = fluid.default_startup_program()\n",
    "with fluid.program_guard(train_program, startup_program):\n",
    "    with fluid.unique_name.guard():\n",
    "        gw, loss, acc, pred = build_model(dataset,\n",
    "                            config=config,\n",
    "                            phase=\"train\",\n",
    "                            main_prog=train_program)\n",
    "\n",
    "test_program = fluid.Program()\n",
    "with fluid.program_guard(test_program, startup_program):\n",
    "    with fluid.unique_name.guard():\n",
    "        _gw, v_loss, v_acc, v_pred = build_model(dataset,\n",
    "            config=config,\n",
    "            phase=\"test\",\n",
    "            main_prog=test_program)\n",
    "\n",
    "\n",
    "test_program = test_program.clone(for_test=True)\n",
    "\n",
    "exe = fluid.Executor(place)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 开始训练过程\n",
    "\n",
    "图神经网络采用FullBatch的训练方式，每一步训练就会把所有整张图训练样本全部训练一遍。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "use_label_e = True\n",
    "label_rate = 0.66\n",
    "epoch = 4000\n",
    "exe.run(startup_program)\n",
    "max_val_acc = 0\n",
    "\n",
    "# 这里可以恢复训练\n",
    "pretrained = True\n",
    "if pretrained:\n",
    "    def name_filter(var):\n",
    "        res = var.name in os.listdir('./output')\n",
    "        return res\n",
    "    fluid.io.load_vars(exe, './output',predicate=name_filter)\n",
    "    max_val_acc = 0.63\n",
    "\n",
    "earlystop = 0\n",
    "# 将图数据变成 feed_dict 用于传入Paddle Excecutor\n",
    "feed_dict = gw.to_feed(dataset.graph)\n",
    "for epoch in range(epoch):\n",
    "    # Full Batch 训练\n",
    "    # 设定图上面那些节点要获取\n",
    "    # node_index: 未知label节点的nid    \n",
    "    # node_label: 未知label\n",
    "    # label_idx: 已知label节点的nid    \n",
    "    # label: 已知label\n",
    "    \n",
    "    if use_label_e:\n",
    "        # 在训练集中抽取部分数据，其Label已知，并可以输入网络训练\n",
    "        train_idx_temp = np.array(train_index, dtype=\"int64\")\n",
    "        train_lab_temp = np.array(train_label, dtype=\"int64\")\n",
    "        state = np.random.get_state()\n",
    "        np.random.shuffle(train_idx_temp)\n",
    "        np.random.set_state(state)\n",
    "        np.random.shuffle(train_lab_temp)\n",
    "\n",
    "        label_idx=train_idx_temp[:int(label_rate*len(train_idx_temp))]\n",
    "        unlabel_idx=train_idx_temp[int(label_rate*len(train_idx_temp)):]\n",
    "        label=train_lab_temp[:int(label_rate*len(train_idx_temp))]\n",
    "        unlabel=train_lab_temp[int(label_rate*len(train_idx_temp)):]\n",
    "\n",
    "        feed_dict[\"node_index\"] = unlabel_idx\n",
    "        feed_dict[\"node_label\"] = unlabel\n",
    "        feed_dict['label_idx']= label_idx\n",
    "        feed_dict['label']= label\n",
    "    else:\n",
    "        feed_dict[\"node_label\"] = np.array(train_label, dtype=\"int64\")\n",
    "        feed_dict[\"node_index\"] = np.array(train_index, dtype=\"int64\")\n",
    "        \n",
    "\n",
    "    train_loss, train_acc = exe.run(train_program,\n",
    "                                feed=feed_dict,\n",
    "                                fetch_list=[loss, acc],\n",
    "                                return_numpy=True)\n",
    "\n",
    "    # Full Batch 验证\n",
    "    # 设定图上面那些节点要获取\n",
    "    # node_index: 未知label节点的nid    \n",
    "    # node_label: 未知label\n",
    "    # label_idx: 已知label节点的nid    \n",
    "    # label: 已知label\n",
    "    \n",
    "    feed_dict[\"node_index\"] = np.array(val_index, dtype=\"int64\")\n",
    "    feed_dict[\"node_label\"] = np.array(val_label, dtype=\"int64\")\n",
    "    if use_label_e:\n",
    "        feed_dict['label_idx'] = np.array(train_index, dtype=\"int64\")\n",
    "        feed_dict['label'] = np.array(train_label, dtype=\"int64\")\n",
    "    val_loss, val_acc = exe.run(test_program,\n",
    "                            feed=feed_dict,\n",
    "                            fetch_list=[v_loss, v_acc],\n",
    "                            return_numpy=True)\n",
    "    print(\"Epoch\", epoch, \"Train Acc\", train_acc[0], \"Valid Acc\", val_acc[0],\"train loss\",train_loss[0],\"val loss\",val_loss[0])\n",
    "    \n",
    "    # 保存历史最优验证精度对应的模型\n",
    "    if val_acc[0] > max_val_acc:\n",
    "        max_val_acc = val_acc[0]\n",
    "        print(val_acc[0])\n",
    "        fluid.io.save_persistables(exe, './output', train_program)\n",
    "    \n",
    "    # 训练精度持续大于验证精度，结束训练\n",
    "    if train_acc[0] > val_acc[0]:\n",
    "        earlystop += 1\n",
    "        if earlystop == 40:\n",
    "            break\n",
    "    else:\n",
    "        earlystop = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 对测试集进行预测\n",
    "\n",
    "训练完成后，我们对测试集进行预测。预测的时候，由于不知道测试集合的标签，我们随意给一些测试label。最终我们获得测试数据的预测结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pretrained = True\r\n",
    "if pretrained:\r\n",
    "    def name_filter(var):\r\n",
    "        res = var.name in os.listdir('./output')\r\n",
    "        return res\r\n",
    "    fluid.io.load_vars(exe, './output',predicate=name_filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "feed_dict[\"node_index\"] = np.array(test_index, dtype=\"int64\")\n",
    "feed_dict[\"node_label\"] = np.array(test_label, dtype=\"int64\") #假标签\n",
    "test_prediction = exe.run(test_program,\n",
    "                            feed=feed_dict,\n",
    "                            fetch_list=[v_pred],\n",
    "                            return_numpy=True)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 生成提交文件\n",
    "\n",
    "最后一步，我们可以使用pandas轻松生成提交文件，最后下载 submission.csv 提交就好了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "submission = pd.DataFrame(data={\n",
    "                            \"nid\": test_index.reshape(-1),\n",
    "                            \"label\": test_prediction.reshape(-1)\n",
    "                        })\n",
    "submission.to_csv(\"submission.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def publicnum(num, d = 0):\r\n",
    "    dictnum = {}\r\n",
    "    for i in range(len(num)):\r\n",
    "        if num[i] in dictnum.keys():\r\n",
    "            dictnum[num[i]] += 1\r\n",
    "        else:\r\n",
    "            dictnum.setdefault(num[i], 1)\r\n",
    "    maxnum = 0\r\n",
    "    maxkey = 0\r\n",
    "    for k, v in dictnum.items():\r\n",
    "        if v > maxnum:\r\n",
    "            maxnum = v\r\n",
    "            maxkey = k\r\n",
    "    return maxkey\r\n",
    "\r\n",
    "df0=pd.read_csv(\"submission0.76136.csv\")\r\n",
    "df1=pd.read_csv(\"submission0.757822.csv\")\r\n",
    "df2=pd.read_csv(\"submission0.7583.csv\")\r\n",
    "df3=pd.read_csv(\"submission0.75758.csv\")\r\n",
    "df4=pd.read_csv(\"submission0.75921.csv\")\r\n",
    "df5=pd.read_csv(\"submission0.75782.csv\")\r\n",
    "df6=pd.read_csv(\"submission0.75956.csv\")\r\n",
    "df7=pd.read_csv(\"submission0.75801.csv\")\r\n",
    "df8=pd.read_csv(\"submission0.75884.csv\")\r\n",
    "#df9=pd.read_csv(\"submission9.csv\")\r\n",
    "#df10=pd.read_csv(\"submission10.csv\")\r\n",
    "\r\n",
    "\r\n",
    "\r\n",
    "\r\n",
    "nids=[]\r\n",
    "labels=[]\r\n",
    "\r\n",
    "for i in range(df4.shape[0]):\r\n",
    "    label_zs=[]\r\n",
    "    label_zs.append(df0.label[i])\r\n",
    "    label_zs.append(df1.label[i])\r\n",
    "    label_zs.append(df2.label[i])\r\n",
    "    label_zs.append(df3.label[i])\r\n",
    "    label_zs.append(df4.label[i])\r\n",
    "    label_zs.append(df5.label[i])\r\n",
    "    label_zs.append(df6.label[i])\r\n",
    "    label_zs.append(df7.label[i])\r\n",
    "    label_zs.append(df8.label[i])\r\n",
    "    #label_zs.append(df9.label[i])\r\n",
    "    #label_zs.append(df10.label[i])\r\n",
    "    lab=publicnum(label_zs, d = 0)\r\n",
    "    labels.append(lab)\r\n",
    "    nids.append(df4.nid[i])\r\n",
    "\r\n",
    "\r\n",
    "submission = pd.DataFrame(data={\r\n",
    "                            \"nid\": nids,\r\n",
    "                            \"label\": labels\r\n",
    "                        })\r\n",
    "submission.to_csv(\"submissiona.csv\", index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.8.4 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}