1809354.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 常规赛：遥感影像地块分割实战\n",
    "\n",
    ">比赛页面传送门：[ 常规赛：遥感影像地块分割](https://aistudio.baidu.com/aistudio/competition/detail/63)\n",
    "\n",
    ">欢迎参加 [飞桨领航团实战速成营](https://aistudio.baidu.com/aistudio/education/group/info/16606) \n",
    "\n",
    "## 赛题介绍\n",
    "\n",
    "本赛题由 2020 CCF BDCI 遥感影像地块分割 初赛赛题改编而来。遥感影像地块分割, 旨在对遥感影像进行像素级内容解析，对遥感影像中感兴趣的类别进行提取和分类，在城乡规划、防汛救灾等领域具有很高的实用价值，在工业界也受到了广泛关注。现有的遥感影像地块分割数据处理方法局限于特定的场景和特定的数据来源，且精度无法满足需求。因此在实际应用中，仍然大量依赖于人工处理，需要消耗大量的人力、物力、财力。本赛题旨在衡量遥感影像地块分割模型在多个类别（如建筑、道路、林地等）上的效果，利用人工智能技术，对多来源、多场景的异构遥感影像数据进行充分挖掘，打造高效、实用的算法，提高遥感影像的分析提取能力。 赛题任务 本赛题旨在对遥感影像进行像素级内容解析，并对遥感影像中感兴趣的类别进行提取和分类，以衡量遥感影像地块分割模型在多个类别（如建筑、道路、林地等）上的效果。\n",
    "\n",
    "## 数据说明\n",
    "本赛题提供了多个地区已脱敏的遥感影像数据，各参赛选手可以基于这些数据构建自己的地块分割模型。\n",
    "\n",
    "## 赛题任务\n",
    "本赛题旨在对遥感影像进行像素级内容解析，并对遥感影像中感兴趣的类别进行提取和分类，以衡量遥感影像地块分割模型在多个类别（如建筑、道路、林地等）上的效果。\n",
    "\n",
    "## 训练数据集\n",
    "样例图片及其标注如下图所示：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/fb7d813b0bb0449e9c49d5587d591af5e1e6fc5ba94544b6bbbf8ee3d19c4c25)\n",
    "\n",
    "训练数据集文件名称：train_and_label.zip\n",
    "包含2个子文件，分别为：训练数据集（原始图片）文件、训练数据集（标注图片）文件，详细介绍如下：\n",
    "\n",
    "训练数据集（原始图片）文件名称：img_train\n",
    "包含66,653张分辨率为2m/pixel，尺寸为256 * 256的JPG图片，每张图片的名称形如T000123.jpg。\n",
    "\n",
    "训练数据集（标注图片）文件名称：lab_train\n",
    "包含66,653张分辨率为2m/pixel，尺寸为256 * 256的PNG图片，每张图片的名称形如T000123.png。\n",
    "备注： 全部PNG图片共包括4种分类，像素值分别为0、1、2、3。此外，像素值255为未标注区域，表示对应区域的所属类别并不确定，在评测中也不会考虑这部分区域。\n",
    "\n",
    "测试数据集\n",
    "测试数据集文件名称：img_test.zip，详细介绍如下：\n",
    "包含4,609张分辨率为2m/pixel，尺寸为256 * 256的JPG图片，文件名称形如123.jpg。\n",
    "\n",
    "## 提交内容及格式\n",
    "* 以zip压缩包形式提交结果文件，文件命名为 result.zip；\n",
    "* zip压缩包中的图片格式必须为单通道PNG；\n",
    "* PNG文件数需要与测试数据集中的文件数相同，且zip压缩包文件名需要与测试数据集中的文件名一一对应；\n",
    "* 单通道PNG图片中的像素值必须介于0~3之间，像素值不能为255。如果存在未标注区域，评测系统会自动忽略对应区域的提交结果。\n",
    "\n",
    "## 提交示例\n",
    "提交文件命名为：result.zip，zip文件的组织方式如下所示：\n",
    "\n",
    ">主目录                                                                        \n",
    ">├── 1.png         #每个结果文件命名为：测试数据集图片名称+.png                      \n",
    ">├── 2.png                                                              \n",
    ">├── 3.png                                                    \n",
    ">├── ...                                                     \n",
    "    \n",
    "备注： 主目录中必须包含与测试数据集相同数目、名称相对应的单通道PNG图片，且每张单通道PNG图片中的像素值必须介于0~3之间，像素值不能为255。\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/e7ca4bdb7c594d928f900c817abc60c471c50b1fce7a4cc4b8ff360ecc721596)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 第一步 环境配置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pip install paddlex -i https://mirror.baidu.com/pypi/simple\r\n",
    "!pip install imgaug -i https://mirror.baidu.com/pypi/simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 设置使用0号GPU卡（如无GPU，执行此代码后仍然会使用CPU训练模型）\r\n",
    "import matplotlib\r\n",
    "import os\r\n",
    "import paddlex as pdx\r\n",
    "\r\n",
    "os.environ['CUDA_VISIBLE_DEVICES'] = '0'\r\n",
    "\r\n",
    "import numpy as np\r\n",
    "from paddlex.seg import transforms\r\n",
    "import imgaug.augmenters as iaa\r\n",
    "\r\n",
    "from tqdm import tqdm\r\n",
    "import cv2\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 第二步 解压数据集\n",
    "\n",
    "解压数据集并读取信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!unzip -q data/data77571/train_and_label.zip\r\n",
    "!unzip -q data/data77571/img_test.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total: 66652\n",
      "img_train/T135602.jpg\n",
      "lab_train/T135602.png\n",
      "('img_train/T135602.jpg', 'lab_train/T135602.png')\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\r\n",
    "\r\n",
    "datas = []\r\n",
    "image_base = 'img_train'   # 训练集原图路径\r\n",
    "annos_base = 'lab_train'   # 训练集标签路径\r\n",
    "\r\n",
    "# 读取原图文件名\r\n",
    "ids_ = [v.split('.')[0] for v in os.listdir(image_base)]\r\n",
    "\r\n",
    "# 将训练集的图像集和标签路径写入datas中\r\n",
    "for id_ in ids_:\r\n",
    "    img_pt0 = os.path.join(image_base, '{}.jpg'.format(id_))\r\n",
    "    img_pt1 = os.path.join(annos_base, '{}.png'.format(id_))\r\n",
    "    datas.append((img_pt0.replace('/home/aistudio', ''), img_pt1.replace('/home/aistudio', '')))\r\n",
    "    if os.path.exists(img_pt0) and os.path.exists(img_pt1):\r\n",
    "        pass\r\n",
    "    else:\r\n",
    "        raise \"path invalid!\"\r\n",
    "\r\n",
    "# 打印datas的长度和具体存储例子\r\n",
    "print('total:', len(datas))\r\n",
    "print(datas[0][0])\r\n",
    "print(datas[0][1])\r\n",
    "print(datas[0][:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: 59987\n",
      "valid: 6665\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\r\n",
    "\r\n",
    "# 四类标签，这里用处不大，比赛评测是以0、1、2、3类来对比评测的\r\n",
    "labels = ['0', '1', '2',  '3']\r\n",
    "\r\n",
    "# 将labels写入标签文件\r\n",
    "with open('labels.txt', 'w') as f:\r\n",
    "    for v in labels:\r\n",
    "        f.write(v+'\\n')\r\n",
    "\r\n",
    "# 随机打乱datas\r\n",
    "np.random.seed(5)\r\n",
    "np.random.shuffle(datas)\r\n",
    "\r\n",
    "# 验证集与训练集的划分，0.10表示10%为训练集，90%为训练集\r\n",
    "split_num = int(0.10*len(datas))\r\n",
    "\r\n",
    "# 划分训练集和验证集\r\n",
    "train_data = datas[:-split_num]\r\n",
    "valid_data = datas[-split_num:]\r\n",
    "\r\n",
    "# 写入训练集list\r\n",
    "with open('train_list.txt', 'w') as f:\r\n",
    "    for img, lbl in train_data:\r\n",
    "        f.write(img + ' ' + lbl + '\\n')\r\n",
    "\r\n",
    "# 写入验证集list\r\n",
    "with open('valid_list.txt', 'w') as f:\r\n",
    "    for img, lbl in valid_data:\r\n",
    "        f.write(img + ' ' + lbl + '\\n')\r\n",
    "\r\n",
    "# 打印训练集和测试集大小\r\n",
    "print('train:', len(train_data))\r\n",
    "print('valid:', len(valid_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from paddlex.seg import transforms\r\n",
    "import imgaug.augmenters as iaa\r\n",
    "\r\n",
    "# 定义训练和验证时的transforms\r\n",
    "train_transforms = transforms.Compose([\r\n",
    "    transforms.RandomHorizontalFlip(),\r\n",
    "    transforms.Resize(target_size=300),\r\n",
    "    transforms.RandomPaddingCrop(crop_size=256),\r\n",
    "    transforms.RandomBlur(prob=0.1),\r\n",
    "    transforms.RandomRotate(rotate_range=15),\r\n",
    "    # transforms.RandomDistort(brightness_range=0.5),\r\n",
    "    transforms.Normalize()\r\n",
    "])\r\n",
    "eval_transforms = transforms.Compose([\r\n",
    "    transforms.Resize(256),\r\n",
    "    transforms.Normalize()\r\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 读取语义 分割任务数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 17:07:17 [INFO]\t59987 samples in file train_list.txt\n",
      "2021-04-15 17:07:17 [INFO]\t6665 samples in file valid_list.txt\n"
     ]
    }
   ],
   "source": [
    "data_dir = './'\r\n",
    "\r\n",
    "# 定义训练和验证数据集\r\n",
    "train_dataset = pdx.datasets.SegDataset(\r\n",
    "    data_dir=data_dir,                # 数据集路径\r\n",
    "    file_list='train_list.txt',       # 训练集图片文件list路径\r\n",
    "    label_list='labels.txt',          # 训练集标签文件list路径\r\n",
    "    transforms=train_transforms,      # train_transforms\r\n",
    "    shuffle=True)                     # 数据集是否打乱\r\n",
    "    \r\n",
    "eval_dataset = pdx.datasets.SegDataset(\r\n",
    "    data_dir=data_dir,                # 数据集路径\r\n",
    "    file_list='valid_list.txt',       # 验证集图片文件list路径\r\n",
    "    label_list='labels.txt',          # 验证集标签文件list路径\r\n",
    "    transforms=eval_transforms)       # eval_transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 模型训练\n",
    "\n",
    "利用paddlex.seg.DeepLabv3p 构建 DeepLabv3p分割器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/math_op_patch.py:298: UserWarning: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlex/cv/nets/xception.py:316\n",
      "The behavior of expression A + B has been unified with elementwise_add(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_add(X, Y, axis=0) instead of A + B. This transitional warning will be dropped in the future.\n",
      "  op_type, op_type, EXPRESSION_MAP[method_name]))\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/math_op_patch.py:298: UserWarning: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlex/cv/nets/segmentation/model_utils/loss.py:74\n",
      "The behavior of expression A * B has been unified with elementwise_mul(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_mul(X, Y, axis=0) instead of A * B. This transitional warning will be dropped in the future.\n",
      "  op_type, op_type, EXPRESSION_MAP[method_name]))\n",
      "2021-04-15 17:10:59,848 - INFO - If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000040] in Optimizer will not take effect, and it will only be applied to other Parameters!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 17:11:03 [WARNING]\tPath of pretrain_weights('output/best_model') is not exists!\n",
      "2021-04-15 17:11:03 [WARNING]\tPretrain_weights will be forced to set as 'IMAGENET', if you don't want to use pretrain weights, set pretrain_weights=None.\n",
      "2021-04-15 17:11:03 [INFO]\tDownloading Xception65_deeplab_pretrained.tar from https://paddle-imagenet-models-name.bj.bcebos.com/Xception65_deeplab_pretrained.tar\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 157790/157790 [00:03<00:00, 51079.90KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 17:11:07 [INFO]\tDecompressing output/pretrain/Xception65_deeplab_pretrained.tar...\n",
      "2021-04-15 17:11:12 [INFO]\tLoad pretrain weights from output/pretrain/Xception65_deeplab_pretrained.\n",
      "2021-04-15 17:11:13 [INFO]\tThere are 660 varaibles in output/pretrain/Xception65_deeplab_pretrained are loaded.\n",
      "2021-04-15 17:16:33 [INFO]\t[TRAIN] Epoch=1/50, Step=1000/3749, loss=1.057388, lr=0.00199, time_each_step=0.32s, eta=18:10:55\n",
      "2021-04-15 17:21:49 [INFO]\t[TRAIN] Epoch=1/50, Step=2000/3749, loss=1.155253, lr=0.001981, time_each_step=0.31s, eta=17:59:57\n",
      "2021-04-15 17:27:04 [INFO]\t[TRAIN] Epoch=1/50, Step=3000/3749, loss=0.751901, lr=0.001971, time_each_step=0.31s, eta=17:54:7\n",
      "2021-04-15 17:31:00 [INFO]\t[TRAIN] Epoch 1 finished, loss=0.917681, lr=0.001982 .\n",
      "2021-04-15 17:31:00 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/417 [00:00<?, ?it/s]share_vars_from is set, scope is ignored.\n",
      "100%|██████████| 417/417 [00:46<00:00,  8.90it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 17:31:47 [INFO]\t[EVAL] Finished, Epoch=1, miou=0.475065, category_iou=[0.46248874 0.6842755  0.46208414 0.29141169], oacc=0.669852, category_acc=[0.52720975 0.8398549  0.6997477  0.66774741], kappa=0.538276, category_F1-score=[0.6324681  0.81254581 0.63208967 0.45130719] .\n",
      "2021-04-15 17:31:51 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 17:31:54 [INFO]\tModel saved in output/epoch_1.\n",
      "2021-04-15 17:31:54 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_1, miou=0.4750650181287811\n",
      "2021-04-15 17:33:19 [INFO]\t[TRAIN] Epoch=2/50, Step=251/3749, loss=0.653578, lr=0.001962, time_each_step=0.32s, eta=16:52:49\n",
      "2021-04-15 17:38:35 [INFO]\t[TRAIN] Epoch=2/50, Step=1251/3749, loss=0.791433, lr=0.001952, time_each_step=0.32s, eta=16:47:29\n",
      "2021-04-15 17:43:51 [INFO]\t[TRAIN] Epoch=2/50, Step=2251/3749, loss=0.929406, lr=0.001942, time_each_step=0.32s, eta=16:42:10\n",
      "2021-04-15 17:49:06 [INFO]\t[TRAIN] Epoch=2/50, Step=3251/3749, loss=0.517848, lr=0.001933, time_each_step=0.32s, eta=16:36:54\n",
      "2021-04-15 17:51:43 [INFO]\t[TRAIN] Epoch 2 finished, loss=0.791017, lr=0.001946 .\n",
      "2021-04-15 17:51:43 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:48<00:00,  8.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 17:52:32 [INFO]\t[EVAL] Finished, Epoch=2, miou=0.514808, category_iou=[0.50556693 0.71899005 0.47273095 0.36194291], oacc=0.699331, category_acc=[0.60145341 0.88222203 0.66220885 0.58345688], kappa=0.584945, category_F1-score=[0.67159675 0.83652614 0.6419787  0.53150966] .\n",
      "2021-04-15 17:52:35 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 17:52:39 [INFO]\tModel saved in output/epoch_2.\n",
      "2021-04-15 17:52:39 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_2, miou=0.5148077108152598\n",
      "2021-04-15 17:55:25 [INFO]\t[TRAIN] Epoch=3/50, Step=502/3749, loss=1.021577, lr=0.001923, time_each_step=0.31s, eta=16:32:45\n",
      "2021-04-15 18:00:41 [INFO]\t[TRAIN] Epoch=3/50, Step=1502/3749, loss=0.521504, lr=0.001913, time_each_step=0.32s, eta=16:27:38\n",
      "2021-04-15 18:05:57 [INFO]\t[TRAIN] Epoch=3/50, Step=2502/3749, loss=0.50451, lr=0.001904, time_each_step=0.32s, eta=16:22:23\n",
      "2021-04-15 18:11:13 [INFO]\t[TRAIN] Epoch=3/50, Step=3502/3749, loss=0.537592, lr=0.001894, time_each_step=0.31s, eta=16:17:5\n",
      "2021-04-15 18:12:31 [INFO]\t[TRAIN] Epoch 3 finished, loss=0.735221, lr=0.00191 .\n",
      "2021-04-15 18:12:31 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.46it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 18:13:15 [INFO]\t[EVAL] Finished, Epoch=3, miou=0.529088, category_iou=[0.53289639 0.73784724 0.50404411 0.34156452], oacc=0.721408, category_acc=[0.64743652 0.82151638 0.64967573 0.71270663], kappa=0.611551, category_F1-score=[0.69528038 0.84915086 0.67025177 0.50920327] .\n",
      "2021-04-15 18:13:19 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 18:13:23 [INFO]\tModel saved in output/epoch_3.\n",
      "2021-04-15 18:13:23 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_3, miou=0.5290880663311684\n",
      "2021-04-15 18:17:25 [INFO]\t[TRAIN] Epoch=4/50, Step=753/3749, loss=0.464619, lr=0.001884, time_each_step=0.32s, eta=16:10:27\n",
      "2021-04-15 18:22:41 [INFO]\t[TRAIN] Epoch=4/50, Step=1753/3749, loss=0.8133, lr=0.001875, time_each_step=0.32s, eta=16:4:58\n",
      "2021-04-15 18:27:57 [INFO]\t[TRAIN] Epoch=4/50, Step=2753/3749, loss=0.797517, lr=0.001865, time_each_step=0.31s, eta=15:59:38\n",
      "2021-04-15 18:33:11 [INFO]\t[TRAIN] Epoch 4 finished, loss=0.692151, lr=0.001874 .\n",
      "2021-04-15 18:33:11 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:43<00:00,  9.55it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 18:33:54 [INFO]\t[EVAL] Finished, Epoch=4, miou=0.533264, category_iou=[0.52394751 0.72766556 0.50590312 0.37554076], oacc=0.717281, category_acc=[0.63018719 0.8381934  0.73790028 0.60571555], kappa=0.606895, category_F1-score=[0.68761884 0.84236854 0.67189332 0.54602636] .\n",
      "2021-04-15 18:33:58 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 18:34:02 [INFO]\tModel saved in output/epoch_4.\n",
      "2021-04-15 18:34:02 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_4, miou=0.5332642398406447\n",
      "2021-04-15 18:34:08 [INFO]\t[TRAIN] Epoch=5/50, Step=4/3749, loss=0.615469, lr=0.001855, time_each_step=0.55s, eta=16:5:5\n",
      "2021-04-15 18:39:24 [INFO]\t[TRAIN] Epoch=5/50, Step=1004/3749, loss=0.573621, lr=0.001846, time_each_step=0.32s, eta=15:44:58\n",
      "2021-04-15 18:44:40 [INFO]\t[TRAIN] Epoch=5/50, Step=2004/3749, loss=1.159075, lr=0.001836, time_each_step=0.31s, eta=15:39:38\n",
      "2021-04-15 18:49:55 [INFO]\t[TRAIN] Epoch=5/50, Step=3004/3749, loss=0.348223, lr=0.001826, time_each_step=0.32s, eta=15:34:23\n",
      "2021-04-15 18:53:50 [INFO]\t[TRAIN] Epoch 5 finished, loss=0.655801, lr=0.001837 .\n",
      "2021-04-15 18:53:50 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:47<00:00,  8.82it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 18:54:38 [INFO]\t[EVAL] Finished, Epoch=5, miou=0.540043, category_iou=[0.51146194 0.7421405  0.51064848 0.39592062], oacc=0.720589, category_acc=[0.68322494 0.85241905 0.62844392 0.6061271 ], kappa=0.615705, category_F1-score=[0.67677779 0.85198696 0.67606526 0.56725378] .\n",
      "2021-04-15 18:54:41 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 18:54:45 [INFO]\tModel saved in output/epoch_5.\n",
      "2021-04-15 18:54:45 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_5, miou=0.5400428836272727\n",
      "2021-04-15 18:56:14 [INFO]\t[TRAIN] Epoch=6/50, Step=255/3749, loss=0.853975, lr=0.001817, time_each_step=0.31s, eta=15:30:29\n",
      "2021-04-15 19:01:30 [INFO]\t[TRAIN] Epoch=6/50, Step=1255/3749, loss=0.698838, lr=0.001807, time_each_step=0.32s, eta=15:25:19\n",
      "2021-04-15 19:06:45 [INFO]\t[TRAIN] Epoch=6/50, Step=2255/3749, loss=0.93899, lr=0.001797, time_each_step=0.32s, eta=15:20:9\n",
      "2021-04-15 19:12:00 [INFO]\t[TRAIN] Epoch=6/50, Step=3255/3749, loss=0.566521, lr=0.001787, time_each_step=0.32s, eta=15:14:50\n",
      "2021-04-15 19:14:36 [INFO]\t[TRAIN] Epoch 6 finished, loss=0.628796, lr=0.001801 .\n",
      "2021-04-15 19:14:36 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.31it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 19:15:21 [INFO]\t[EVAL] Finished, Epoch=6, miou=0.54146, category_iou=[0.50611321 0.74738864 0.53172817 0.38060985], oacc=0.72042, category_acc=[0.68564421 0.84441871 0.68968752 0.55237451], kappa=0.61468, category_F1-score=[0.67207857 0.85543493 0.69428529 0.55136481] .\n",
      "2021-04-15 19:15:25 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 19:15:29 [INFO]\tModel saved in output/epoch_6.\n",
      "2021-04-15 19:15:29 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_6, miou=0.5414599639535875\n",
      "2021-04-15 19:18:14 [INFO]\t[TRAIN] Epoch=7/50, Step=506/3749, loss=0.593041, lr=0.001778, time_each_step=0.31s, eta=15:9:26\n",
      "2021-04-15 19:23:30 [INFO]\t[TRAIN] Epoch=7/50, Step=1506/3749, loss=0.921098, lr=0.001768, time_each_step=0.31s, eta=15:4:12\n",
      "2021-04-15 19:28:45 [INFO]\t[TRAIN] Epoch=7/50, Step=2506/3749, loss=0.646682, lr=0.001758, time_each_step=0.32s, eta=14:59:3\n",
      "2021-04-15 19:34:01 [INFO]\t[TRAIN] Epoch=7/50, Step=3506/3749, loss=0.679944, lr=0.001749, time_each_step=0.32s, eta=14:53:47\n",
      "2021-04-15 19:35:18 [INFO]\t[TRAIN] Epoch 7 finished, loss=0.598099, lr=0.001764 .\n",
      "2021-04-15 19:35:18 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 19:36:03 [INFO]\t[EVAL] Finished, Epoch=7, miou=0.555209, category_iou=[0.55211509 0.73188883 0.53782941 0.39900188], oacc=0.735052, category_acc=[0.69294803 0.8374801  0.65584585 0.66708355], kappa=0.633869, category_F1-score=[0.71143576 0.84519147 0.69946563 0.57040936] .\n",
      "2021-04-15 19:36:07 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 19:36:11 [INFO]\tModel saved in output/epoch_7.\n",
      "2021-04-15 19:36:11 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_7, miou=0.5552088031315799\n",
      "2021-04-15 19:40:14 [INFO]\t[TRAIN] Epoch=8/50, Step=757/3749, loss=0.712736, lr=0.001739, time_each_step=0.31s, eta=14:45:47\n",
      "2021-04-15 19:45:30 [INFO]\t[TRAIN] Epoch=8/50, Step=1757/3749, loss=0.698226, lr=0.001729, time_each_step=0.32s, eta=14:40:39\n",
      "2021-04-15 19:50:45 [INFO]\t[TRAIN] Epoch=8/50, Step=2757/3749, loss=0.779395, lr=0.001719, time_each_step=0.32s, eta=14:35:27\n",
      "2021-04-15 19:55:59 [INFO]\t[TRAIN] Epoch 8 finished, loss=0.576587, lr=0.001728 .\n",
      "2021-04-15 19:55:59 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 19:56:45 [INFO]\t[EVAL] Finished, Epoch=8, miou=0.526046, category_iou=[0.47049723 0.73356584 0.5152857  0.38483614], oacc=0.706956, category_acc=[0.73695537 0.81214428 0.68742978 0.50758101], kappa=0.597083, category_F1-score=[0.63991584 0.8463086  0.68011689 0.55578581] .\n",
      "2021-04-15 19:56:49 [INFO]\tModel saved in output/epoch_8.\n",
      "2021-04-15 19:56:49 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_7, miou=0.5552088031315799\n",
      "2021-04-15 19:57:05 [INFO]\t[TRAIN] Epoch=9/50, Step=8/3749, loss=0.301834, lr=0.001709, time_each_step=0.99s, eta=15:8:12\n",
      "2021-04-15 20:02:21 [INFO]\t[TRAIN] Epoch=9/50, Step=1008/3749, loss=0.778836, lr=0.0017, time_each_step=0.32s, eta=14:21:1\n",
      "2021-04-15 20:07:37 [INFO]\t[TRAIN] Epoch=9/50, Step=2008/3749, loss=0.871551, lr=0.00169, time_each_step=0.32s, eta=14:15:41\n",
      "2021-04-15 20:12:53 [INFO]\t[TRAIN] Epoch=9/50, Step=3008/3749, loss=0.424036, lr=0.00168, time_each_step=0.31s, eta=14:10:22\n",
      "2021-04-15 20:16:46 [INFO]\t[TRAIN] Epoch 9 finished, loss=0.549261, lr=0.001691 .\n",
      "2021-04-15 20:16:46 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.18it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 20:17:32 [INFO]\t[EVAL] Finished, Epoch=9, miou=0.558869, category_iou=[0.54379337 0.73191814 0.55345871 0.40630484], oacc=0.732847, category_acc=[0.70736477 0.84723278 0.70808291 0.57493641], kappa=0.632345, category_F1-score=[0.70448984 0.84521101 0.71255027 0.57783323] .\n",
      "2021-04-15 20:17:36 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 20:17:39 [INFO]\tModel saved in output/epoch_9.\n",
      "2021-04-15 20:17:39 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_9, miou=0.5588687657289196\n",
      "2021-04-15 20:19:05 [INFO]\t[TRAIN] Epoch=10/50, Step=259/3749, loss=0.408433, lr=0.00167, time_each_step=0.31s, eta=14:12:59\n",
      "2021-04-15 20:24:21 [INFO]\t[TRAIN] Epoch=10/50, Step=1259/3749, loss=0.728883, lr=0.001661, time_each_step=0.31s, eta=14:7:43\n",
      "2021-04-15 20:29:36 [INFO]\t[TRAIN] Epoch=10/50, Step=2259/3749, loss=0.662366, lr=0.001651, time_each_step=0.31s, eta=14:2:29\n",
      "2021-04-15 20:34:53 [INFO]\t[TRAIN] Epoch=10/50, Step=3259/3749, loss=0.465133, lr=0.001641, time_each_step=0.32s, eta=13:57:16\n",
      "2021-04-15 20:37:27 [INFO]\t[TRAIN] Epoch 10 finished, loss=0.523019, lr=0.001655 .\n",
      "2021-04-15 20:37:27 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.45it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 20:38:12 [INFO]\t[EVAL] Finished, Epoch=10, miou=0.561193, category_iou=[0.55951992 0.72705254 0.53404501 0.42415267], oacc=0.739739, category_acc=[0.69535206 0.81107156 0.746057   0.64463218], kappa=0.63796, category_F1-score=[0.71755406 0.84195764 0.69625729 0.59565619] .\n",
      "2021-04-15 20:38:16 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 20:38:20 [INFO]\tModel saved in output/epoch_10.\n",
      "2021-04-15 20:38:20 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_10, miou=0.5611925362683929\n",
      "2021-04-15 20:41:07 [INFO]\t[TRAIN] Epoch=11/50, Step=510/3749, loss=0.881775, lr=0.001631, time_each_step=0.32s, eta=13:44:12\n",
      "2021-04-15 20:46:22 [INFO]\t[TRAIN] Epoch=11/50, Step=1510/3749, loss=0.232217, lr=0.001621, time_each_step=0.31s, eta=13:38:44\n",
      "2021-04-15 20:51:38 [INFO]\t[TRAIN] Epoch=11/50, Step=2510/3749, loss=0.519772, lr=0.001611, time_each_step=0.32s, eta=13:33:36\n",
      "2021-04-15 20:56:54 [INFO]\t[TRAIN] Epoch=11/50, Step=3510/3749, loss=0.318015, lr=0.001602, time_each_step=0.31s, eta=13:28:19\n",
      "2021-04-15 20:58:09 [INFO]\t[TRAIN] Epoch 11 finished, loss=0.504464, lr=0.001618 .\n",
      "2021-04-15 20:58:09 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:46<00:00,  9.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 20:58:56 [INFO]\t[EVAL] Finished, Epoch=11, miou=0.543, category_iou=[0.48169541 0.74534687 0.548028   0.39692921], oacc=0.715898, category_acc=[0.73862562 0.84631274 0.73162816 0.49401487], kappa=0.611357, category_F1-score=[0.65019491 0.85409597 0.7080337  0.56828823] .\n",
      "2021-04-15 20:58:59 [INFO]\tModel saved in output/epoch_11.\n",
      "2021-04-15 20:58:59 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_10, miou=0.5611925362683929\n",
      "2021-04-15 21:03:05 [INFO]\t[TRAIN] Epoch=12/50, Step=761/3749, loss=0.262578, lr=0.001592, time_each_step=0.31s, eta=13:21:36\n",
      "2021-04-15 21:08:21 [INFO]\t[TRAIN] Epoch=12/50, Step=1761/3749, loss=0.258287, lr=0.001582, time_each_step=0.31s, eta=13:16:28\n",
      "2021-04-15 21:13:37 [INFO]\t[TRAIN] Epoch=12/50, Step=2761/3749, loss=0.459281, lr=0.001572, time_each_step=0.32s, eta=13:11:15\n",
      "2021-04-15 21:18:50 [INFO]\t[TRAIN] Epoch 12 finished, loss=0.475472, lr=0.001581 .\n",
      "2021-04-15 21:18:50 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:46<00:00,  8.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 21:19:36 [INFO]\t[EVAL] Finished, Epoch=12, miou=0.559204, category_iou=[0.53982628 0.72839688 0.56361157 0.40498113], oacc=0.734465, category_acc=[0.75140283 0.80488572 0.73580776 0.56582387], kappa=0.632945, category_F1-score=[0.70115219 0.84285836 0.72090995 0.57649334] .\n",
      "2021-04-15 21:19:40 [INFO]\tModel saved in output/epoch_12.\n",
      "2021-04-15 21:19:40 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_10, miou=0.5611925362683929\n",
      "2021-04-15 21:19:59 [INFO]\t[TRAIN] Epoch=13/50, Step=12/3749, loss=0.449826, lr=0.001562, time_each_step=1.05s, eta=13:51:34\n",
      "2021-04-15 21:25:14 [INFO]\t[TRAIN] Epoch=13/50, Step=1012/3749, loss=0.481544, lr=0.001552, time_each_step=0.32s, eta=13:0:24\n",
      "2021-04-15 21:30:30 [INFO]\t[TRAIN] Epoch=13/50, Step=2012/3749, loss=0.736421, lr=0.001542, time_each_step=0.31s, eta=12:54:59\n",
      "2021-04-15 21:35:45 [INFO]\t[TRAIN] Epoch=13/50, Step=3012/3749, loss=1.233038, lr=0.001533, time_each_step=0.32s, eta=12:49:47\n",
      "2021-04-15 21:39:38 [INFO]\t[TRAIN] Epoch 13 finished, loss=0.450517, lr=0.001544 .\n",
      "2021-04-15 21:39:38 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 21:40:22 [INFO]\t[EVAL] Finished, Epoch=13, miou=0.568896, category_iou=[0.54613492 0.75374235 0.54621073 0.42949755], oacc=0.74528, category_acc=[0.75035113 0.83316347 0.64529753 0.65378701], kappa=0.648995, category_F1-score=[0.70645183 0.85958163 0.70651525 0.60090701] .\n",
      "2021-04-15 21:40:26 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 21:40:30 [INFO]\tModel saved in output/epoch_13.\n",
      "2021-04-15 21:40:30 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_13, miou=0.5688963884734536\n",
      "2021-04-15 21:41:57 [INFO]\t[TRAIN] Epoch=14/50, Step=263/3749, loss=0.33929, lr=0.001523, time_each_step=0.31s, eta=12:48:58\n",
      "2021-04-15 21:47:13 [INFO]\t[TRAIN] Epoch=14/50, Step=1263/3749, loss=0.874373, lr=0.001513, time_each_step=0.32s, eta=12:44:0\n",
      "2021-04-15 21:52:29 [INFO]\t[TRAIN] Epoch=14/50, Step=2263/3749, loss=0.472211, lr=0.001503, time_each_step=0.31s, eta=12:38:33\n",
      "2021-04-15 21:57:45 [INFO]\t[TRAIN] Epoch=14/50, Step=3263/3749, loss=0.822065, lr=0.001493, time_each_step=0.32s, eta=12:33:24\n",
      "2021-04-15 22:00:18 [INFO]\t[TRAIN] Epoch 14 finished, loss=0.426007, lr=0.001507 .\n",
      "2021-04-15 22:00:18 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:46<00:00,  9.04it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 22:01:05 [INFO]\t[EVAL] Finished, Epoch=14, miou=0.586643, category_iou=[0.57216771 0.74725973 0.58054084 0.44660218], oacc=0.756491, category_acc=[0.75014442 0.83225222 0.71721331 0.63984959], kappa=0.663796, category_F1-score=[0.72787109 0.85535049 0.73461036 0.61744989] .\n",
      "2021-04-15 22:01:08 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 22:01:12 [INFO]\tModel saved in output/epoch_14.\n",
      "2021-04-15 22:01:12 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_14, miou=0.586642613579397\n",
      "2021-04-15 22:03:58 [INFO]\t[TRAIN] Epoch=15/50, Step=514/3749, loss=0.299526, lr=0.001483, time_each_step=0.31s, eta=12:22:15\n",
      "2021-04-15 22:09:14 [INFO]\t[TRAIN] Epoch=15/50, Step=1514/3749, loss=0.595826, lr=0.001473, time_each_step=0.31s, eta=12:16:57\n",
      "2021-04-15 22:14:29 [INFO]\t[TRAIN] Epoch=15/50, Step=2514/3749, loss=0.637451, lr=0.001463, time_each_step=0.32s, eta=12:11:49\n",
      "2021-04-15 22:19:45 [INFO]\t[TRAIN] Epoch=15/50, Step=3514/3749, loss=0.302955, lr=0.001453, time_each_step=0.31s, eta=12:6:33\n",
      "2021-04-15 22:20:59 [INFO]\t[TRAIN] Epoch 15 finished, loss=0.4016, lr=0.001469 .\n",
      "2021-04-15 22:20:59 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 22:21:44 [INFO]\t[EVAL] Finished, Epoch=15, miou=0.556386, category_iou=[0.51409226 0.73517705 0.56235837 0.41391549], oacc=0.726798, category_acc=[0.71370756 0.86041793 0.71358623 0.5396624 ], kappa=0.626109, category_F1-score=[0.67907653 0.84737987 0.71988396 0.5854883 ] .\n",
      "2021-04-15 22:21:48 [INFO]\tModel saved in output/epoch_15.\n",
      "2021-04-15 22:21:48 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_14, miou=0.586642613579397\n",
      "2021-04-15 22:25:54 [INFO]\t[TRAIN] Epoch=16/50, Step=765/3749, loss=0.528675, lr=0.001443, time_each_step=0.32s, eta=11:57:7\n",
      "2021-04-15 22:31:09 [INFO]\t[TRAIN] Epoch=16/50, Step=1765/3749, loss=0.533738, lr=0.001433, time_each_step=0.31s, eta=11:51:47\n",
      "2021-04-15 22:36:25 [INFO]\t[TRAIN] Epoch=16/50, Step=2765/3749, loss=0.141581, lr=0.001423, time_each_step=0.31s, eta=11:46:30\n",
      "2021-04-15 22:41:36 [INFO]\t[TRAIN] Epoch 16 finished, loss=0.388246, lr=0.001432 .\n",
      "2021-04-15 22:41:36 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.32it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 22:42:20 [INFO]\t[EVAL] Finished, Epoch=16, miou=0.579714, category_iou=[0.56170715 0.75367526 0.56636871 0.43710626], oacc=0.750899, category_acc=[0.75911127 0.83668127 0.72140362 0.59642969], kappa=0.656633, category_F1-score=[0.71935017 0.859538   0.72316142 0.60831446] .\n",
      "2021-04-15 22:42:24 [INFO]\tModel saved in output/epoch_16.\n",
      "2021-04-15 22:42:24 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_14, miou=0.586642613579397\n",
      "2021-04-15 22:42:34 [INFO]\t[TRAIN] Epoch=17/50, Step=16/3749, loss=0.651012, lr=0.001413, time_each_step=0.53s, eta=11:53:28\n",
      "2021-04-15 22:47:49 [INFO]\t[TRAIN] Epoch=17/50, Step=1016/3749, loss=0.32751, lr=0.001403, time_each_step=0.32s, eta=11:35:7\n",
      "2021-04-15 22:53:05 [INFO]\t[TRAIN] Epoch=17/50, Step=2016/3749, loss=0.652866, lr=0.001393, time_each_step=0.31s, eta=11:29:42\n",
      "2021-04-15 22:58:20 [INFO]\t[TRAIN] Epoch=17/50, Step=3016/3749, loss=0.355241, lr=0.001383, time_each_step=0.31s, eta=11:24:29\n",
      "2021-04-15 23:02:12 [INFO]\t[TRAIN] Epoch 17 finished, loss=0.369997, lr=0.001395 .\n",
      "2021-04-15 23:02:12 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 23:02:57 [INFO]\t[EVAL] Finished, Epoch=17, miou=0.552762, category_iou=[0.52391065 0.73496767 0.53637959 0.41578969], oacc=0.732655, category_acc=[0.72942487 0.80997466 0.6524455  0.64288679], kappa=0.630505, category_F1-score=[0.68758709 0.84724077 0.69823837 0.58736081] .\n",
      "2021-04-15 23:03:00 [INFO]\tModel saved in output/epoch_17.\n",
      "2021-04-15 23:03:00 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_14, miou=0.586642613579397\n",
      "2021-04-15 23:04:29 [INFO]\t[TRAIN] Epoch=18/50, Step=267/3749, loss=0.308367, lr=0.001373, time_each_step=0.31s, eta=11:18:10\n",
      "2021-04-15 23:09:44 [INFO]\t[TRAIN] Epoch=18/50, Step=1267/3749, loss=0.650298, lr=0.001363, time_each_step=0.32s, eta=11:13:10\n",
      "2021-04-15 23:14:59 [INFO]\t[TRAIN] Epoch=18/50, Step=2267/3749, loss=0.483824, lr=0.001353, time_each_step=0.32s, eta=11:7:53\n",
      "2021-04-15 23:20:16 [INFO]\t[TRAIN] Epoch=18/50, Step=3267/3749, loss=0.369007, lr=0.001343, time_each_step=0.32s, eta=11:2:40\n",
      "2021-04-15 23:22:48 [INFO]\t[TRAIN] Epoch 18 finished, loss=0.354996, lr=0.001357 .\n",
      "2021-04-15 23:22:48 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 23:23:34 [INFO]\t[EVAL] Finished, Epoch=18, miou=0.592883, category_iou=[0.58815729 0.76079058 0.57700873 0.44557485], oacc=0.761964, category_acc=[0.74629794 0.8502263  0.72739997 0.63367368], kappa=0.671425, category_F1-score=[0.74067889 0.86414658 0.7317762  0.61646735] .\n",
      "2021-04-15 23:23:38 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-15 23:23:41 [INFO]\tModel saved in output/epoch_18.\n",
      "2021-04-15 23:23:41 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_18, miou=0.5928828616272388\n",
      "2021-04-15 23:26:30 [INFO]\t[TRAIN] Epoch=19/50, Step=518/3749, loss=0.324869, lr=0.001333, time_each_step=0.32s, eta=10:58:58\n",
      "2021-04-15 23:31:50 [INFO]\t[TRAIN] Epoch=19/50, Step=1518/3749, loss=0.621572, lr=0.001323, time_each_step=0.32s, eta=10:53:42\n",
      "2021-04-15 23:37:09 [INFO]\t[TRAIN] Epoch=19/50, Step=2518/3749, loss=0.235591, lr=0.001313, time_each_step=0.32s, eta=10:48:25\n",
      "2021-04-15 23:42:29 [INFO]\t[TRAIN] Epoch=19/50, Step=3518/3749, loss=0.122427, lr=0.001303, time_each_step=0.33s, eta=10:43:5\n",
      "2021-04-15 23:43:42 [INFO]\t[TRAIN] Epoch 19 finished, loss=0.330397, lr=0.00132 .\n",
      "2021-04-15 23:43:42 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-15 23:44:28 [INFO]\t[EVAL] Finished, Epoch=19, miou=0.580336, category_iou=[0.57236322 0.73927824 0.5644361  0.44526594], oacc=0.748339, category_acc=[0.74572172 0.86016838 0.72641837 0.5808767 ], kappa=0.654716, category_F1-score=[0.72802927 0.85009773 0.72158409 0.61617164] .\n",
      "2021-04-15 23:44:32 [INFO]\tModel saved in output/epoch_19.\n",
      "2021-04-15 23:44:32 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_18, miou=0.5928828616272388\n",
      "2021-04-15 23:48:41 [INFO]\t[TRAIN] Epoch=20/50, Step=769/3749, loss=0.481703, lr=0.001293, time_each_step=0.32s, eta=10:42:5\n",
      "2021-04-15 23:53:59 [INFO]\t[TRAIN] Epoch=20/50, Step=1769/3749, loss=0.333928, lr=0.001283, time_each_step=0.32s, eta=10:36:36\n",
      "2021-04-15 23:59:18 [INFO]\t[TRAIN] Epoch=20/50, Step=2769/3749, loss=0.323097, lr=0.001273, time_each_step=0.32s, eta=10:31:24\n",
      "2021-04-16 00:04:31 [INFO]\t[TRAIN] Epoch 20 finished, loss=0.307356, lr=0.001282 .\n",
      "2021-04-16 00:04:31 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.27it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 00:05:16 [INFO]\t[EVAL] Finished, Epoch=20, miou=0.599091, category_iou=[0.60218976 0.75760249 0.57760952 0.45896138], oacc=0.766246, category_acc=[0.77324728 0.84050437 0.75327274 0.61951557], kappa=0.677233, category_F1-score=[0.75170841 0.86208627 0.73225917 0.62916179] .\n",
      "2021-04-16 00:05:20 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 00:05:24 [INFO]\tModel saved in output/epoch_20.\n",
      "2021-04-16 00:05:24 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 00:05:35 [INFO]\t[TRAIN] Epoch=21/50, Step=20/3749, loss=0.294947, lr=0.001263, time_each_step=0.55s, eta=10:40:26\n",
      "2021-04-16 00:10:54 [INFO]\t[TRAIN] Epoch=21/50, Step=1020/3749, loss=0.169111, lr=0.001253, time_each_step=0.32s, eta=10:20:37\n",
      "2021-04-16 00:16:14 [INFO]\t[TRAIN] Epoch=21/50, Step=2020/3749, loss=0.284607, lr=0.001242, time_each_step=0.32s, eta=10:15:16\n",
      "2021-04-16 00:21:34 [INFO]\t[TRAIN] Epoch=21/50, Step=3020/3749, loss=0.3757, lr=0.001232, time_each_step=0.33s, eta=10:10:3\n",
      "2021-04-16 00:25:27 [INFO]\t[TRAIN] Epoch 21 finished, loss=0.289363, lr=0.001244 .\n",
      "2021-04-16 00:25:27 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.28it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 00:26:12 [INFO]\t[EVAL] Finished, Epoch=21, miou=0.582431, category_iou=[0.57612003 0.75072095 0.56369062 0.43919254], oacc=0.754938, category_acc=[0.79032001 0.82430627 0.684881   0.62647663], kappa=0.66204, category_F1-score=[0.73106111 0.85761349 0.72097461 0.61033187] .\n",
      "2021-04-16 00:26:16 [INFO]\tModel saved in output/epoch_21.\n",
      "2021-04-16 00:26:16 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 00:27:51 [INFO]\t[TRAIN] Epoch=22/50, Step=271/3749, loss=0.693003, lr=0.001222, time_each_step=0.32s, eta=10:3:34\n",
      "2021-04-16 00:33:10 [INFO]\t[TRAIN] Epoch=22/50, Step=1271/3749, loss=0.284107, lr=0.001212, time_each_step=0.32s, eta=9:58:32\n",
      "2021-04-16 00:38:30 [INFO]\t[TRAIN] Epoch=22/50, Step=2271/3749, loss=0.321328, lr=0.001202, time_each_step=0.32s, eta=9:53:4\n",
      "2021-04-16 00:43:49 [INFO]\t[TRAIN] Epoch=22/50, Step=3271/3749, loss=0.398203, lr=0.001192, time_each_step=0.32s, eta=9:47:48\n",
      "2021-04-16 00:46:22 [INFO]\t[TRAIN] Epoch 22 finished, loss=0.282526, lr=0.001206 .\n",
      "2021-04-16 00:46:22 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 00:47:07 [INFO]\t[EVAL] Finished, Epoch=22, miou=0.581841, category_iou=[0.56349596 0.75669083 0.5717593  0.43541798], oacc=0.751544, category_acc=[0.79193688 0.83975879 0.71595568 0.57366734], kappa=0.658541, category_F1-score=[0.72081537 0.86149573 0.72754053 0.60667762] .\n",
      "2021-04-16 00:47:11 [INFO]\tModel saved in output/epoch_22.\n",
      "2021-04-16 00:47:11 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 00:50:03 [INFO]\t[TRAIN] Epoch=23/50, Step=522/3749, loss=0.602755, lr=0.001182, time_each_step=0.32s, eta=9:42:23\n",
      "2021-04-16 00:55:23 [INFO]\t[TRAIN] Epoch=23/50, Step=1522/3749, loss=0.225152, lr=0.001171, time_each_step=0.32s, eta=9:37:13\n",
      "2021-04-16 01:00:42 [INFO]\t[TRAIN] Epoch=23/50, Step=2522/3749, loss=0.291161, lr=0.001161, time_each_step=0.32s, eta=9:31:50\n",
      "2021-04-16 01:06:01 [INFO]\t[TRAIN] Epoch=23/50, Step=3522/3749, loss=0.299847, lr=0.001151, time_each_step=0.32s, eta=9:26:32\n",
      "2021-04-16 01:07:13 [INFO]\t[TRAIN] Epoch 23 finished, loss=0.264396, lr=0.001168 .\n",
      "2021-04-16 01:07:13 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:47<00:00,  8.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 01:08:01 [INFO]\t[EVAL] Finished, Epoch=23, miou=0.595046, category_iou=[0.59291828 0.75099853 0.5758524  0.46041619], oacc=0.76477, category_acc=[0.76508129 0.82289767 0.72437519 0.66858539], kappa=0.674236, category_F1-score=[0.74444281 0.85779458 0.73084561 0.63052737] .\n",
      "2021-04-16 01:08:05 [INFO]\tModel saved in output/epoch_23.\n",
      "2021-04-16 01:08:05 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 01:12:15 [INFO]\t[TRAIN] Epoch=24/50, Step=773/3749, loss=0.31016, lr=0.001141, time_each_step=0.32s, eta=9:20:9\n",
      "2021-04-16 01:17:35 [INFO]\t[TRAIN] Epoch=24/50, Step=1773/3749, loss=0.045081, lr=0.001131, time_each_step=0.32s, eta=9:14:47\n",
      "2021-04-16 01:22:55 [INFO]\t[TRAIN] Epoch=24/50, Step=2773/3749, loss=0.254351, lr=0.00112, time_each_step=0.32s, eta=9:9:31\n",
      "2021-04-16 01:28:06 [INFO]\t[TRAIN] Epoch 24 finished, loss=0.253731, lr=0.001129 .\n",
      "2021-04-16 01:28:06 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:46<00:00,  8.92it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 01:28:53 [INFO]\t[EVAL] Finished, Epoch=24, miou=0.594987, category_iou=[0.57567856 0.75677524 0.58837293 0.45912231], oacc=0.761142, category_acc=[0.79522552 0.8367689  0.74003113 0.59757082], kappa=0.67128, category_F1-score=[0.73070559 0.86155044 0.74084986 0.62931298] .\n",
      "2021-04-16 01:28:57 [INFO]\tModel saved in output/epoch_24.\n",
      "2021-04-16 01:28:57 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 01:29:09 [INFO]\t[TRAIN] Epoch=25/50, Step=24/3749, loss=0.211588, lr=0.00111, time_each_step=0.34s, eta=9:3:30\n",
      "2021-04-16 01:34:25 [INFO]\t[TRAIN] Epoch=25/50, Step=1024/3749, loss=0.08266, lr=0.0011, time_each_step=0.31s, eta=8:56:49\n",
      "2021-04-16 01:39:41 [INFO]\t[TRAIN] Epoch=25/50, Step=2024/3749, loss=0.094154, lr=0.00109, time_each_step=0.31s, eta=8:51:36\n",
      "2021-04-16 01:44:57 [INFO]\t[TRAIN] Epoch=25/50, Step=3024/3749, loss=0.381485, lr=0.001079, time_each_step=0.32s, eta=8:46:23\n",
      "2021-04-16 01:48:46 [INFO]\t[TRAIN] Epoch 25 finished, loss=0.238193, lr=0.001091 .\n",
      "2021-04-16 01:48:46 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 01:49:31 [INFO]\t[EVAL] Finished, Epoch=25, miou=0.596388, category_iou=[0.58060129 0.762523   0.58816002 0.4542692 ], oacc=0.765213, category_acc=[0.79122349 0.82690196 0.71413115 0.64246829], kappa=0.675728, category_F1-score=[0.73465876 0.86526303 0.74068105 0.6247388 ] .\n",
      "2021-04-16 01:49:35 [INFO]\tModel saved in output/epoch_25.\n",
      "2021-04-16 01:49:35 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 01:51:07 [INFO]\t[TRAIN] Epoch=26/50, Step=275/3749, loss=0.413077, lr=0.001069, time_each_step=0.31s, eta=8:34:6\n",
      "2021-04-16 01:56:23 [INFO]\t[TRAIN] Epoch=26/50, Step=1275/3749, loss=0.40769, lr=0.001059, time_each_step=0.32s, eta=8:29:5\n",
      "2021-04-16 02:01:38 [INFO]\t[TRAIN] Epoch=26/50, Step=2275/3749, loss=0.111365, lr=0.001048, time_each_step=0.31s, eta=8:23:41\n",
      "2021-04-16 02:06:53 [INFO]\t[TRAIN] Epoch=26/50, Step=3275/3749, loss=0.314589, lr=0.001038, time_each_step=0.31s, eta=8:18:27\n",
      "2021-04-16 02:09:22 [INFO]\t[TRAIN] Epoch 26 finished, loss=0.224537, lr=0.001052 .\n",
      "2021-04-16 02:09:22 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 02:10:07 [INFO]\t[EVAL] Finished, Epoch=26, miou=0.591517, category_iou=[0.5676238  0.76413105 0.58137517 0.4529366 ], oacc=0.758355, category_acc=[0.75696693 0.85702721 0.73409495 0.5994798 ], kappa=0.667676, category_F1-score=[0.7241837  0.86629738 0.73527798 0.62347745] .\n",
      "2021-04-16 02:10:11 [INFO]\tModel saved in output/epoch_26.\n",
      "2021-04-16 02:10:11 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 02:13:05 [INFO]\t[TRAIN] Epoch=27/50, Step=526/3749, loss=0.119304, lr=0.001028, time_each_step=0.32s, eta=8:11:33\n",
      "2021-04-16 02:18:21 [INFO]\t[TRAIN] Epoch=27/50, Step=1526/3749, loss=0.181213, lr=0.001017, time_each_step=0.31s, eta=8:6:13\n",
      "2021-04-16 02:23:37 [INFO]\t[TRAIN] Epoch=27/50, Step=2526/3749, loss=0.232079, lr=0.001007, time_each_step=0.32s, eta=8:1:5\n",
      "2021-04-16 02:28:52 [INFO]\t[TRAIN] Epoch=27/50, Step=3526/3749, loss=0.236695, lr=0.000997, time_each_step=0.31s, eta=7:55:45\n",
      "2021-04-16 02:30:03 [INFO]\t[TRAIN] Epoch 27 finished, loss=0.21697, lr=0.001014 .\n",
      "2021-04-16 02:30:03 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 02:30:47 [INFO]\t[EVAL] Finished, Epoch=27, miou=0.597408, category_iou=[0.59244899 0.75830753 0.57971359 0.45916183], oacc=0.765917, category_acc=[0.7804149  0.83355243 0.71106758 0.65074823], kappa=0.676818, category_F1-score=[0.7440728  0.86254255 0.73394772 0.62935011] .\n",
      "2021-04-16 02:30:51 [INFO]\tModel saved in output/epoch_27.\n",
      "2021-04-16 02:30:51 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 02:35:03 [INFO]\t[TRAIN] Epoch=28/50, Step=777/3749, loss=0.277906, lr=0.000986, time_each_step=0.32s, eta=7:51:9\n",
      "2021-04-16 02:40:19 [INFO]\t[TRAIN] Epoch=28/50, Step=1777/3749, loss=0.114446, lr=0.000976, time_each_step=0.31s, eta=7:45:50\n",
      "2021-04-16 02:45:34 [INFO]\t[TRAIN] Epoch=28/50, Step=2777/3749, loss=0.266003, lr=0.000965, time_each_step=0.32s, eta=7:40:40\n",
      "2021-04-16 02:50:41 [INFO]\t[TRAIN] Epoch 28 finished, loss=0.205339, lr=0.000975 .\n",
      "2021-04-16 02:50:41 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 02:51:25 [INFO]\t[EVAL] Finished, Epoch=28, miou=0.593298, category_iou=[0.59437052 0.75120275 0.58480678 0.44281043], oacc=0.763117, category_acc=[0.7710861  0.82204841 0.7398763  0.63828511], kappa=0.671997, category_F1-score=[0.74558644 0.85792779 0.7380165  0.61381651] .\n",
      "2021-04-16 02:51:29 [INFO]\tModel saved in output/epoch_28.\n",
      "2021-04-16 02:51:29 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_20, miou=0.5990907885794976\n",
      "2021-04-16 02:51:45 [INFO]\t[TRAIN] Epoch=29/50, Step=28/3749, loss=0.149101, lr=0.000955, time_each_step=0.34s, eta=7:35:25\n",
      "2021-04-16 02:57:01 [INFO]\t[TRAIN] Epoch=29/50, Step=1028/3749, loss=0.106304, lr=0.000945, time_each_step=0.32s, eta=7:28:27\n",
      "2021-04-16 03:02:17 [INFO]\t[TRAIN] Epoch=29/50, Step=2028/3749, loss=0.13488, lr=0.000934, time_each_step=0.31s, eta=7:23:4\n",
      "2021-04-16 03:07:32 [INFO]\t[TRAIN] Epoch=29/50, Step=3028/3749, loss=0.177151, lr=0.000924, time_each_step=0.32s, eta=7:17:53\n",
      "2021-04-16 03:11:20 [INFO]\t[TRAIN] Epoch 29 finished, loss=0.192429, lr=0.000936 .\n",
      "2021-04-16 03:11:20 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.40it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 03:12:04 [INFO]\t[EVAL] Finished, Epoch=29, miou=0.599915, category_iou=[0.59731866 0.76152846 0.58655478 0.45425858], oacc=0.765895, category_acc=[0.78102854 0.8452685  0.75434959 0.60402192], kappa=0.677282, category_F1-score=[0.74790169 0.86462237 0.73940691 0.62472876] .\n",
      "2021-04-16 03:12:08 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 03:12:12 [INFO]\tModel saved in output/epoch_29.\n",
      "2021-04-16 03:12:12 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_29, miou=0.5999151192829655\n",
      "2021-04-16 03:13:44 [INFO]\t[TRAIN] Epoch=30/50, Step=279/3749, loss=0.315189, lr=0.000913, time_each_step=0.31s, eta=7:13:9\n",
      "2021-04-16 03:18:59 [INFO]\t[TRAIN] Epoch=30/50, Step=1279/3749, loss=0.226481, lr=0.000903, time_each_step=0.32s, eta=7:8:6\n",
      "2021-04-16 03:24:14 [INFO]\t[TRAIN] Epoch=30/50, Step=2279/3749, loss=0.185118, lr=0.000892, time_each_step=0.32s, eta=7:2:50\n",
      "2021-04-16 03:29:30 [INFO]\t[TRAIN] Epoch=30/50, Step=3279/3749, loss=0.16752, lr=0.000882, time_each_step=0.32s, eta=6:57:33\n",
      "2021-04-16 03:31:58 [INFO]\t[TRAIN] Epoch 30 finished, loss=0.190948, lr=0.000896 .\n",
      "2021-04-16 03:31:58 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 03:32:43 [INFO]\t[EVAL] Finished, Epoch=30, miou=0.600888, category_iou=[0.59177199 0.7634719  0.58668578 0.46162218], oacc=0.767934, category_acc=[0.78114318 0.84103344 0.71471003 0.6456338 ], kappa=0.679955, category_F1-score=[0.74353864 0.86587362 0.73951098 0.63165733] .\n",
      "2021-04-16 03:32:47 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 03:32:51 [INFO]\tModel saved in output/epoch_30.\n",
      "2021-04-16 03:32:51 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_30, miou=0.6008879615783003\n",
      "2021-04-16 03:35:43 [INFO]\t[TRAIN] Epoch=31/50, Step=530/3749, loss=0.068108, lr=0.000871, time_each_step=0.32s, eta=6:50:11\n",
      "2021-04-16 03:40:58 [INFO]\t[TRAIN] Epoch=31/50, Step=1530/3749, loss=0.05303, lr=0.000861, time_each_step=0.31s, eta=6:44:50\n",
      "2021-04-16 03:46:13 [INFO]\t[TRAIN] Epoch=31/50, Step=2530/3749, loss=0.090371, lr=0.00085, time_each_step=0.32s, eta=6:39:38\n",
      "2021-04-16 03:51:29 [INFO]\t[TRAIN] Epoch=31/50, Step=3530/3749, loss=0.30353, lr=0.00084, time_each_step=0.31s, eta=6:34:21\n",
      "2021-04-16 03:52:38 [INFO]\t[TRAIN] Epoch 31 finished, loss=0.17698, lr=0.000857 .\n",
      "2021-04-16 03:52:38 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 03:53:24 [INFO]\t[EVAL] Finished, Epoch=31, miou=0.60882, category_iou=[0.60790726 0.76831327 0.5993288  0.45973025], oacc=0.77323, category_acc=[0.77753426 0.84947309 0.75322357 0.62965936], kappa=0.687067, category_F1-score=[0.75614717 0.86897868 0.7494754  0.62988384] .\n",
      "2021-04-16 03:53:28 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 03:53:31 [INFO]\tModel saved in output/epoch_31.\n",
      "2021-04-16 03:53:31 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_31, miou=0.6088198952873279\n",
      "2021-04-16 03:57:42 [INFO]\t[TRAIN] Epoch=32/50, Step=781/3749, loss=0.198802, lr=0.000829, time_each_step=0.32s, eta=6:28:40\n",
      "2021-04-16 04:02:57 [INFO]\t[TRAIN] Epoch=32/50, Step=1781/3749, loss=0.106352, lr=0.000818, time_each_step=0.31s, eta=6:23:22\n",
      "2021-04-16 04:08:12 [INFO]\t[TRAIN] Epoch=32/50, Step=2781/3749, loss=0.264833, lr=0.000808, time_each_step=0.32s, eta=6:18:10\n",
      "2021-04-16 04:13:17 [INFO]\t[TRAIN] Epoch 32 finished, loss=0.169527, lr=0.000817 .\n",
      "2021-04-16 04:13:17 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 04:14:02 [INFO]\t[EVAL] Finished, Epoch=32, miou=0.60422, category_iou=[0.59700573 0.76164278 0.60389451 0.45433579], oacc=0.769906, category_acc=[0.78607277 0.83361752 0.74536242 0.63294889], kappa=0.682181, category_F1-score=[0.74765634 0.86469606 0.7530352  0.62480177] .\n",
      "2021-04-16 04:14:06 [INFO]\tModel saved in output/epoch_32.\n",
      "2021-04-16 04:14:06 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_31, miou=0.6088198952873279\n",
      "2021-04-16 04:14:20 [INFO]\t[TRAIN] Epoch=33/50, Step=32/3749, loss=0.199292, lr=0.000797, time_each_step=0.33s, eta=6:11:11\n",
      "2021-04-16 04:19:36 [INFO]\t[TRAIN] Epoch=33/50, Step=1032/3749, loss=0.189186, lr=0.000786, time_each_step=0.32s, eta=6:4:51\n",
      "2021-04-16 04:24:52 [INFO]\t[TRAIN] Epoch=33/50, Step=2032/3749, loss=0.187396, lr=0.000776, time_each_step=0.31s, eta=5:59:29\n",
      "2021-04-16 04:30:07 [INFO]\t[TRAIN] Epoch=33/50, Step=3032/3749, loss=0.141018, lr=0.000765, time_each_step=0.32s, eta=5:54:18\n",
      "2021-04-16 04:33:53 [INFO]\t[TRAIN] Epoch 33 finished, loss=0.161461, lr=0.000777 .\n",
      "2021-04-16 04:33:53 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 04:34:37 [INFO]\t[EVAL] Finished, Epoch=33, miou=0.608865, category_iou=[0.60388151 0.76791071 0.58924945 0.47441736], oacc=0.773397, category_acc=[0.77923855 0.84974911 0.73659792 0.6436588 ], kappa=0.687504, category_F1-score=[0.75302509 0.86872115 0.74154432 0.64353198] .\n",
      "2021-04-16 04:34:42 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 04:34:46 [INFO]\tModel saved in output/epoch_33.\n",
      "2021-04-16 04:34:46 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_33, miou=0.608864756686981\n",
      "2021-04-16 04:36:21 [INFO]\t[TRAIN] Epoch=34/50, Step=283/3749, loss=0.071658, lr=0.000754, time_each_step=0.31s, eta=5:49:43\n",
      "2021-04-16 04:41:36 [INFO]\t[TRAIN] Epoch=34/50, Step=1283/3749, loss=0.160911, lr=0.000744, time_each_step=0.32s, eta=5:44:33\n",
      "2021-04-16 04:46:51 [INFO]\t[TRAIN] Epoch=34/50, Step=2283/3749, loss=0.173385, lr=0.000733, time_each_step=0.32s, eta=5:39:15\n",
      "2021-04-16 04:52:07 [INFO]\t[TRAIN] Epoch=34/50, Step=3283/3749, loss=0.125366, lr=0.000722, time_each_step=0.32s, eta=5:34:0\n",
      "2021-04-16 04:54:34 [INFO]\t[TRAIN] Epoch 34 finished, loss=0.154127, lr=0.000737 .\n",
      "2021-04-16 04:54:34 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.46it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 04:55:18 [INFO]\t[EVAL] Finished, Epoch=34, miou=0.608011, category_iou=[0.60717988 0.76451666 0.59417326 0.46617294], oacc=0.773538, category_acc=[0.76982546 0.84061991 0.75116445 0.65392459], kappa=0.686818, category_F1-score=[0.75558423 0.86654513 0.74543122 0.63590443] .\n",
      "2021-04-16 04:55:22 [INFO]\tModel saved in output/epoch_34.\n",
      "2021-04-16 04:55:22 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_33, miou=0.608864756686981\n",
      "2021-04-16 04:58:15 [INFO]\t[TRAIN] Epoch=35/50, Step=534/3749, loss=0.175455, lr=0.000712, time_each_step=0.32s, eta=5:26:55\n",
      "2021-04-16 05:03:30 [INFO]\t[TRAIN] Epoch=35/50, Step=1534/3749, loss=0.136112, lr=0.000701, time_each_step=0.31s, eta=5:21:26\n",
      "2021-04-16 05:08:45 [INFO]\t[TRAIN] Epoch=35/50, Step=2534/3749, loss=0.062245, lr=0.00069, time_each_step=0.32s, eta=5:16:23\n",
      "2021-04-16 05:14:01 [INFO]\t[TRAIN] Epoch=35/50, Step=3534/3749, loss=0.382284, lr=0.000679, time_each_step=0.31s, eta=5:11:6\n",
      "2021-04-16 05:15:08 [INFO]\t[TRAIN] Epoch 35 finished, loss=0.14935, lr=0.000697 .\n",
      "2021-04-16 05:15:08 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:47<00:00,  8.74it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 05:15:56 [INFO]\t[EVAL] Finished, Epoch=35, miou=0.5925, category_iou=[0.57549349 0.76304576 0.58122642 0.45023614], oacc=0.760798, category_acc=[0.79353416 0.84095313 0.70710933 0.61057732], kappa=0.670872, category_F1-score=[0.73055648 0.86559949 0.73515901 0.62091425] .\n",
      "2021-04-16 05:16:00 [INFO]\tModel saved in output/epoch_35.\n",
      "2021-04-16 05:16:00 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_33, miou=0.608864756686981\n",
      "2021-04-16 05:20:12 [INFO]\t[TRAIN] Epoch=36/50, Step=785/3749, loss=0.055515, lr=0.000668, time_each_step=0.32s, eta=5:5:14\n",
      "2021-04-16 05:25:27 [INFO]\t[TRAIN] Epoch=36/50, Step=1785/3749, loss=0.211259, lr=0.000657, time_each_step=0.31s, eta=4:59:51\n",
      "2021-04-16 05:30:43 [INFO]\t[TRAIN] Epoch=36/50, Step=2785/3749, loss=0.160596, lr=0.000647, time_each_step=0.31s, eta=4:54:36\n",
      "2021-04-16 05:35:46 [INFO]\t[TRAIN] Epoch 36 finished, loss=0.143554, lr=0.000656 .\n",
      "2021-04-16 05:35:46 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:43<00:00,  9.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 05:36:30 [INFO]\t[EVAL] Finished, Epoch=36, miou=0.60585, category_iou=[0.59309549 0.76844987 0.59586628 0.46598688], oacc=0.770117, category_acc=[0.78357605 0.85063093 0.7419764  0.62089559], kappa=0.683398, category_F1-score=[0.74458247 0.86906605 0.74676217 0.63573131] .\n",
      "2021-04-16 05:36:34 [INFO]\tModel saved in output/epoch_36.\n",
      "2021-04-16 05:36:34 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_33, miou=0.608864756686981\n",
      "2021-04-16 05:36:52 [INFO]\t[TRAIN] Epoch=37/50, Step=36/3749, loss=0.048143, lr=0.000636, time_each_step=0.33s, eta=4:48:42\n",
      "2021-04-16 05:42:07 [INFO]\t[TRAIN] Epoch=37/50, Step=1036/3749, loss=0.21106, lr=0.000625, time_each_step=0.32s, eta=4:42:26\n",
      "2021-04-16 05:47:23 [INFO]\t[TRAIN] Epoch=37/50, Step=2036/3749, loss=0.11619, lr=0.000614, time_each_step=0.31s, eta=4:37:8\n",
      "2021-04-16 05:52:37 [INFO]\t[TRAIN] Epoch=37/50, Step=3036/3749, loss=0.100042, lr=0.000603, time_each_step=0.31s, eta=4:31:55\n",
      "2021-04-16 05:56:22 [INFO]\t[TRAIN] Epoch 37 finished, loss=0.138133, lr=0.000616 .\n",
      "2021-04-16 05:56:22 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 05:57:08 [INFO]\t[EVAL] Finished, Epoch=37, miou=0.606783, category_iou=[0.60318112 0.76946365 0.58691679 0.46756894], oacc=0.772062, category_acc=[0.78525702 0.84596747 0.74858385 0.62684208], kappa=0.685486, category_F1-score=[0.75248031 0.86971399 0.73969448 0.63720201] .\n",
      "2021-04-16 05:57:11 [INFO]\tModel saved in output/epoch_37.\n",
      "2021-04-16 05:57:11 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_33, miou=0.608864756686981\n",
      "2021-04-16 05:58:50 [INFO]\t[TRAIN] Epoch=38/50, Step=287/3749, loss=0.187261, lr=0.000592, time_each_step=0.31s, eta=4:26:20\n",
      "2021-04-16 06:04:05 [INFO]\t[TRAIN] Epoch=38/50, Step=1287/3749, loss=0.164518, lr=0.000581, time_each_step=0.32s, eta=4:21:29\n",
      "2021-04-16 06:09:19 [INFO]\t[TRAIN] Epoch=38/50, Step=2287/3749, loss=0.054056, lr=0.00057, time_each_step=0.31s, eta=4:15:56\n",
      "2021-04-16 06:14:34 [INFO]\t[TRAIN] Epoch=38/50, Step=3287/3749, loss=0.207177, lr=0.000559, time_each_step=0.31s, eta=4:10:40\n",
      "2021-04-16 06:17:00 [INFO]\t[TRAIN] Epoch 38 finished, loss=0.132725, lr=0.000574 .\n",
      "2021-04-16 06:17:00 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 06:17:45 [INFO]\t[EVAL] Finished, Epoch=38, miou=0.613884, category_iou=[0.61224923 0.77300416 0.59318133 0.47710006], oacc=0.777879, category_acc=[0.78709444 0.851399   0.72336013 0.66204159], kappa=0.693662, category_F1-score=[0.759497   0.87197106 0.74465011 0.64599558] .\n",
      "2021-04-16 06:17:49 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 06:17:53 [INFO]\tModel saved in output/epoch_38.\n",
      "2021-04-16 06:17:53 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_38, miou=0.6138836934695744\n",
      "2021-04-16 06:20:46 [INFO]\t[TRAIN] Epoch=39/50, Step=538/3749, loss=0.117927, lr=0.000548, time_each_step=0.31s, eta=4:5:20\n",
      "2021-04-16 06:26:01 [INFO]\t[TRAIN] Epoch=39/50, Step=1538/3749, loss=0.100869, lr=0.000537, time_each_step=0.31s, eta=4:0:4\n",
      "2021-04-16 06:31:16 [INFO]\t[TRAIN] Epoch=39/50, Step=2538/3749, loss=0.298274, lr=0.000525, time_each_step=0.32s, eta=3:54:53\n",
      "2021-04-16 06:36:31 [INFO]\t[TRAIN] Epoch=39/50, Step=3538/3749, loss=0.103746, lr=0.000514, time_each_step=0.32s, eta=3:49:37\n",
      "2021-04-16 06:37:38 [INFO]\t[TRAIN] Epoch 39 finished, loss=0.129755, lr=0.000533 .\n",
      "2021-04-16 06:37:38 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.45it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 06:38:22 [INFO]\t[EVAL] Finished, Epoch=39, miou=0.607766, category_iou=[0.60164877 0.77161315 0.58876331 0.46904013], oacc=0.773217, category_acc=[0.78542902 0.84966377 0.71834987 0.64912111], kappa=0.687375, category_F1-score=[0.75128677 0.87108537 0.74115924 0.6385668 ] .\n",
      "2021-04-16 06:38:26 [INFO]\tModel saved in output/epoch_39.\n",
      "2021-04-16 06:38:26 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_38, miou=0.6138836934695744\n",
      "2021-04-16 06:42:43 [INFO]\t[TRAIN] Epoch=40/50, Step=789/3749, loss=0.118333, lr=0.000503, time_each_step=0.31s, eta=3:41:51\n",
      "2021-04-16 06:47:58 [INFO]\t[TRAIN] Epoch=40/50, Step=1789/3749, loss=0.153908, lr=0.000492, time_each_step=0.32s, eta=3:36:39\n",
      "2021-04-16 06:53:13 [INFO]\t[TRAIN] Epoch=40/50, Step=2789/3749, loss=0.062955, lr=0.000481, time_each_step=0.32s, eta=3:31:23\n",
      "2021-04-16 06:58:15 [INFO]\t[TRAIN] Epoch 40 finished, loss=0.124679, lr=0.000491 .\n",
      "2021-04-16 06:58:15 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 06:59:00 [INFO]\t[EVAL] Finished, Epoch=40, miou=0.610882, category_iou=[0.6050743  0.77177089 0.59796059 0.46872126], oacc=0.775425, category_acc=[0.79258993 0.84385044 0.7346681  0.64521523], kappa=0.690103, category_F1-score=[0.75395176 0.87118588 0.74840468 0.63827123] .\n",
      "2021-04-16 06:59:04 [INFO]\tModel saved in output/epoch_40.\n",
      "2021-04-16 06:59:04 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_38, miou=0.6138836934695744\n",
      "2021-04-16 06:59:24 [INFO]\t[TRAIN] Epoch=41/50, Step=40/3749, loss=0.069628, lr=0.000469, time_each_step=0.32s, eta=3:26:23\n",
      "2021-04-16 07:04:39 [INFO]\t[TRAIN] Epoch=41/50, Step=1040/3749, loss=0.155049, lr=0.000458, time_each_step=0.31s, eta=3:20:41\n",
      "2021-04-16 07:09:55 [INFO]\t[TRAIN] Epoch=41/50, Step=2040/3749, loss=0.024523, lr=0.000447, time_each_step=0.31s, eta=3:15:25\n",
      "2021-04-16 07:15:09 [INFO]\t[TRAIN] Epoch=41/50, Step=3040/3749, loss=0.134456, lr=0.000435, time_each_step=0.31s, eta=3:10:12\n",
      "2021-04-16 07:18:52 [INFO]\t[TRAIN] Epoch 41 finished, loss=0.120369, lr=0.000449 .\n",
      "2021-04-16 07:18:52 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 07:19:38 [INFO]\t[EVAL] Finished, Epoch=41, miou=0.612984, category_iou=[0.60834424 0.76957124 0.59826372 0.47575537], oacc=0.777069, category_acc=[0.78707481 0.84041976 0.74547768 0.65568687], kappa=0.69197, category_F1-score=[0.75648512 0.86978272 0.74864206 0.64476183] .\n",
      "2021-04-16 07:19:41 [INFO]\tModel saved in output/epoch_41.\n",
      "2021-04-16 07:19:41 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_38, miou=0.6138836934695744\n",
      "2021-04-16 07:21:20 [INFO]\t[TRAIN] Epoch=42/50, Step=291/3749, loss=0.116455, lr=0.000424, time_each_step=0.31s, eta=3:3:45\n",
      "2021-04-16 07:26:35 [INFO]\t[TRAIN] Epoch=42/50, Step=1291/3749, loss=0.0422, lr=0.000413, time_each_step=0.32s, eta=2:58:45\n",
      "2021-04-16 07:31:50 [INFO]\t[TRAIN] Epoch=42/50, Step=2291/3749, loss=0.134402, lr=0.000401, time_each_step=0.31s, eta=2:53:22\n",
      "2021-04-16 07:37:05 [INFO]\t[TRAIN] Epoch=42/50, Step=3291/3749, loss=0.053235, lr=0.00039, time_each_step=0.32s, eta=2:48:12\n",
      "2021-04-16 07:39:30 [INFO]\t[TRAIN] Epoch 42 finished, loss=0.118026, lr=0.000406 .\n",
      "2021-04-16 07:39:30 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 07:40:14 [INFO]\t[EVAL] Finished, Epoch=42, miou=0.61474, category_iou=[0.609514   0.77303001 0.59505694 0.48135914], oacc=0.77847, category_acc=[0.79763342 0.84118631 0.74230214 0.65267326], kappa=0.694074, category_F1-score=[0.75738887 0.87198751 0.74612627 0.6498885 ] .\n",
      "2021-04-16 07:40:18 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 07:40:21 [INFO]\tModel saved in output/epoch_42.\n",
      "2021-04-16 07:40:21 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_42, miou=0.6147400218328488\n",
      "2021-04-16 07:43:19 [INFO]\t[TRAIN] Epoch=43/50, Step=542/3749, loss=0.028736, lr=0.000378, time_each_step=0.32s, eta=2:42:23\n",
      "2021-04-16 07:48:34 [INFO]\t[TRAIN] Epoch=43/50, Step=1542/3749, loss=0.106042, lr=0.000367, time_each_step=0.31s, eta=2:37:2\n",
      "2021-04-16 07:53:49 [INFO]\t[TRAIN] Epoch=43/50, Step=2542/3749, loss=0.05293, lr=0.000355, time_each_step=0.32s, eta=2:31:51\n",
      "2021-04-16 07:59:04 [INFO]\t[TRAIN] Epoch=43/50, Step=3542/3749, loss=0.040404, lr=0.000343, time_each_step=0.31s, eta=2:26:35\n",
      "2021-04-16 08:00:09 [INFO]\t[TRAIN] Epoch 43 finished, loss=0.115388, lr=0.000363 .\n",
      "2021-04-16 08:00:09 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 08:00:53 [INFO]\t[EVAL] Finished, Epoch=43, miou=0.610794, category_iou=[0.60557479 0.76848419 0.59729365 0.47182271], oacc=0.77532, category_acc=[0.78633429 0.84114329 0.74102706 0.65096843], kappa=0.689706, category_F1-score=[0.75434018 0.869088   0.74788208 0.64114068] .\n",
      "2021-04-16 08:00:57 [INFO]\tModel saved in output/epoch_43.\n",
      "2021-04-16 08:00:57 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_42, miou=0.6147400218328488\n",
      "2021-04-16 08:05:11 [INFO]\t[TRAIN] Epoch=44/50, Step=793/3749, loss=0.169272, lr=0.000332, time_each_step=0.32s, eta=2:19:58\n",
      "2021-04-16 08:10:27 [INFO]\t[TRAIN] Epoch=44/50, Step=1793/3749, loss=0.204443, lr=0.00032, time_each_step=0.31s, eta=2:14:37\n",
      "2021-04-16 08:15:42 [INFO]\t[TRAIN] Epoch=44/50, Step=2793/3749, loss=0.073894, lr=0.000308, time_each_step=0.31s, eta=2:9:24\n",
      "2021-04-16 08:20:44 [INFO]\t[TRAIN] Epoch 44 finished, loss=0.112406, lr=0.000319 .\n",
      "2021-04-16 08:20:44 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.25it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 08:21:29 [INFO]\t[EVAL] Finished, Epoch=44, miou=0.611214, category_iou=[0.60384822 0.77160622 0.59860759 0.4707955 ], oacc=0.775223, category_acc=[0.79220793 0.84508984 0.74183012 0.63859135], kappa=0.689916, category_F1-score=[0.75299921 0.87108096 0.74891124 0.64019165] .\n",
      "2021-04-16 08:21:33 [INFO]\tModel saved in output/epoch_44.\n",
      "2021-04-16 08:21:33 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_42, miou=0.6147400218328488\n",
      "2021-04-16 08:21:51 [INFO]\t[TRAIN] Epoch=45/50, Step=44/3749, loss=0.113556, lr=0.000296, time_each_step=0.32s, eta=2:3:38\n",
      "2021-04-16 08:27:06 [INFO]\t[TRAIN] Epoch=45/50, Step=1044/3749, loss=0.07392, lr=0.000284, time_each_step=0.32s, eta=1:58:2\n",
      "2021-04-16 08:32:21 [INFO]\t[TRAIN] Epoch=45/50, Step=2044/3749, loss=0.056232, lr=0.000272, time_each_step=0.31s, eta=1:52:40\n",
      "2021-04-16 08:37:36 [INFO]\t[TRAIN] Epoch=45/50, Step=3044/3749, loss=0.128789, lr=0.00026, time_each_step=0.31s, eta=1:47:27\n",
      "2021-04-16 08:41:18 [INFO]\t[TRAIN] Epoch 45 finished, loss=0.109844, lr=0.000274 .\n",
      "2021-04-16 08:41:18 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 08:42:03 [INFO]\t[EVAL] Finished, Epoch=45, miou=0.615275, category_iou=[0.60987062 0.7730298  0.60196106 0.47623716], oacc=0.778229, category_acc=[0.79522239 0.84743915 0.74008314 0.64688852], kappa=0.694125, category_F1-score=[0.75766414 0.87198737 0.7515302  0.64520414] .\n",
      "2021-04-16 08:42:07 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 08:42:11 [INFO]\tModel saved in output/epoch_45.\n",
      "2021-04-16 08:42:11 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_45, miou=0.6152746620596152\n",
      "2021-04-16 08:43:51 [INFO]\t[TRAIN] Epoch=46/50, Step=295/3749, loss=0.076562, lr=0.000248, time_each_step=0.31s, eta=1:41:26\n",
      "2021-04-16 08:49:06 [INFO]\t[TRAIN] Epoch=46/50, Step=1295/3749, loss=0.179864, lr=0.000236, time_each_step=0.32s, eta=1:36:25\n",
      "2021-04-16 08:54:21 [INFO]\t[TRAIN] Epoch=46/50, Step=2295/3749, loss=0.067676, lr=0.000224, time_each_step=0.31s, eta=1:30:59\n",
      "2021-04-16 08:59:37 [INFO]\t[TRAIN] Epoch=46/50, Step=3295/3749, loss=0.174473, lr=0.000212, time_each_step=0.32s, eta=1:25:47\n",
      "2021-04-16 09:02:00 [INFO]\t[TRAIN] Epoch 46 finished, loss=0.107386, lr=0.000229 .\n",
      "2021-04-16 09:02:00 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.31it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 09:02:45 [INFO]\t[EVAL] Finished, Epoch=46, miou=0.615272, category_iou=[0.61417193 0.7732878  0.59807928 0.47554867], oacc=0.779057, category_acc=[0.7868103  0.84623502 0.74049485 0.65961054], kappa=0.694837, category_F1-score=[0.76097462 0.87215149 0.74849764 0.64457199] .\n",
      "2021-04-16 09:02:49 [INFO]\tModel saved in output/epoch_46.\n",
      "2021-04-16 09:02:49 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_45, miou=0.6152746620596152\n",
      "2021-04-16 09:05:50 [INFO]\t[TRAIN] Epoch=47/50, Step=546/3749, loss=0.081533, lr=0.000199, time_each_step=0.32s, eta=1:19:38\n",
      "2021-04-16 09:11:08 [INFO]\t[TRAIN] Epoch=47/50, Step=1546/3749, loss=0.066272, lr=0.000187, time_each_step=0.32s, eta=1:14:25\n",
      "2021-04-16 09:16:26 [INFO]\t[TRAIN] Epoch=47/50, Step=2546/3749, loss=0.079756, lr=0.000174, time_each_step=0.32s, eta=1:9:8\n",
      "2021-04-16 09:21:44 [INFO]\t[TRAIN] Epoch=47/50, Step=3546/3749, loss=0.069354, lr=0.000162, time_each_step=0.32s, eta=1:3:47\n",
      "2021-04-16 09:22:48 [INFO]\t[TRAIN] Epoch 47 finished, loss=0.106154, lr=0.000183 .\n",
      "2021-04-16 09:22:48 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:45<00:00,  9.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 09:23:34 [INFO]\t[EVAL] Finished, Epoch=47, miou=0.614726, category_iou=[0.6147539  0.77095473 0.59950844 0.47368758], oacc=0.778775, category_acc=[0.78709184 0.84321798 0.73825187 0.66408338], kappa=0.694342, category_F1-score=[0.76142117 0.87066565 0.74961585 0.64286025] .\n",
      "2021-04-16 09:23:38 [INFO]\tModel saved in output/epoch_47.\n",
      "2021-04-16 09:23:38 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_45, miou=0.6152746620596152\n",
      "2021-04-16 09:28:01 [INFO]\t[TRAIN] Epoch=48/50, Step=797/3749, loss=0.122975, lr=0.000149, time_each_step=0.32s, eta=0:58:5\n",
      "2021-04-16 09:33:20 [INFO]\t[TRAIN] Epoch=48/50, Step=1797/3749, loss=0.181057, lr=0.000136, time_each_step=0.32s, eta=0:52:45\n",
      "2021-04-16 09:38:38 [INFO]\t[TRAIN] Epoch=48/50, Step=2797/3749, loss=0.088352, lr=0.000123, time_each_step=0.32s, eta=0:47:31\n",
      "2021-04-16 09:43:41 [INFO]\t[TRAIN] Epoch 48 finished, loss=0.103613, lr=0.000135 .\n",
      "2021-04-16 09:43:41 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:43<00:00,  9.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 09:44:25 [INFO]\t[EVAL] Finished, Epoch=48, miou=0.613166, category_iou=[0.61063524 0.77127256 0.59901381 0.47174257], oacc=0.777716, category_acc=[0.79768435 0.83845153 0.7337561  0.65802564], kappa=0.692925, category_F1-score=[0.75825392 0.8708683  0.74922907 0.6410667 ] .\n",
      "2021-04-16 09:44:28 [INFO]\tModel saved in output/epoch_48.\n",
      "2021-04-16 09:44:28 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_45, miou=0.6152746620596152\n",
      "2021-04-16 09:44:48 [INFO]\t[TRAIN] Epoch=49/50, Step=48/3749, loss=0.269607, lr=0.00011, time_each_step=0.32s, eta=0:41:15\n",
      "2021-04-16 09:50:05 [INFO]\t[TRAIN] Epoch=49/50, Step=1048/3749, loss=0.141664, lr=9.6e-05, time_each_step=0.32s, eta=0:35:58\n",
      "2021-04-16 09:55:23 [INFO]\t[TRAIN] Epoch=49/50, Step=2048/3749, loss=0.090244, lr=8.3e-05, time_each_step=0.31s, eta=0:30:33\n",
      "2021-04-16 10:00:41 [INFO]\t[TRAIN] Epoch=49/50, Step=3048/3749, loss=0.104521, lr=6.9e-05, time_each_step=0.32s, eta=0:25:21\n",
      "2021-04-16 10:04:23 [INFO]\t[TRAIN] Epoch 49 finished, loss=0.102689, lr=8.5e-05 .\n",
      "2021-04-16 10:04:23 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:46<00:00,  8.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 10:05:09 [INFO]\t[EVAL] Finished, Epoch=49, miou=0.614877, category_iou=[0.61274147 0.77254999 0.60064243 0.47357422], oacc=0.778587, category_acc=[0.79561334 0.84280223 0.74067444 0.65256217], kappa=0.694262, category_F1-score=[0.75987563 0.87168203 0.7505017  0.64275584] .\n",
      "2021-04-16 10:05:13 [INFO]\tModel saved in output/epoch_49.\n",
      "2021-04-16 10:05:13 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_45, miou=0.6152746620596152\n",
      "2021-04-16 10:06:53 [INFO]\t[TRAIN] Epoch=50/50, Step=299/3749, loss=0.129045, lr=5.5e-05, time_each_step=0.32s, eta=0:19:3\n",
      "2021-04-16 10:12:09 [INFO]\t[TRAIN] Epoch=50/50, Step=1299/3749, loss=0.103878, lr=4e-05, time_each_step=0.32s, eta=0:13:47\n",
      "2021-04-16 10:17:24 [INFO]\t[TRAIN] Epoch=50/50, Step=2299/3749, loss=0.148388, lr=2.5e-05, time_each_step=0.32s, eta=0:8:38\n",
      "2021-04-16 10:22:41 [INFO]\t[TRAIN] Epoch=50/50, Step=3299/3749, loss=0.107657, lr=9e-06, time_each_step=0.32s, eta=0:3:12\n",
      "2021-04-16 10:25:03 [INFO]\t[TRAIN] Epoch 50 finished, loss=0.102158, lr=3.1e-05 .\n",
      "2021-04-16 10:25:03 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 10:25:48 [INFO]\t[EVAL] Finished, Epoch=50, miou=0.615551, category_iou=[0.61357248 0.77289468 0.60102085 0.47471632], oacc=0.779082, category_acc=[0.79452774 0.84376905 0.74046035 0.65481823], kappa=0.694952, category_F1-score=[0.76051431 0.8719014  0.75079703 0.64380697] .\n",
      "2021-04-16 10:25:52 [INFO]\tModel saved in output/best_model.\n",
      "2021-04-16 10:25:56 [INFO]\tModel saved in output/epoch_50.\n",
      "2021-04-16 10:25:56 [INFO]\tCurrent evaluated best model in eval_dataset is epoch_50, miou=0.6155510817035672\n"
     ]
    }
   ],
   "source": [
    "# 分割类别数\r\n",
    "num_classes = len(train_dataset.labels)\r\n",
    "\r\n",
    "# 构建DeepLabv3p分割器\r\n",
    "model = pdx.seg.DeepLabv3p(\r\n",
    "    num_classes=num_classes,  backbone='Xception65', use_bce_loss=False\r\n",
    ")\r\n",
    "\r\n",
    "# 模型训练\r\n",
    "model.train(\r\n",
    "    num_epochs=50,                 # 训练迭代轮数\r\n",
    "    train_dataset=train_dataset,  # 训练集读取\r\n",
    "    train_batch_size=16,           # 训练时批处理图片数\r\n",
    "    eval_dataset=eval_dataset,    # 验证集读取\r\n",
    "    learning_rate=0.002,         # 学习率\r\n",
    "    save_interval_epochs=1,       # 保存模型间隔轮次  \r\n",
    "    save_dir='output',    # 模型保存路径\r\n",
    "    log_interval_steps=1000,       # 日志打印间隔\r\n",
    "    pretrain_weights='output/best_model')  #加载预训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 模型评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/math_op_patch.py:298: UserWarning: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlex/cv/nets/xception.py:316\n",
      "The behavior of expression A + B has been unified with elementwise_add(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_add(X, Y, axis=0) instead of A + B. This transitional warning will be dropped in the future.\n",
      "  op_type, op_type, EXPRESSION_MAP[method_name]))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-16 12:16:23 [INFO]\tModel[DeepLabv3p] loaded.\n",
      "2021-04-16 12:16:23 [INFO]\tStart to evaluating(total_samples=6665, total_steps=417)...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 417/417 [00:44<00:00,  9.36it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "OrderedDict([('miou', 0.6155510817035672),\n",
       "             ('category_iou',\n",
       "              array([0.61357248, 0.77289468, 0.60102085, 0.47471632])),\n",
       "             ('oacc', 0.7790819594372721),\n",
       "             ('category_acc',\n",
       "              array([0.79452774, 0.84376905, 0.74046035, 0.65481823])),\n",
       "             ('kappa', 0.6949516620998974),\n",
       "             ('category_F1-score',\n",
       "              array([0.76051431, 0.8719014 , 0.75079703, 0.64380697]))])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加载模型\r\n",
    "model = pdx.load_model('./output/best_model')\r\n",
    "\r\n",
    "# 模型评估\r\n",
    "model.evaluate(eval_dataset, batch_size=16, epoch_id=None, return_details=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 模型预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4608/4608 [01:25<00:00, 53.92it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\r\n",
    "import cv2\r\n",
    "\r\n",
    "test_base = 'img_testA/'    # 测试集路径\r\n",
    "out_base = 'result/'        # 预测结果保存路径\r\n",
    "\r\n",
    "# 是否存在结果保存路径，如不存在，则创建该路径\r\n",
    "if not os.path.exists(out_base):\r\n",
    "    os.makedirs(out_base)\r\n",
    "\r\n",
    "# 模型预测并保存预测图片\r\n",
    "for im in tqdm(os.listdir(test_base)):\r\n",
    "    if not im.endswith('.jpg'):\r\n",
    "        continue\r\n",
    "    pt = test_base + im\r\n",
    "    result = model.predict(pt)\r\n",
    "    cv2.imwrite(out_base+im.replace('jpg', 'png'), result['label_map'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "\n",
    "我在AI Studio上获得黄金等级，点亮6个徽章，来互关呀~ \n",
    "\n",
    "https://aistudio.baidu.com/aistudio/personalcenter/thirdview/643467\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.0.0b0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}