01-Prepare_HMDB51.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1504a522-3ed6-4fd0-bb11-064a65ef98f9",
   "metadata": {},
   "source": [
    "# PyVideoAI v0.4\n",
    "\n",
    "We'll download the HMDB-51 dataset (2GB) and extract videos into frames of images.\n",
    "\n",
    "You need one HDD (slow) and one SSD (fast) storages. We'll make symbolic links into the `data/` directory.  \n",
    "We assume the path of each is `/mnt/hdd/kiyoon` and `/home/kiyoon` respectively, but change it in `00-storage_location.py` regarding your system.\n",
    "\n",
    "Note that the paths to directories are defined in `dataset_configs/hmdb.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c914c937-be9e-4b9a-995a-78bbdc2282a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PYVIDEOAI_DIR=/home/kiyoon/project/PyVideoAI\n",
      "env: DATA_DIR=/home/kiyoon/project/PyVideoAI/data\n",
      "env: EXPERIMENT_ROOT=/home/kiyoon/project/PyVideoAI/data/experiments\n",
      "env: HDD_PATH=/mnt/hdd/kiyoon\n",
      "env: SSD_PATH=/home/kiyoon\n"
     ]
    }
   ],
   "source": [
    "## IMPORTANT: You must change path values in `00-storage_location.py` before executing below.\n",
    "# Environments for future use\n",
    "from pyvideoai.config import PYVIDEOAI_DIR, DATA_DIR, DEFAULT_EXPERIMENT_ROOT\n",
    "%env PYVIDEOAI_DIR=$PYVIDEOAI_DIR\n",
    "%env DATA_DIR=$DATA_DIR\n",
    "%env EXPERIMENT_ROOT=$DEFAULT_EXPERIMENT_ROOT\n",
    "\n",
    "import os\n",
    "exec(open(os.path.join(PYVIDEOAI_DIR, 'examples', '00-storage_location.py')).read())\n",
    "%env HDD_PATH=$HDD_PATH\n",
    "%env SSD_PATH=$SSD_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "78888c4e-0a11-4cda-8702-d75ac4953aba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘/home/kiyoon/project/PyVideoAI/data’: File exists\n"
     ]
    }
   ],
   "source": [
    "# Directory preparation\n",
    "# Make soft links in data/ directory\n",
    "\n",
    "!mkdir -p \"$HDD_PATH/datasets/hmdb51\"\n",
    "!mkdir -p \"$SSD_PATH/datasets/hmdb51/frames_q5\"\n",
    "!mkdir \"$DATA_DIR\"\n",
    "!ln -s \"$HDD_PATH/datasets/hmdb51\" \"$DATA_DIR/\"\n",
    "!ln -s \"$SSD_PATH/datasets/hmdb51/frames_q5\" \"$DATA_DIR/hmdb51/\"\n",
    "\n",
    "# Make experiments output directory as well\n",
    "!mkdir -p \"$HDD_PATH/PyVideoAI_experiments\"\n",
    "!ln -s \"$HDD_PATH/PyVideoAI_experiments\" \"$EXPERIMENT_ROOT\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb66e7ae-9c95-4ecb-adbe-7e0ba6c959d8",
   "metadata": {},
   "source": [
    "Now that we have all directories setup, we'll download and extract the RAR archived files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "22c3107b-62aa-4226-9e62-c9296f69c4cf",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading video data\n",
      "--2022-08-19 16:18:10--  http://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar\n",
      "Resolving serre-lab.clps.brown.edu (serre-lab.clps.brown.edu)... 128.148.254.114\n",
      "Connecting to serre-lab.clps.brown.edu (serre-lab.clps.brown.edu)|128.148.254.114|:80... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar [following]\n",
      "--2022-08-19 16:18:11--  https://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar\n",
      "Connecting to serre-lab.clps.brown.edu (serre-lab.clps.brown.edu)|128.148.254.114|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2124008126 (2.0G)\n",
      "Saving to: ‘/home/kiyoon/project/PyVideoAI/data/hmdb51/hmdb51_org.rar’\n",
      "\n",
      "/home/kiyoon/projec 100%[===================>]   1.98G  25.3MB/s    in 91s     \n",
      "\n",
      "2022-08-19 16:19:42 (22.4 MB/s) - ‘/home/kiyoon/project/PyVideoAI/data/hmdb51/hmdb51_org.rar’ saved [2124008126/2124008126]\n",
      "\n",
      "Extracting video data\n",
      "\n",
      "UNRAR 5.50 freeware      Copyright (c) 1993-2017 Alexander Roshal\n",
      "\n",
      "\n",
      "Extracting from /home/kiyoon/project/PyVideoAI/data/hmdb51/hmdb51_org.rar\n",
      "\n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/shoot_gun.rar       1  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/sit.rar         3  OK \n",
      "-----------------------------------------------------------\n",
      "----------OUTPUT SIMPLIFIED FOR NOTEBOOK VIEWERS-----------\n",
      "-----------------------------------------------------------\n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/shoot_ball.rar    9 96  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/shoot_bow.rar    9 9 9 99  OK \n",
      "All OK\n",
      "\n",
      "UNRAR 5.61 beta 1 freeware      Copyright (c) 1993-2018 Alexander Roshal\n",
      "\n",
      "\n",
      "Extracting from /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/hug.rar\n",
      "\n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/hug/-_FREE_HUGS_-_Abrazos_Gratis_www_abrazosgratis_org_hug_u_cm_np2_ba_goo_10.avi       1  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/hug/-_FREE_HUGS_-_Abrazos_Gratis_www_abrazosgratis_org_hug_u_cm_np2_ba_goo_2.avi     1  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/hug/-_FREE_HUGS_-_Abrazos_Gratis_www_abrazosgratis_org_hug_u_cm_np2_ba_goo_6.avi     2  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/videos/hug/-_FREE_HUGS_-_Abrazos_Gratis_www_abrazosgratis_org_hug_u_cm_np2_ba_goo_7.avi       3  OK \n",
      "-----------------------------------------------------------\n",
      "----------OUTPUT SIMPLIFIED FOR NOTEBOOK VIEWERS-----------\n",
      "-----------------------------------------------------------\n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/testTrainMulti_7030_splits/walk_test_split2.txt    95  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/testTrainMulti_7030_splits/walk_test_split3.txt    97  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/testTrainMulti_7030_splits/wave_test_split1.txt    98  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/testTrainMulti_7030_splits/wave_test_split2.txt    99  OK \n",
      "Extracting  /home/kiyoon/project/PyVideoAI/data/hmdb51/testTrainMulti_7030_splits/wave_test_split3.txt    99  OK \n",
      "All OK\n",
      "Removing splits rar\n"
     ]
    }
   ],
   "source": [
    "# Download HMDB dataset in `data/hmdb51`\n",
    "# It will also download the splits and extract the rar files\n",
    "!bash \"$PYVIDEOAI_DIR/submodules/video_datasets_api/tools/hmdb/download_hmdb.sh\" \"$DATA_DIR/hmdb51\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828ce293-397c-4647-b6ed-7a5ab10743ef",
   "metadata": {},
   "source": [
    "You should see \\*.avi videos in `data/hmdb51/videos`,  \n",
    "and annotations saved in `data/hmdb51/testTrainMulti_7030_splits`.\n",
    "\n",
    "Next, we're extracting videos into frames of images to `data/hmdb51/frames_q5` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "96df3f15-0241-4c4c-88ed-634731078bb8",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 / 6766\n",
      "/home/kiyoon/project/PyVideoAI/data/hmdb51/frames_q5/ride_bike/Learn_How_To_Ride_A_Bike_at_Any_Age_ride_bike_f_cm_np1_fr_med_9\n",
      "Average processing time per segment: 0.13, ETA: 859\n",
      "2 / 6766\n",
      "/home/kiyoon/project/PyVideoAI/data/hmdb51/frames_q5/ride_bike/Learn_How_To_Ride_A_Bike_at_Any_Age_ride_bike_f_cm_np1_ba_med_4\n",
      "Average processing time per segment: 0.09, ETA: 603\n",
      "-----------------------------------------------------------\n",
      "----------OUTPUT SIMPLIFIED FOR NOTEBOOK VIEWERS-----------\n",
      "-----------------------------------------------------------\n",
      "6765 / 6766\n",
      "/home/kiyoon/project/PyVideoAI/data/hmdb51/frames_q5/throw/Compilationknifethrowing_throw_u_nm_np1_ba_med_17\n",
      "Average processing time per segment: 0.05, ETA: 0\n",
      "6766 / 6766\n",
      "/home/kiyoon/project/PyVideoAI/data/hmdb51/frames_q5/throw/Compilationknifethrowing_throw_u_nm_np1_ba_med_0\n",
      "Average processing time per segment: 0.05, ETA: 0\n",
      "😍 All videos successfully extracted\n"
     ]
    }
   ],
   "source": [
    "# Extract videos into frames of images (quality=5, 1 being the best and 31 being the worst)\n",
    "# This will be saved in the fast SSD storage.\n",
    "# You can skip this if you want to use video decoding dataloader.\n",
    "!bash \"$PYVIDEOAI_DIR/submodules/video_datasets_api/tools/hmdb/hmdb_extract_frames.sh\" \"$DATA_DIR/hmdb51/videos\" \"$DATA_DIR/hmdb51/frames_q5\" 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2031b4e7-c16e-4fe1-b9d4-357998d9f2a2",
   "metadata": {},
   "source": [
    "We'll convert the official split labels into the format that our dataloader expects, and we provide code for this.  \n",
    "If you want to know the details of the format, below explains what it is like:\n",
    "\n",
    "**Format used in** `FramesSparsesampleDataloader` in `dataloader/frames_sparsesample_dataloader.py`    \n",
    "**or** `FramesDensesampleDataloader` in `dataloader/frames_densesample_dataloader.py`  \n",
    "\n",
    "```\n",
    "num_classes if multilabel else 0\n",
    "path/to/frames/dir_1/{:05d}.jpg video_id_1 label_1 start_frame_idx_1 end_frame_idx_1\n",
    "path/to/frames/dir_2/{:05d}.jpg video_id_2 label_2 start_frame_idx_2 end_frame_idx_2\n",
    "...\n",
    "path/to/frames/dir_N/{:05d}.jpg video_id_N label_N start_frame_idx_N end_frame_idx_N\n",
    "```\n",
    "\n",
    "### 1. First line: one-hot encoding or not\n",
    "\n",
    "#### TL;DR\n",
    "The first line has to be 0 for single-label classification, and `num_classes` for multi-label classification task.\n",
    "In HMDB-51, it has to be 0.\n",
    "\n",
    "#### Details\n",
    "If the value is greater than 0, the dataloader will one-hot encode the labels and return as array of size `num_classes` (which is necessary for multi-label).  \n",
    "If the value is 0, the dataloader will return the labels as integers (which is for single-label).\n",
    "\n",
    "### 2. Second line ~ end\n",
    "#### Image path\n",
    "This has to be Python-style formattable string, where the {:05d} part will be replaced by the frame index with 5-length zero paddings.  \n",
    "Also, this path can be relative and you can provide `path_prefix` argument to the dataloader.\n",
    "\n",
    "#### Video ID\n",
    "Here, the video_id has to be unique per video, and will be used for multiple-clip evaluation (multicrop evaluation).\n",
    "To elaborate, the accuracy calculator will average the prediction of the videos with the same video ID,  \n",
    "and the dataloader will automatically sample each video into multiple clips in different spatial-temporal location,  \n",
    "so all you need to do is to make sure that the video ID is unique.\n",
    "\n",
    "#### Label\n",
    "One label integer for single-label, and comma-separated labels for multi-label task.\n",
    "\n",
    "#### Start/end frame index\n",
    "Video will be sampled from `start_frame_idx` to `end_frame_idx` (including edge indices).  \n",
    "Since the extraction code above count frames from zero, the `start_frame_idx` has to be always zero in this HMDB-51 example.  \n",
    "Note that when you have long untrimmed video, you can define subclips changing this starting and ending frame index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04283255-b76f-414c-8f6b-881d68db9e17",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Generate splits in the FramesSparsesampleDataloader format (using the official splits)\n",
    "# in `data/hmdb51/splits_frames` directory\n",
    "!bash \"$PYVIDEOAI_DIR/tools/datasets/hmdb_splits_to_csv_frame_extracted.sh\" \"$DATA_DIR/hmdb51/\"{testTrainMulti_7030_splits,frames_q5,splits_frames}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66221963-c15d-4cda-945b-1176a7d1cc1f",
   "metadata": {},
   "source": [
    "Let's take a look at how the actual CSV files look like.  \n",
    "Note that the first line is 0, and the Video IDs are unique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b5c8285a-9562-4d44-b3ca-b141f397191a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "brush_hair/April_09_brush_hair_u_nm_np1_ba_goo_0/{:05d}.jpg 0 0 0 407\n",
      "brush_hair/April_09_brush_hair_u_nm_np1_ba_goo_1/{:05d}.jpg 1 0 0 393\n",
      "brush_hair/April_09_brush_hair_u_nm_np1_ba_goo_2/{:05d}.jpg 2 0 0 321\n",
      "brush_hair/Aussie_Brunette_Brushing_Hair_II_brush_hair_u_nm_np1_ri_med_3/{:05d}.jpg 3 0 0 157\n",
      "brush_hair/Aussie_Brunette_Brushing_Hair_II_brush_hair_u_nm_np2_le_goo_0/{:05d}.jpg 4 0 0 135\n",
      "brush_hair/Aussie_Brunette_Brushing_Hair_II_brush_hair_u_nm_np2_le_goo_1/{:05d}.jpg 5 0 0 357\n",
      "brush_hair/Aussie_Brunette_Brushing_Hair_II_brush_hair_u_nm_np2_le_med_2/{:05d}.jpg 6 0 0 189\n",
      "brush_hair/Blonde_being_brushed_brush_hair_f_nm_np2_ri_med_0/{:05d}.jpg 7 0 0 77\n",
      "brush_hair/Blonde_being_brushed_brush_hair_u_cm_np2_ri_med_1/{:05d}.jpg 8 0 0 272\n"
     ]
    }
   ],
   "source": [
    "# Print 10 lines\n",
    "!head \"$DATA_DIR/hmdb51/splits_frames/train1.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ace24d7-bb75-42ac-91c6-02ef0f05c9ee",
   "metadata": {},
   "source": [
    "Now the data preparation is complete!  \n",
    "In the next tutorial, you'll learn how to run an example code with an example configuration!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}