data_split_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Splitting & Analysis of Common Voice 8.0\n",
    "*By Dragoș Alexandru Bălan*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this Jupyter Notebook you can find the code related to the splitting of data into the different subsets used throughout my experiments, as well as the code used for analyzing the distribution of data according to sex and age for each split.\n",
    "\n",
    "**IMPORTANT:** The experiments have been conducted in a Linux environment. I have no knowledge on how the notebook works on Windows or MacOS, so no support is provided for those operating systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dependencies\n",
    "First, the dependencies required to run this notebook are found below. I recommend to first create and activate a virtual environment, then run the commands below in that environment. You will only need to run it once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install librosa\n",
    "!pip install pandas\n",
    "!pip install numpy\n",
    "!pip install datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating recording lengths in train, test, and dev splits of Common Voice (Experiment 1)\n",
    "I personally suggest using a local version of the dataset since splitting the data later on will require it. However, if you are interested in using only the train, test, and dev splits from Common Voice and not have to download the entire dataset, then you can also code related to Huggingface. Be advised that using Huggingface will require you to create an account on their platform. More details about that can be found below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Local dataset downloaded from Common Voice website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 573
    },
    "id": "Gh1GQ-y77gop",
    "outputId": "a0eb3cc7-8857-4d13-bbf3-d9752f67336d"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from librosa import load\n",
    "\n",
    "# CHANGE THE PATH IF NEEDED!!! The path assumed here is the\n",
    "# parent directory of the folder that contains the dataset\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df_train = pd.read_csv(path + 'train.tsv', sep='\\t')\n",
    "df_test = pd.read_csv(path + 'test.tsv', sep='\\t')\n",
    "df_dev = pd.read_csv(path + 'dev.tsv', sep='\\t')\n",
    "\n",
    "# Initialize an empty array to which we will append the\n",
    "# length of each recording in seconds\n",
    "lengths = []\n",
    "for i, row in df_train.iterrows():\n",
    "  # Load the audio from the path defined in the TSV\n",
    "  aud, sr = load(path + 'clips/' + row['path'], sr=None)\n",
    "  # Divide the length of the audio array by the sampling rate to\n",
    "  # get the number of seconds of the recording.\n",
    "  lengths.append(len(aud) / sr)\n",
    "# Lastly, assign the calculated array of lengths in seconds to\n",
    "# a new column named 'length'\n",
    "df_train['length'] = lengths\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_test.iterrows():\n",
    "  aud, sr = load(path + 'clips/' + row['path'], sr=None)\n",
    "  lengths.append(len(aud) / sr)\n",
    "\n",
    "df_test['length'] = lengths\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_dev.iterrows():\n",
    "  aud, sr = load(path + 'clips/' + row['path'], sr=None)\n",
    "  lengths.append(len(aud) / sr)\n",
    "\n",
    "df_dev['length'] = lengths\n",
    "\n",
    "# Save the new dataframes that contain an extra 'length' column as CSVs\n",
    "# No index required since the filename can act as an index\n",
    "df_train.to_csv(path + 'train_len.csv', index=False)\n",
    "df_test.to_csv(path + 'test_len.csv', index=False)\n",
    "df_dev.to_csv(path + 'dev_len.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset from Huggingface\n",
    "\n",
    "In case you don't have the dataset cached already, you will need to login to Huggingface so an authentication token can be stored locally. First, you will need to register for an account here: https://huggingface.co/join\n",
    "\n",
    "After creating an account, you will need an access token in order to login to Huggingface on this notebook. You can create one by going to the profile picture icon on the top-right of the website -> Settings -> Access Tokens. In here, click on 'New token'. Input any name you want but **make sure to set the role to 'write'**.\n",
    "\n",
    "Once your access token is generated, run the cell below, copy the access token, and paste it when requested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 331,
     "referenced_widgets": [
      "8d93c23aa9d640dab17c9c7df5e609ae",
      "37c508ec74924cc5a37711423daf9eb0",
      "4b939608195547fd8e21228779a939e3",
      "58257f72024144c8a437c2658970dd20",
      "af920b1a26554201afbf9dff71fb225a",
      "7c3a5f65b68940039b3aa70e24abbc4f",
      "ea0037a16a8a4d45b60e90b2ef6e76ea",
      "dacf9d061d9748178d04cec7d8431841",
      "97a27d14f73941639fff55a70dcbeeb5",
      "79db6b7cc29a4b77acec3c72fe41a810",
      "c087cd01d03c46639be31c7b6006832c",
      "3b3e01d4e06f466c9accae86263a6f15",
      "e9b379e152c94b44955360781c7c296d",
      "d6576544d1cc4f0fac79226055896b7f",
      "7e451f718bdf42c39549e2438b3c2334",
      "32d0626444cb43b4a872f58f60320f8d",
      "f3750fe3d5ef453cb252e537c6b0cd8b"
     ]
    },
    "id": "mlMSH3T3EazV",
    "outputId": "f2866a18-0494-47d0-ff22-7a30f8b6988d",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you are logged in, you can download and cache the dataset and its splits by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "# Load the Common Voice 8.0 splits from Huggingface. I recommend keeping this in a separate\n",
    "# cell since loading datasets from Huggingface can be quite slow. After loading, the datasets\n",
    "# can be used and modified throughout the notebook\n",
    "cv_train = load_dataset(\"mozilla-foundation/common_voice_8_0\", \"fy-NL\", split=\"train\", use_auth_token=True)\n",
    "cv_test = load_dataset(\"mozilla-foundation/common_voice_8_0\", \"fy-NL\", split=\"test\")\n",
    "cv_dev = load_dataset(\"mozilla-foundation/common_voice_8_0\", \"fy-NL\", split=\"validation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df_train = pd.DataFrame(cv_train)\n",
    "df_test = pd.DataFrame(cv_test)\n",
    "df_dev = pd.DataFrame(cv_dev)\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_train.iterrows():\n",
    "  # Calculate the length of the recording in seconds\n",
    "  # We do so by dividing the length of the audio array to the\n",
    "  # sampling rate\n",
    "  lengths.append(len(row['audio']['array']) / row['audio']['array'])\n",
    "# Lastly, assign the calculated lengths in seconds to a new column\n",
    "# named 'length'\n",
    "df_train['length'] = lengths\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_test.iterrows():\n",
    "  lengths.append(len(row['audio']['array']) / row['audio']['array'])\n",
    "\n",
    "df_test['length'] = lengths\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_dev.iterrows():\n",
    "  lengths.append(len(row['audio']['array']) / row['audio']['array'])\n",
    "\n",
    "df_dev['length'] = lengths\n",
    "\n",
    "df_train.to_csv(path + 'train_len.csv', index=False)\n",
    "df_test.to_csv(path + 'test_len.csv', index=False)\n",
    "df_dev.to_csv(path + 'dev_len.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating speaker statistics based on sex and age from train, test, and dev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df_train = pd.read_csv(path + 'train.csv')\n",
    "df_test = pd.read_csv(path + 'test.csv')\n",
    "df_dev = pd.read_csv(path + 'dev.csv')\n",
    "# Replace undefined values in the dataframes with string 'unknown'\n",
    "df_train = df_train.replace(np.nan, 'unknown', regex=True)\n",
    "df_test = df_test.replace(np.nan, 'unknown', regex=True)\n",
    "df_dev = df_dev.replace(np.nan, 'unknown', regex=True)\n",
    "\n",
    "# Create the dictionary with keys based on age and sex (column name is 'gender')\n",
    "len_test = {g:{el:0 for el in df_test['age'].unique()} for g in df_test['gender'].unique()}\n",
    "for i, row in df_test.iterrows():\n",
    "  # For each recording, add the length to the corresponding sum of sex and age combination\n",
    "  len_test[row['gender']][row['age']] += row['length']\n",
    "# Create a dataframe out of the statistics calculated\n",
    "df = pd.DataFrame(len_test)\n",
    "# Save the dataframe to a CSV\n",
    "df.to_csv(path + 'test_stats.csv')\n",
    "\n",
    "\n",
    "len_dev = {g:{el:0 for el in df_dev['age'].unique()} for g in df_dev['gender'].unique()}\n",
    "for i, row in df_dev.iterrows():\n",
    "  len_dev[row['gender']][row['age']] += row['length']\n",
    "df = pd.DataFrame(len_dev)\n",
    "df.to_csv(path + 'dev_stats.csv')\n",
    "\n",
    "\n",
    "len_train = {g:{el:0 for el in df_train['age'].unique()} for g in df_train['gender'].unique()}\n",
    "for i, row in df_train.iterrows():\n",
    "  len_train[row['gender']][row['age']] += row['length']\n",
    "df = pd.DataFrame(len_train)\n",
    "df.to_csv(path + 'train_stats.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiments 2, 6, 7 train dataset\n",
    "\n",
    "This dataset contains all of the validated data from Common Voice 8.0 (~50h) except for the data that is found in the test or dev splits from above. The amount of data of this dataset sums up to ~41h making it the largest train dataset used in my experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting the train set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "# Load the TSV files corresponding to all of the validated data (validated.tsv),\n",
    "# the test split (test.tsv), and the dev split (dev.tsv)\n",
    "df_large = pd.read_csv(path + 'validated.tsv', sep='\\t')\n",
    "df_test = pd.read_csv(path + 'test.tsv', sep='\\t')\n",
    "df_dev = pd.read_csv(path + 'dev.tsv', sep='\\t')\n",
    "\n",
    "print('Number of entries before removing any data: ' + str(len(df_large)))\n",
    "\n",
    "# Removes the entries of test from validated\n",
    "df_large = pd.merge(df_large, df_test, on=list(df_test.columns), how='outer', indicator=True)\\\n",
    "       .query(\"_merge != 'both'\")\\\n",
    "       .drop('_merge', axis=1)\\\n",
    "       .reset_index(drop=True)\n",
    "\n",
    "print('After removing data from test: ' + str(len(df_large)))\n",
    "# Removes the entries of dev from validated - test\n",
    "df_large = pd.merge(df_large, df_dev, on=list(df_dev.columns), how='outer', indicator=True)\\\n",
    "       .query(\"_merge != 'both'\")\\\n",
    "       .drop('_merge', axis=1)\\\n",
    "       .reset_index(drop=True)\n",
    "    \n",
    "print('After removing data from test + dev (the resulting dataset): ' + str(len(df_large)))\n",
    "# MAKE SURE TO CHANGE THE PATH. You must insert an absolute path here so that, \n",
    "# during training, the XLS-R model will know where to look for the audio\n",
    "df_large['path'] = '/scratch/s3944867/cv-corpus-8.0-2022-01-19/fy-NL/clips/' + df_large['path']\n",
    "df_large['sentence'] = '\"' + df_large['sentence'] + '\"'\n",
    "# Save the resulting dataset\n",
    "df_large.to_csv(path + 'train_large.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating length of each recording\n",
    "\n",
    "**IMPORTANT:** This will take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from librosa import load\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df_train = pd.read_csv(path + 'train_large.csv')\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_train.iterrows():\n",
    "  aud, sr = load(row['path'], sr=None)\n",
    "  lengths.append(len(aud) / sr)\n",
    "\n",
    "df_train['length'] = lengths\n",
    "df_train.to_csv(path + 'train_large_len.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistics about speakers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df_train = pd.read_csv(path + 'train_large_len.csv')\n",
    "\n",
    "df_train = df_train.replace(np.nan, 'unknown', regex=True)\n",
    "\n",
    "len_train = {g:{el:0 for el in df_train['age'].unique()} for g in df_train['gender'].unique()}\n",
    "for i, row in df_train.iterrows():\n",
    "  len_train[row['gender']][row['age']] += row['length']\n",
    "df = pd.DataFrame(len_train)\n",
    "df.to_csv(path + 'train_large_stats.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiments 3,4,5 datasets (10h, 1h, 10m splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the dataset into 10h, 1h, and 10m\n",
    "\n",
    "Run all the cells below so that no issue regarding undefined variables or packages are encountered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uG8kX4_RWUCN",
    "outputId": "5aae13aa-2030-4a33-81db-bb3805d71221"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df = pd.read_csv(path + 'train_large_len.csv')\n",
    "# Randomly sample data such that it adds up to ~10 min\n",
    "df_10m = df.sample(frac=0.0041) # corresponds to ~10 min\n",
    "# Value should be very close to 600 (60s * 10m)\n",
    "print(np.sum(df_10m['length']))\n",
    "# Remove the 'length' column so that the file can be used during training\n",
    "df_10m = df_10m.drop(['length'], axis=1)\n",
    "\n",
    "df_10m.to_csv(path + 'train_10m.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_1h = df.sample(frac=0.0246) # corresponds to ~1 h\n",
    "# Value should be very close to 3600 (3600s in an hour)\n",
    "print(np.sum(df_1h['length']))\n",
    "df_1h = df_1h.drop(['length'], axis=1)\n",
    "\n",
    "df_1h.to_csv(path + 'train_1h.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_10h = df.sample(frac=0.244) # corresponds to ~10 h\n",
    "# Value should be very close to 36000 (3600s * 10h)\n",
    "print(np.sum(df_10h['length']))\n",
    "df_10h = df_10h.drop(['length'], axis=1)\n",
    "\n",
    "df_10h.to_csv(path + 'train_10h.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating lengths & Analyzing the 10h, 1h, and 10m train splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from librosa import load\n",
    "\n",
    "path = './cv-corpus-8.0-2022-01-19/fy-NL/'\n",
    "df_10m = pd.read_csv(path + 'train_10m.csv')\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_10m.iterrows():\n",
    "  aud, sr = load(row['path'], sr=None)\n",
    "  lengths.append(len(aud) / sr)\n",
    "\n",
    "df_10m['length'] = lengths\n",
    "df_10m = df_10m.replace(np.nan, 'unknown', regex=True)\n",
    "\n",
    "\n",
    "len_10m = {g:{el:0 for el in df_10m['age'].unique()} for g in df_10m['gender'].unique()}\n",
    "for i, row in df_10m.iterrows():\n",
    "  len_10m[row['gender']][row['age']] += row['length']\n",
    "df = pd.DataFrame(len_10m)\n",
    "df_sum = df.sum(numeric_only = True)\n",
    "df.to_csv(path + 'train_10m_stats.csv')\n",
    "\n",
    "\n",
    "df_1h = pd.read_csv(path + 'train_1h.csv')\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_1h.iterrows():\n",
    "  aud, sr = load(row['path'], sr=None)\n",
    "  lengths.append(len(aud) / sr)\n",
    "\n",
    "df_1h['length'] = lengths\n",
    "df_1h = df_1h.replace(np.nan, 'unknown', regex=True)\n",
    "\n",
    "\n",
    "len_1h = {g:{el:0 for el in df_1h['age'].unique()} for g in df_1h['gender'].unique()}\n",
    "for i, row in df_1h.iterrows():\n",
    "  len_1h[row['gender']][row['age']] += row['length']\n",
    "df = pd.DataFrame(len_1h)\n",
    "df_sum = df.sum(numeric_only = True)\n",
    "df.to_csv(path + 'train_1h_stats.csv')\n",
    "\n",
    "\n",
    "df_10h = pd.read_csv(path + 'train_10h.csv')\n",
    "\n",
    "lengths = []\n",
    "for i, row in df_10h.iterrows():\n",
    "  aud, sr = load(row['path'], sr=None)\n",
    "  lengths.append(len(aud) / sr)\n",
    "\n",
    "df_10h['length'] = lengths\n",
    "df_10h = df_10h.replace(np.nan, 'unknown', regex=True)\n",
    "\n",
    "\n",
    "len_10h = {g:{el:0 for el in df_10h['age'].unique()} for g in df_10h['gender'].unique()}\n",
    "for i, row in df_10h.iterrows():\n",
    "  len_10h[row['gender']][row['age']] += row['length']\n",
    "df = pd.DataFrame(len_10h)\n",
    "df_sum = df.sum(numeric_only = True)\n",
    "df.to_csv(path + 'train_10h_stats.csv')"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}