kin_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 301,
   "id": "82e47c98",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "from scipy import signal\n",
    "from scipy.fft import fft, fftfreq\n",
    "from sklearn import preprocessing\n",
    "from sklearn.preprocessing import MinMaxScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe0a756",
   "metadata": {},
   "source": [
    "## Initial pre-processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "id": "3d7df82c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# sampling rate\n",
    "FS = 100 \n",
    "\n",
    "# import and clean data\n",
    "path_kin = 'data/subject1_smile_1_landmarks.csv'\n",
    "raw = pd.read_csv(path_kin)\n",
    "\n",
    "raw['time'] = raw['Frame_number'] / FS\n",
    "raw = raw.iloc[::-1]\n",
    "raw = raw.set_index('time')\n",
    "\n",
    "#create new df for storing processed data\n",
    "data = pd.DataFrame(index = raw.index)\n",
    "\n",
    "# Find displacement of target markers\n",
    "pos_top = raw[['landmark_62_x','landmark_62_y']].to_numpy()\n",
    "pos_bottom = raw[['landmark_66_x','landmark_66_y']].to_numpy()\n",
    "data['deltaV'] = np.linalg.norm(pos_top - pos_bottom, axis=1)\n",
    "\n",
    "pos_left = raw[['landmark_51_x','landmark_51_y']].to_numpy()\n",
    "pos_right = raw[['landmark_57_x','landmark_57_y']].to_numpy()\n",
    "data['deltaH'] = np.linalg.norm(pos_right - pos_left, axis=1)\n",
    "\n",
    "# scale data from 0 to 1\n",
    "scaler = MinMaxScaler()\n",
    "data.deltaH = scaler.fit_transform(data.deltaH.values.reshape(-1,1))\n",
    "data.deltaV = scaler.fit_transform(data.deltaV.values.reshape(-1,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41bc5aa8",
   "metadata": {},
   "source": [
    "## 1. Compare lowpass filter at different cuttoff frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 315,
   "id": "6a5f3372",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_low = ['Raw','20 Hz*','22.5 Hz','25_Hz','30 Hz','32.5 Hz','35 Hz']\n",
    "Vlow = [(data.deltaV.values)] # adding raw data\n",
    "Hlow = [data.deltaH.values] # adding raw data\n",
    "\n",
    "passes = [20,22.5,25,30,32.5,35]\n",
    "\n",
    "ORDER = 2\n",
    "FS = 100 \n",
    "lcoeff = []\n",
    "#cols = []\n",
    "for p in passes:\n",
    "    lcoeff.append(sp.signal.bessel(ORDER, Wn = p/(FS / 2), btype = 'lowpass'))\n",
    "    #cols.append(f'{p} Hz')\n",
    "\n",
    "\n",
    "for c in range(len(lcoeff)): # will apply first coeff first to both deltaV/H, then append \n",
    "    #print(c)\n",
    "    Vlow.append(sp.signal.filtfilt(lcoeff[c][0],lcoeff[c][1],data['deltaV']))\n",
    "    Hlow.append(sp.signal.filtfilt(lcoeff[c][0],lcoeff[c][1],data['deltaH']))\n",
    "\n",
    "Vlow = pd.DataFrame(np.asarray(Vlow).transpose(),columns = cols_low) \n",
    "Hlow = pd.DataFrame(np.asarray(Hlow).transpose(),columns = cols_low) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f76f491",
   "metadata": {},
   "source": [
    "##### skipme\n",
    "%matplotlib notebook\n",
    "plot(Vlow)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a71052",
   "metadata": {},
   "source": [
    "Observation:\n",
    "- All of the filters give similar trends for temporal resolution. The 25 Hz signal seems to retain more shape in when activating versus others. \n",
    "- None of the lowpass filters performed very well in removing noise when the subject is holding the 'smile' position. \n",
    "- The 25 Hz filter collects more relevant information regarding motor neuron activation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfbf2ed0",
   "metadata": {},
   "source": [
    "## 2. Find ideal kernel size for median filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 305,
   "id": "cb927df1",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_med = ['Raw','17 N','19 N','21_N*','23_N','25 N']\n",
    "Vmed = [(data.deltaV.values)] # adding raw data\n",
    "Hmed = [(data.deltaH.values)] # adding raw data\n",
    "\n",
    "ksize = [17,19,21,23,25]\n",
    "\n",
    "FS = 100 \n",
    "lcoeff = []\n",
    "#cols = []\n",
    "for k in range(len(ksize)):\n",
    "    Vmed.append(sp.signal.medfilt(data['deltaV'],ksize[k]))\n",
    "    Hmed.append(sp.signal.medfilt(data['deltaH'],ksize[k]))\n",
    "\n",
    "Vmed = pd.DataFrame(np.asarray(Vmed).transpose(),columns = cols_med) \n",
    "Hmed = pd.DataFrame(np.asarray(Hmed).transpose(),columns = cols_med) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "040a9d75",
   "metadata": {},
   "source": [
    "##### skipme\n",
    "%matplotlib notebook\n",
    "plot(Hmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7b2797d",
   "metadata": {},
   "source": [
    "Observation: \n",
    "- little difference in activation times between kernel sizes. Median filter is more appropriate to eliminate outliers in the data. Suggest it has more ability at performing with better temporal resolution versus spatial resolution, as some outliers can still be seen, especially with the lower end (17) and higher-end (25) kernel sizes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f10feb0",
   "metadata": {},
   "source": [
    "## 3. Compare filter (1), (2), and original filter selected in paper. \n",
    "selecting lowpass 25 Hz and median k = 23"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "id": "987c71ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "V = pd.DataFrame(index=data.index)\n",
    "H = pd.DataFrame(index=data.index)\n",
    "\n",
    "V['Raw'] = Vmed['Raw'].values \n",
    "V['Median'] = Vmed['23_N'].values\n",
    "V['Lowpass'] = Vlow['25_Hz'].values\n",
    "\n",
    "H['Raw'] = Hmed['Raw'].values \n",
    "H['Median'] = Hmed['23_N'].values\n",
    "H['Lowpass'] = Hlow['25_Hz'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec39fe79",
   "metadata": {},
   "source": [
    "##### skipme\n",
    "%matplotlib notebook\n",
    "plot(V)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9fef919",
   "metadata": {},
   "source": [
    "## Extra: Plotting function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "id": "cc013396",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot(df):\n",
    "    cols = list(df.columns)\n",
    "    for col in df.columns:\n",
    "        plt.plot(df.index,df[col],'--o')\n",
    "        plt.grid()\n",
    "        plt.title('Filter Comparison')\n",
    "        plt.xlabel('Sample')\n",
    "        plt.ylabel('Normalized Distance')\n",
    "        plt.legend(cols,loc='best')\n",
    "        plt.grid()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}