linear-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning the Linear Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to learn the following linear model:\n",
    "\n",
    "\\begin{gather}\n",
    "X_{n+1} = T X_{n} + D\n",
    "\\end{gather}\n",
    "\n",
    "When:\n",
    "\n",
    "\\begin{gather}\n",
    "T =\n",
    " \\begin{bmatrix} \n",
    "    A_{1} & B_{1} & C_{1} \\\\\n",
    "    A_{2} & B_{2} & C_{2} \\\\\n",
    "    A_{3} & B_{3} & C_{3} \n",
    "    \\end{bmatrix} , \n",
    "     D =\n",
    " \\begin{bmatrix} \n",
    "    D_{1}\\\\\n",
    "    D_{2}\\\\\n",
    "    D_{3} \n",
    "    \\end{bmatrix} , \n",
    "       X_{n} =\n",
    " \\begin{bmatrix} \n",
    "    HR_{n}\\\\\n",
    "    BP_{n}\\\\\n",
    "    RR_{n} \n",
    "    \\end{bmatrix}\n",
    "\\end{gather}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Training\n",
    "With given 3 signals, blood pressure heart rate and respiration rate, from second 0 to n, lets assume the following equations:\n",
    "\n",
    "\\begin{gather}\n",
    " \\begin{bmatrix} HR_{1} \\\\ \\vdots \\\\ HR_{n} \\end{bmatrix}=\n",
    " \\begin{bmatrix} \n",
    "    HR_{0} & BP_{0} & RR_{0} \\\\\n",
    "    \\vdots & \\vdots & \\vdots \\\\\n",
    "    HR_{n-1} & BP_{n-1} & RR_{n-1} \n",
    "    \\end{bmatrix}\n",
    "  \\begin{bmatrix}\n",
    "   A_{1}   \\\\\n",
    "   B_{1}  \\\\\n",
    "   C_{1} \n",
    "   \\end{bmatrix} +\n",
    "   \\begin{bmatrix} D_{1} \\\\ \\vdots \\\\ D_{1} \\end{bmatrix}\n",
    "\\end{gather}\n",
    "\n",
    "\\begin{gather}\n",
    " \\begin{bmatrix} BP_{1} \\\\ \\vdots \\\\ BP_{n} \\end{bmatrix}=\n",
    " \\begin{bmatrix} \n",
    "    HR_{0} & BP_{0} & RR_{0} \\\\\n",
    "    \\vdots & \\vdots & \\vdots \\\\\n",
    "    HR_{n-1} & BP_{n-1} & RR_{n-1} \n",
    "    \\end{bmatrix}\n",
    "  \\begin{bmatrix}\n",
    "   A_{2}   \\\\\n",
    "   B_{2}  \\\\\n",
    "   C_{2} \n",
    "   \\end{bmatrix} +\n",
    "   \\begin{bmatrix} D_{2} \\\\ \\vdots \\\\ D_{2} \\end{bmatrix}\n",
    "\\end{gather}\n",
    "\n",
    "\\begin{gather}\n",
    " \\begin{bmatrix} RR_{1} \\\\ \\vdots \\\\ RR_{n} \\end{bmatrix}=\n",
    " \\begin{bmatrix} \n",
    "    HR_{0} & BP_{0} & RR_{0} \\\\\n",
    "    \\vdots & \\vdots & \\vdots \\\\\n",
    "    HR_{n-1} & BP_{n-1} & RR_{n-1} \n",
    "    \\end{bmatrix}\n",
    "  \\begin{bmatrix}\n",
    "   A_{3}   \\\\\n",
    "   B_{3}  \\\\\n",
    "   C_{3} \n",
    "   \\end{bmatrix} +\n",
    "   \\begin{bmatrix} D_{3} \\\\ \\vdots \\\\ D_{3} \\end{bmatrix}\n",
    "\\end{gather}\n",
    "\n",
    "Each one of the equations above describing a linear model, accordingly we will try to regress the following for i= {1,2,3}:\n",
    "\n",
    "\\begin{gather}\n",
    "  \\begin{bmatrix}\n",
    "   A_{i}   \\\\\n",
    "   B_{i}  \\\\\n",
    "   C_{i} \n",
    "   \\end{bmatrix} , D_{i}\n",
    "\\end{gather}\n",
    "\n",
    "Methods for regression:\n",
    "1. Least Squares (Manually while assuming D=0), for example:\n",
    "\n",
    "\\begin{gather}\n",
    "    \\overline{\n",
    "    \\begin{bmatrix}\n",
    "        A_{1}\\\\B_{1}\\\\C_{1}\n",
    "    \\end{bmatrix}}= \n",
    "    \\left(\n",
    "        \\begin{bmatrix}\n",
    "            HR_{0}&BP_{0}&RR_{0}\\\\ \\vdots&\\vdots&\\vdots\\\\ HR_{n-1} & BP_{n-1} & RR_{n-1}\n",
    "        \\end{bmatrix}^T\n",
    "        \\begin{bmatrix}\n",
    "            HR_{0}&BP_{0}&RR_{0}\\\\ \\vdots&\\vdots&\\vdots\\\\ HR_{n-1} & BP_{n-1} & RR_{n-1}\n",
    "        \\end{bmatrix}\n",
    "    \\right)^{-1}\n",
    "    \\begin{bmatrix}\n",
    "        HR_{0}&BP_{0}&RR_{0}\\\\ \\vdots&\\vdots&\\vdots\\\\ HR_{n-1} & BP_{n-1} & RR_{n-1}\n",
    "    \\end{bmatrix}^T\n",
    "    \\begin{bmatrix}\n",
    "        HR_{1}\\\\ \\vdots\\\\ HR_{n}\n",
    "    \\end{bmatrix}\n",
    "\\end{gather}\n",
    "\n",
    "2. Least Squares (sklearn)\n",
    "\n",
    "3. LassoCV (sklearn)\n",
    "\n",
    "4. RidgeCV (sklearn)\n",
    "\n",
    "manual page for the methods form sklearn library:\n",
    "https://scikit-learn.org/stable/modules/linear_model.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Model = namedtuple('Model', ['T', 'D', 'start_train_point', 'end_train_point', 'patient_id', 'reg_method', 'scaler'])\n",
    "def estimate_T(start_train_point, end_train_point, patient_id, reg_method):\n",
    "\n",
    "    D = np.array([[0],[0],[0]])\n",
    "    \n",
    "    reg1=reg2=reg3=reg4=None\n",
    "    scaler=None\n",
    "    \n",
    "    time, sys_bp_train, dia_bp_train = bp(start_train_point,end_train_point,patient_id,0,0)\n",
    "    time, hr_train = hr(start_train_point,end_train_point,patient_id,0,0)\n",
    "    time, rr_train = rr(start_train_point,end_train_point,patient_id,0,0)\n",
    "    \n",
    "    # scaling\n",
    "    data_train = np.column_stack((hr_train, sys_bp_train, rr_train, dia_bp_train))\n",
    "    scaler = StandardScaler()\n",
    "    scaler.fit(data_train)\n",
    "    data_train_scaled = scaler.transform(data_train)\n",
    "    sys_bp_train = data_train_scaled[:,1]\n",
    "    hr_train = data_train_scaled[:,0]\n",
    "    rr_train = data_train_scaled[:,2]\n",
    "    dia_bp_train = data_train_scaled[:,3]\n",
    "    \n",
    "    # x is the 3 signals data from 0 to n-1\n",
    "    x = np.column_stack((hr_train, sys_bp_train, rr_train, dia_bp_train))\n",
    "    x = np.delete(x, -1, axis=0)\n",
    "\n",
    "    # y_i is the data from 1 to n for the i signal\n",
    "    y_sys_bp = np.delete(sys_bp_train, 0)\n",
    "    y_rr = np.delete(rr_train, 0)\n",
    "    y_hr = np.delete(hr_train, 0)\n",
    "    y_dia_bp = np.delete(dia_bp_train, 0)\n",
    "    \n",
    "    if reg_method == 1:\n",
    "        hr_coef = np.dot(np.linalg.inv(np.dot(x.transpose(),x)),np.dot(x.transpose(),y_hr))\n",
    "        sys_bp_coef = np.dot(np.linalg.inv(np.dot(x.transpose(),x)),np.dot(x.transpose(),y_sys_bp))\n",
    "        rr_coef = np.dot(np.linalg.inv(np.dot(x.transpose(),x)),np.dot(x.transpose(),y_rr))\n",
    "        dia_bp_coef = np.dot(np.linalg.inv(np.dot(x.transpose(),x)),np.dot(x.transpose(),y_dia_bp))\n",
    "    \n",
    "    else:\n",
    "        if reg_method == 2:\n",
    "            reg1 = linear_model.LinearRegression(fit_intercept=False)\n",
    "            reg2 = linear_model.LinearRegression(fit_intercept=False)\n",
    "            reg3 = linear_model.LinearRegression(fit_intercept=False)\n",
    "            reg4 = linear_model.LinearRegression(fit_intercept=False)\n",
    "            \n",
    "        if reg_method == 3:\n",
    "            reg1 = linear_model.LassoCV(cv = 5, fit_intercept=False)\n",
    "            reg2 = linear_model.LassoCV(cv = 5, fit_intercept=False)\n",
    "            reg3 = linear_model.LassoCV(cv = 5, fit_intercept=False)\n",
    "            reg4 = linear_model.LassoCV(cv = 5, fit_intercept=False)\n",
    "            \n",
    "\n",
    "        if reg_method == 4:\n",
    "            reg1 = linear_model.RidgeCV(fit_intercept=False)\n",
    "            reg2 = linear_model.RidgeCV(fit_intercept=False)\n",
    "            reg3 = linear_model.RidgeCV(fit_intercept=False)\n",
    "            reg4 = linear_model.RidgeCV(fit_intercept=False)\n",
    "        \n",
    "        reg1.fit(x, y_hr)\n",
    "        hr_coef = reg1.coef_\n",
    "        hr_intercept = reg1.intercept_\n",
    "        \n",
    "        reg2.fit(x, y_sys_bp)\n",
    "        sys_bp_coef = reg2.coef_\n",
    "        sys_bp_intercept = reg2.intercept_\n",
    "        \n",
    "        reg3.fit(x, y_rr)\n",
    "        rr_coef = reg3.coef_\n",
    "        rr_intercept = reg3.intercept_\n",
    "        \n",
    "        reg4.fit(x, y_dia_bp)\n",
    "        dia_bp_coef = reg4.coef_\n",
    "        dia_bp_intercept = reg4.intercept_\n",
    "        \n",
    "        D = np.row_stack((hr_intercept, sys_bp_intercept, rr_intercept, dia_bp_intercept))\n",
    "        \n",
    "    D = scaler.mean_.reshape(4,1)\n",
    "    T = np.row_stack((hr_coef, sys_bp_coef, rr_coef, dia_bp_coef))\n",
    "    model = Model(T, D, start_train_point, end_train_point, patient_id, reg_method, scaler)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Testing\n",
    "\n",
    "Lets test the regressed matrix by caculating the following:\n",
    "\n",
    "\\begin{gather}\n",
    "\\overline{X}_{n+1} = \\overline{T} X_{n} + \\overline{D}\n",
    "\\end{gather}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Prediction = namedtuple('Prediction', ['time', 'data_test', 'data_pred', 'patient_id', 'model', 'white_noise_rank'])\n",
    "def test_T(model, start_point, end_point, patient_id):\n",
    "\n",
    "    time, sys_bp_test, dia_bp_test = bp(start_point,end_point,patient_id,0,0)\n",
    "    time, hr_test = hr(start_point,end_point,patient_id,0,0)\n",
    "    time, rr_test = rr(start_point,end_point,patient_id,0,0)\n",
    "    \n",
    "    # scaling\n",
    "    data_test = np.column_stack((hr_test, sys_bp_test, rr_test, dia_bp_test))\n",
    "    scaler = StandardScaler()\n",
    "    scaler.fit(data_test)\n",
    "    data_test_scaled = scaler.transform(data_test)\n",
    "    \n",
    "    data_pred = np.dot(model.T, data_test_scaled.transpose())# + np.repeat(model.D, len(time), axis=1)\n",
    "    \n",
    "    # reverse scaling\n",
    "    data_pred = model.scaler.inverse_transform(data_pred.transpose())\n",
    "    data_pred = data_pred.transpose()\n",
    "\n",
    "    time = np.delete(time, 0)\n",
    "    data_test = np.delete(data_test.transpose(), 0, axis=1)\n",
    "    data_pred = np.delete(data_pred, -1, axis=1)\n",
    "    \n",
    "    white_noise_rank = white_noise(data_test, data_pred)\n",
    "    prediction = Prediction(time, data_test, data_pred, patient_id, model, white_noise_rank)\n",
    "    \n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# White Noise testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "White_noise_rank = namedtuple('White_noise_rank', ['mean', 'alpha_ac', 'left_mean', 'right_mean'])\n",
    "def white_noise(data_test, data_pred):\n",
    "    error = data_test - data_pred\n",
    "    left_mean = right_mean = mean = 0\n",
    "    alpha_ac = -np.inf # autocorelation\n",
    "    ranks = []\n",
    "    \n",
    "    for signal in [0,1,2,3]:\n",
    "        mean = np.abs(np.mean(error[signal]))\n",
    "        left_mean = np.abs(np.mean(error[signal][0:(int)(len(error[signal])/2)]))\n",
    "        right_mean = np.abs(np.mean(error[signal][(int)(len(error[signal])/2):]))                \n",
    "                \n",
    "        for alpha_ in [0.15, 0.13, 0.11, 0.09, 0.07, 0.05, 0.03, 0.01]:\n",
    "            h, pV, Q, cV = lbqtest(error[signal], range(1, min(20, len(error[signal]))), alpha=alpha_)\n",
    "            if not any(h):      \n",
    "                alpha_ac = alpha_\n",
    "                break;\n",
    "        white_noise_rank = White_noise_rank(mean, alpha_ac, left_mean, right_mean)\n",
    "        ranks.append(white_noise_rank)\n",
    "    \n",
    "    return ranks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Printing Prediction Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_prediction_results(predictions):\n",
    "    if(type(predictions) != list):\n",
    "        predictions = [[predictions]]\n",
    "    \n",
    "    start = np.inf\n",
    "    end = -np.inf\n",
    "    for prediction_one_model in predictions:\n",
    "        for prediction in prediction_one_model:\n",
    "            start = min(start, prediction.time[0])\n",
    "            end = max(end, prediction.time[-1])\n",
    "    time_total, sys_bp_original_total, dia_bp_original_total = bp(start,end,predictions[0][0].patient_id,0,0)\n",
    "    time_total, hr_original_total = hr(start,end,predictions[0][0].patient_id,0,0)\n",
    "    time_total, rr_original_total = rr(start,end,predictions[0][0].patient_id,0,0)\n",
    "    \n",
    "    error_color = 'gray'\n",
    "    original_color = 'black'\n",
    "    prediction_colors = ['C0', 'C1', 'C2', 'C3', 'C4']\n",
    "    \n",
    "    fig, ax = plt.subplots(nrows=4, ncols=1, figsize=(24, 20))\n",
    "    \n",
    "    ax[0].set_title('Heart Rate')\n",
    "    ax[0].set_xlabel('Time [sec]')\n",
    "    ax[0].set_ylabel('Heart Rate [bpm]')\n",
    "    ax[0].plot(time_total, hr_original_total, linewidth=2, color=original_color, label='Data')    \n",
    "    ax2_hr = ax[0].twinx()\n",
    "    ax2_hr.set_ylabel('Error [bpm]', color=error_color)\n",
    "    ax2_hr.tick_params(axis='y', labelcolor=error_color)\n",
    "\n",
    "    ax[1].set_title('Systolic Blood Pressure')\n",
    "    ax[1].set_xlabel('Time [sec]')\n",
    "    ax[1].set_ylabel('Systolic Blood Pressure [mmHg]')\n",
    "    ax[1].plot(time_total, sys_bp_original_total, linewidth=2, color=original_color, label='Data')\n",
    "    ax2_bp = ax[1].twinx()\n",
    "    ax2_bp.set_ylabel('Error [mmHg]', color=error_color)\n",
    "    ax2_bp.tick_params(axis='y', labelcolor=error_color)\n",
    "\n",
    "\n",
    "    ax[2].set_title('Respiration Rate')\n",
    "    ax[2].set_xlabel('Time [sec]')\n",
    "    ax[2].set_ylabel('Respiration Rate [Hz]')\n",
    "    ax[2].plot(time_total, rr_original_total, linewidth=2, color=original_color, label='Data')\n",
    "    ax2_rr = ax[2].twinx()\n",
    "    ax2_rr.set_ylabel('Error [Hz]', color=error_color)\n",
    "    ax2_rr.tick_params(axis='y', labelcolor=error_color)\n",
    "    \n",
    "    ax[3].set_title('Diastolic Blood Pressure')\n",
    "    ax[3].set_xlabel('Time [sec]')\n",
    "    ax[3].set_ylabel('Diastolic Blood Pressure [mmHg]')\n",
    "    ax[3].plot(time_total, dia_bp_original_total, linewidth=2, color=original_color, label='Data')\n",
    "    ax2_bp = ax[3].twinx()\n",
    "    ax2_bp.set_ylabel('Error [mmHg]', color=error_color)\n",
    "    ax2_bp.tick_params(axis='y', labelcolor=error_color)\n",
    "    \n",
    "    model_index = 0\n",
    "    for prediction_one_model in predictions:\n",
    "        model_index = model_index + 1\n",
    "        print(\"*******************************************************************************************************\")\n",
    "        print(\"Model Number: \", model_index)\n",
    "        print(\"Estimation Method: \", prediction_one_model[0].model.reg_method)\n",
    "        print(\"T=\\n\", prediction_one_model[0].model.T)\n",
    "        print(\"D=\\n\", prediction_one_model[0].model.D)\n",
    "        print(\"Training Period: [%s, %s]\" % (prediction_one_model[0].model.start_train_point, prediction_one_model[0].model.end_train_point))\n",
    "        print(\"Training based on %s data\" % (prediction_one_model[0].model.patient_id))\n",
    "        for prediction in prediction_one_model:\n",
    "            print(\"Testing the model based on %s data on the period [%s, %s]\"  % (prediction.patient_id, prediction.time[0], prediction.time[-1]))\n",
    "            print('Mean squared error: %.2f'\n",
    "              % mean_squared_error(prediction.data_test, prediction.data_pred))\n",
    "            # The coefficient of determination: 1 is perfect prediction\n",
    "            print('Coefficient of determination: %.2f'\n",
    "              % r2_score(prediction.data_test, prediction.data_pred))\n",
    "            print(\"The Noise Is White: \")\n",
    "            for i in [0,1,2]:\n",
    "                print(prediction.white_noise_rank[i])\n",
    "\n",
    "            ax[0].plot(prediction.time, prediction.data_pred[0], linewidth=2, color=prediction_colors[model_index-1], label=\"Predicted by model \" + str(model_index))\n",
    "            ax2_hr.plot(prediction.time, prediction.data_test[0] - prediction.data_pred[0], '--', color=error_color, linewidth=0.5)\n",
    "            \n",
    "            ax[1].plot(prediction.time, prediction.data_pred[1], linewidth=2, color=prediction_colors[model_index-1], label=\"Predicted by model \" + str(model_index))\n",
    "            ax2_bp.plot(prediction.time, prediction.data_test[1] - prediction.data_pred[1], ':', color=error_color, linewidth=0.5)\n",
    "            \n",
    "            ax[2].plot(prediction.time, prediction.data_pred[2], linewidth=2, color=prediction_colors[model_index-1], label=\"Predicted by model \" + str(model_index))\n",
    "            ax2_rr.plot(prediction.time, prediction.data_test[2] - prediction.data_pred[2], '--', color=error_color, linewidth=0.5)\n",
    "            \n",
    "            ax[3].plot(prediction.time, prediction.data_pred[3], linewidth=2, color=prediction_colors[model_index-1], label=\"Predicted by model \" + str(model_index))\n",
    "            ax2_bp.plot(prediction.time, prediction.data_test[3] - prediction.data_pred[3], ':', color=error_color, linewidth=0.5)\n",
    "        print(\"*******************************************************************************************************\")\n",
    "            \n",
    "    ax[0].legend(loc='best')\n",
    "    ax[1].legend(loc='best')\n",
    "    ax[2].legend(loc='best')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}