ALiu-Ferrara_Assign 5.Rmd

---
title: "IS 605 FUNDAMENTALS OF COMPUTATIONAL MATHEMATICS - Week 5"
author: "Ann Liu-Ferrara"
date: "March 2, 2017"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

## Problem Set 1

In this problem set we'll work out some properties of the least squares solution that we
reviewed in the weekly readings. Consider the unsolvable system Ax = b as given below:

\[ \left[  \begin{array}{cc}
    1       & 0  \\
    1       & 1  \\
    1       & 3  \\
    1       & 4
  \end{array} \right]
%
\left[  \begin{array}{cc}
    x1  \\
    x2  
  \end{array} \right]
=
\left[  \begin{array}{cc}
    0  \\
    8  \\
    8  \\
    20
\end{array} \right]
\]

Write R markdown script to compute $A^TA$ and $A^Tb$.

```{r echo = TRUE}
A <- matrix(c(1, 1, 1, 1, 0, 1, 3, 4), nrow = 4)
b <- c(0, 8, 8, 20)

# compute A^TA and A^Tb
(ata <- t(A) %*% A)
(atb <- t(A) %*% b)

```

Solve for $\hat{x}$ in R using the above two computed matrices.

As $A^TA$ $\hat{x}$ = $A^Tb$, so $\hat{x}$ = $A^Tb$ $A^TA^{-1}$

```{r echo = TRUE}
# calculate xhat called leastSqr
(leastSqr <- solve(ata) %*% atb)

```

What is the squared error of this solution?

\[b = 
\left[  \begin{array}{cc}
    0  \\
    8  \\
    8  \\
    20
  \end{array} 
\right]
\]
  
and  

\[ A \hat{x} =
\left[  \begin{array}{cc}
    1       & 0  \\
    1       & 1  \\
    1       & 3  \\
    1       & 4
  \end{array} 
\right]
%
\left[  \begin{array}{cc}
    1  \\
    4
\end{array} 
\right]
= 
\left[  \begin{array}{cc}
    1  \\
    5  \\
    13  \\
    7
  \end{array} 
\right]
\]

Hence errSqr = sum((b - A $\hat{x})^2$)

```{r echo = TRUE}

# squred error
(errSqr <- sum((b - A %*% leastSqr)^2))

```

Instead of b = [0; 8; 8; 20], start with p = [1; 5; 13; 17] and find the exact solution (i.e. show that this system is solvable as all equations are consistent with each other. This should result in an error vector e = 0).

```{r echo = TRUE}
p <- c(1, 5, 13, 17)
atp <- t(A) %*% p
(leastSqr <- solve(ata) %*% atp)
# squred error
(errSqr <- sum((p - A %*% leastSqr)^2))

```

Show that the error e = b - p = [-1; 3; -5; 3].
```{r echo = TRUE}

e <- c(-1, 3, -5, 3)
isTRUE(all.equal(e, (b - p)))

```

Show that the error e is orthogonal to p and to each of the columns of A.
```{r echo = TRUE}

# the error e is orthogonal to p and to each of the columns of A when the transpose of e 
# 0 means the angle between them is 90 degrees
t(e) %*% p
t(e) %*% A[, 1]
t(e) %*% A[, 2]

```

## Problem Set 2

Consider the modified auto-mpg data (obtained from the UC Irvine Machine Learning dataset). This dataset contains 5 columns: displacement, horsepower, weight, acceleration, mpg. We are going to model mpg as a function of the other four variables. 

Write an R markdown script that takes in the auto-mpg data, extracts an A matrix from the first 4 columns and b vector from the fifth (mpg) column. Using the least squares approach, your code should compute the best fitting solution. That is, find the best fitting equation that expresses mpg in terms of the other 4 variables. Finally, calculate the fitting error between the predicted mpg of your model and the actual mpg. Your script should be able to load in the 5 column data set, extract A and b, and perform the rest of the calculations. Please have adequate comments in your code to make it easy to follow your work.

```{r  echo=TRUE}

# read in data table, and rename
auto.mpg <- read.table('auto-mpg.data', quote='\'', comment.char='')
names(auto.mpg) <- c('displacement', 'horsepower', 'weight', 'acceleration', 'mpg')
head(auto.mpg)

# create A and b
A <- data.matrix(auto.mpg[1:4])
b <- data.matrix(auto.mpg[5])
A <- cbind(1, A)

# compute A^TA and A^Tb
(ata <- t(A) %*% A)
(atb <- t(A) %*% b)

# calculate xhat
(leastSqr <- solve(ata) %*% atb)

# calculate the squared error
(errSqr <- sum((b - A %*% leastSqr)^2))

# the least square equation for 'displacement', 'horsepower', 'weight', 'acceleration' expressing 'mpg'
paste0('mpg = ', round(leastSqr[2, 1], digits=2), '*displacement + (', 
                     round(leastSqr[3, 1], digits=2), '*horsepower) + (', 
                     round(leastSqr[4, 1], digits=2), '*weight) + (',           
                     round(leastSqr[5, 1], digits=2), '*acceleration) + (', 
                     round(leastSqr[1, 1], digits=2), ')')

```



Reference:

https://github.com/wwells/CUNY_DATA_605/blob/master/Week05/WWells_Assign5.Rmd