05-Penalized-Linear-Regression.Rmd

# Penalized Linear Regressions: A Simulation Experiment

## Data Generating Process: Approximately Sparse

1. Import libraries

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**R code**
```{r libraries-r,echo= TRUE,message=FALSE,warning=FALSE}
# No libraries requiered
```
:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**Python code**
```{python libraries-python, message=FALSE,warning=FALSE}
import random
import numpy as np
import math
import matplotlib.pyplot as plt
```
:::
::::::

2. To set seed

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**R code**
```{r seed-r,echo= TRUE,message=FALSE,warning=FALSE}
set.seed(1)
n = 100
p = 400
```
:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**Python code**
```{python seed-python, message=FALSE,warning=FALSE}
random.seed(1)
n = 100
p = 400
```
:::
::::::

3. To create variables

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r variables-r,echo= TRUE,message=FALSE,warning=FALSE}
Z= runif(n)-1/2
W = matrix(runif(n*p)-1/2, n, p)
```
:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python variables-python, message=FALSE,warning=FALSE}
Z = np.random.uniform( low = 0 , high = 1 , size = n) - 1/2 
W = ( np.random.uniform( low = 0 , high = 1 , size = n * p ) - 1/2 ).\
        reshape( n , p )
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r variables-two-r,echo= TRUE,message=FALSE,warning=FALSE}
beta = 1/seq(1:p)^2   # approximately sparse beta
gX = exp(4*Z)+ W%*%beta  # leading term nonlinear
X = cbind(Z, Z^2, Z^3, W )  # polynomials in Zs will be approximating exp(4*Z)
```
:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python variables-two-python, message=FALSE,warning=FALSE}
beta = ((1/ np.arange(1, p + 1 )) ** 2)
gX = np.exp( 4 * Z ) + (W @ beta)
X = np.concatenate( ( Z.reshape(Z.size, 1), Z.reshape(Z.size, 1) \
                     ** 2, Z.reshape(Z.size, 1) ** 3, W ) , axis = 1 )
```
:::
::::::

4. Generate $Y$

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r y-r, message=FALSE,warning=FALSE}
Y = gX + rnorm(n)   #generate Y
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python y-python, message=FALSE,warning=FALSE}
mean = 0
sd = 1
Y = gX + np.random.normal( mean , sd, n )
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r plot-r, message=FALSE,warning=FALSE}
#
#
#
#
# 
plot(gX,Y, xlab="g(X)", ylab="Y", title("Y vs g(X)")) 
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python plot-python, message=FALSE,warning=FALSE}
fig = plt.figure()
fig.suptitle('Y vs g(X)')
ax = fig.add_subplot(111)
plt.scatter( Y, gX)
plt.xlabel('g(X)')
plt.ylabel('Y')
plt.show()
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r parameter-r, message=FALSE,warning=FALSE}
print( c("theoretical R2:", var(gX)/var(Y)))
dim(X)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python parameter-python, message=FALSE,warning=FALSE}
print( f"theoretical R2:, {np.var(gX) / np.var( Y )}" ) 
X.shape
```
:::
::::::

We use package Glmnet to carry out predictions using cross-validated lasso, ridge, and elastic net.

We should know that **cv.glmnet** function in r **standarize** ***X*** data by default. So, we have to standarize our data before the execution of sklearn package. The **normalize** parameter will help for this. However, the function cv.glamnet  is also standarizing the **Y** [variable](https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html) and then unstadarize the coefficients from the regression. To do this with sklearn, we will standarize the **Y** variable before fitting with **StandardScaler** function. Finally, the r-function uses 10 folds by default so we will adjust our model to use $cv=10$ ten folds.\
\
The parameter $l1_{ratio}$ corresponds to **alpha** in the glmnet R package while **alpha** corresponds to the **lambda** parameter in **glmnet**. Specifically, $l1_{ratio} = 1$ is the lasso penalty. Currently, $l1_{ratio} <= 0.01$ is not reliable, unless you supply your own sequence of **alpha**.

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r lib-r, message=FALSE,warning=FALSE}
library(glmnet)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python lib-python, message=FALSE,warning=FALSE}
from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import RidgeCV, ElasticNetCV
import statsmodels.api as sm
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r estimation-r, message=FALSE,warning=FALSE}
fit.lasso.cv   <- cv.glmnet(X, Y, family="gaussian", alpha=1)  # family gaussian means that we'll be using square loss
fit.ridge   <- cv.glmnet(X, Y, family="gaussian", alpha=0)     # family gaussian means that we'll be using square loss
fit.elnet   <- cv.glmnet(X, Y, family="gaussian", alpha=.5)    # family gaussian means that we'll be using square loss

yhat.lasso.cv    <- predict(fit.lasso.cv, newx = X)            # predictions
yhat.ridge   <- predict(fit.ridge, newx = X)
yhat.elnet   <- predict(fit.elnet, newx = X)

MSE.lasso.cv <- summary(lm((gX-yhat.lasso.cv)^2~1))$coef[1:2]  # report MSE and standard error for MSE for approximating g(X)
MSE.ridge <- summary(lm((gX-yhat.ridge)^2~1))$coef[1:2]        # report MSE and standard error for MSE for approximating g(X)
MSE.elnet <- summary(lm((gX-yhat.elnet)^2~1))$coef[1:2]        # report MSE and standard error for MSE for approximating g(X)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python estimation-python, message=FALSE,warning=FALSE}
# Reshaping Y variable
Y_vec = Y.reshape( Y.size, 1)

# Scalar distribution
scaler = StandardScaler()
scaler.fit( Y_vec )
std_Y = scaler.transform( Y_vec )

# Regressions
fit_lasso_cv = LassoCV(cv = 10 , random_state = 0 , normalize = True ).fit( X, std_Y )
fit_ridge = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.0001 ).fit( X, std_Y )
fit_elnet = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.5, max_iter = 100000 ).fit( X, std_Y )

# Predictions
yhat_lasso_cv = scaler.inverse_transform( fit_lasso_cv.predict( X ) )
yhat_ridge = scaler.inverse_transform( fit_ridge.predict( X ) )
yhat_elnet = scaler.inverse_transform( fit_elnet.predict( X ) )

MSE_lasso_cv = sm.OLS( ((gX - yhat_lasso_cv)**2 ) , np.ones( yhat_lasso_cv.shape )  ).fit().summary2().tables[1].round(3)
MSE_ridge = sm.OLS( ((gX - yhat_ridge)**2 ) , np.ones( yhat_ridge.size )  ).fit().summary2().tables[1].round(3)
MSE_elnet = sm.OLS( ((gX - yhat_elnet)**2 ) , np.ones( yhat_elnet.size )  ).fit().summary2().tables[1].round(3)
# our coefficient of MSE_elnet are far from r output
```
:::
::::::

Here we compute the lasso and ols post lasso using plug-in choices for penalty levels, using package hdm.

Rlasso functionality: it is searching the right set of regressors. This function was made for the case of ***p*** regressors and ***n*** observations where ***p >>>> n***. It assumes that the error is i.i.d. The errors may be non-Gaussian or heteroscedastic.

The post lasso function makes OLS with the selected ***T*** regressors.
To select those parameters, they use $\lambda$ as variable to penalize **Funny thing: the function rlasso was named like that because it is the "rigorous" Lasso.**
We find a Python code that tries to replicate the main function of hdm r-package. I was made by [Max Huppertz](https://maxhuppertz.github.io/code/). 

His library is this [repository](https://github.com/maxhuppertz/hdmpy). Download its repository and copy this folder to your site-packages folder. In my case it is located here ***C:\Python\Python38\Lib\site-packages*** .

We need to install this package ***pip install multiprocess***.

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**R code**
```{r lib-two-r, message=FALSE,warning=FALSE}
library(hdm) 
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**Python code**
```{python lib-two-python, message=FALSE,warning=FALSE}
import hdmpy
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r second-prediction-r, message=FALSE,warning=FALSE}
fit.rlasso  <- rlasso(Y~X,  post=FALSE)      # lasso with plug-in penalty level
fit.rlasso.post <- rlasso(Y~X,  post=TRUE)    # post-lasso with plug-in penalty level

yhat.rlasso   <- predict(fit.rlasso)            #predict g(X) for values of X
yhat.rlasso.post   <- predict(fit.rlasso.post)  #predict g(X) for values of X

MSE.lasso <- summary(lm((gX-yhat.rlasso)^2~1))$coef[1:2]       # report MSE and standard error for MSE for approximating g(X)
MSE.lasso.post <- summary(lm((gX-yhat.rlasso.post)^2~1))$coef[1:2]  # report MSE and standard error for MSE for approximating g(X)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python second-prediction-python, message=FALSE,warning=FALSE}
fit_rlasso = hdmpy.rlasso(X, Y, post = False)
fit_rlasso_post = hdmpy.rlasso(X, Y, post = True)

yhat_rlasso = Y - fit_rlasso.est['residuals'].reshape( Y.size,  )
yhat_rlasso_post = Y - fit_rlasso_post.est['residuals'].reshape( Y.size ,  )

MSE_lasso = sm.OLS( ((gX - yhat_rlasso)**2 ) , np.ones( yhat_rlasso.size )  ).fit().summary2().tables[1].round(3)
MSE_lasso_post = sm.OLS( ((gX - yhat_rlasso_post)**2 ) , np.ones( yhat_rlasso_post.size )  ).fit().summary2().tables[1].round(3)
```
:::
::::::

Next we code up lava, which alternates the fitting of lasso and ridge

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r function-lava-r, message=FALSE,warning=FALSE}

lava.predict<- function(X,Y, iter=5){
    
g1 = predict(rlasso(X, Y, post=F))  #lasso step fits "sparse part"
m1 =  predict(glmnet(X, as.vector(Y-g1), family="gaussian", alpha=0, lambda =20),newx=X ) #ridge step fits the "dense" part

    
i=1
while(i<= iter) {
g1 = predict(rlasso(X, Y, post=F))   #lasso step fits "sparse part"
m1 = predict(glmnet(X, as.vector(Y-g1), family="gaussian",  alpha=0, lambda =20),newx=X );  #ridge step fits the "dense" part
i = i+1 }

return(g1+m1);
    }

```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python function-lava-python, message=FALSE,warning=FALSE}
def lava_predict( x, y, iteration = 5 ):
    
    g1_rlasso = hdmpy.rlasso( x, y , post = False )
    g1 = y - g1_rlasso.est['residuals'].reshape( g1_rlasso.est['residuals'].size, )
    
    new_dep_var = y-g1
    new_dep_var_vec = new_dep_var.reshape( new_dep_var.size, 1 )
    
    # Scalar distribution
    scaler = StandardScaler()
    scaler.fit( new_dep_var_vec )
    std_new_dep_var_vec = scaler.transform( new_dep_var_vec )
    
    fit_ridge_m1 = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.0001, alphas = np.array([20]) ).fit( x, std_new_dep_var_vec )
    m1 = scaler.inverse_transform( fit_ridge_m1.predict( x ) )
    
    i = 1
    while i <= iteration:
        
        g1_rlasso = hdmpy.rlasso( x, y , post = False )
        g1 = y - g1_rlasso.est['residuals'].reshape( g1_rlasso.est['residuals'].size, )

        new_dep_var = y-g1
        new_dep_var_vec = new_dep_var.reshape( new_dep_var.size, 1 )

        # Scalar distribution
        scaler = StandardScaler()
        scaler.fit( new_dep_var_vec )
        std_new_dep_var_vec = scaler.transform( new_dep_var_vec )

        fit_ridge_m1 = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.0001, alphas = np.array([20]) ).fit( x, std_new_dep_var_vec )
        m1 = scaler.inverse_transform( fit_ridge_m1.predict( x ) )
        
        i = i + 1
        
    return ( g1 + m1 )
```
:::
::::::


:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r estimation-lava-r, message=FALSE,warning=FALSE}
yhat.lava = lava.predict(X,Y)
MSE.lava <- summary(lm((gX-yhat.lava)^2~1))$coef[1:2]# report MSE and standard error for MSE for approximating g(X)
MSE.lava
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python estimation-lava-python, message=FALSE,warning=FALSE}
yhat_lava = lava_predict( X, Y )
MSE_lava = sm.OLS( ((gX - yhat_lava)**2 ) , np.ones( yhat_lava.size )  ).fit().summary2().tables[1].round(3)
MSE_lava
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r result-r, message=FALSE,warning=FALSE}
library(xtable)
table<- matrix(0, 6, 2)
table[1,1:2]   <- MSE.lasso.cv
table[2,1:2]   <- MSE.ridge
table[3,1:2]   <- MSE.elnet
table[4,1:2]   <- MSE.lasso
table[5,1:2]   <- MSE.lasso.post
table[6,1:2]   <- MSE.lava

colnames(table)<- c("MSA", "S.E. for MSA")
rownames(table)<- c("Cross-Validated Lasso", "Cross-Validated ridge","Cross-Validated elnet",
                    "Lasso","Post-Lasso","Lava")
tab <- xtable(table, digits =3)
print(tab,type="latex")
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python result-python, message=FALSE,warning=FALSE}
import pandas as pd
table2 = np.zeros( (6, 2) )

table2[0, 0:] = MSE_lasso_cv.iloc[0, 0:2].to_list()
table2[1, 0:] = MSE_ridge.iloc[0, 0:2].to_list()
table2[2, 0:] = MSE_elnet.iloc[0, 0:2].to_list()
table2[3, 0:] = MSE_lasso.iloc[0, 0:2].to_list()
table2[4, 0:] = MSE_lasso_post.iloc[0, 0:2].to_list()
table2[5, 0:] = MSE_lava.iloc[0, 0:2].to_list()



table2_pandas = pd.DataFrame( table2, columns = [ "MSA","S.E. for MSA" ])
table2_pandas.index = [ "Cross-Validated Lasso",\
                       "Cross-Validated Ridge", "Cross-Validated elnet",\
                       "Lasso", "Post-Lasso", "Lava" ]
table2_pandas = table2_pandas.round(3)
table2_html = table2_pandas.to_html()
table2_pandas
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r plot-two-r, message=FALSE,warning=FALSE}
plot(gX, gX, pch=19, cex=1, ylab="predicted value", xlab="true g(X)")
points(gX, yhat.rlasso, col=2, pch=18, cex = 1.5 )
points(gX,  yhat.rlasso.post, col=3, pch=17,  cex = 1.2  )
points( gX, yhat.lasso.cv,col=4, pch=19,  cex = 1.2 )
legend("bottomright", 
  legend = c("rLasso", "Post-rLasso", "CV Lasso"),col = c(2,3,4), 
  pch = c(18,17, 19),bty = "n", 
  pt.cex = 1.3, cex = 1.2, 
  text.col = "black", horiz = F ,inset = c(0.1, 0.1))
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python plot-two-python, message=FALSE,warning=FALSE}
import matplotlib.pyplot as plt
fig = plt.figure()
ax1 = fig.add_subplot(111)
ax1.scatter( gX, gX , marker = '.', c = 'black' )
ax1.scatter( gX, yhat_rlasso , marker = 'D' , c = 'red' , label = 'rLasso' )
ax1.scatter( gX, yhat_rlasso_post , marker = '^' , c = 'green' , label = 'Post-rLasso')
ax1.scatter( gX, yhat_lasso_cv , marker = 'o' , c = 'blue' , label = 'CV Lasso')
plt.legend(loc='lower right')
plt.show()
```
:::
::::::

## Data Generating Process: Approximately Sparse + Small Dense Part

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**R code**
```{r sparse-r, message=FALSE,warning=FALSE}
set.seed(1)

n = 100
p = 400

Z= runif(n)-1/2
W = matrix(runif(n*p)-1/2, n, p)

beta = rnorm(p)*.2    # dense beta
gX = exp(4*Z)+ W%*%beta # leading term nonlinear
X = cbind(Z, Z^2, Z^3, W ) # polynomials in Zs will be approximating exp(4*Z)

Y = gX + rnorm(n)   #generate Y
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

**Python code**
```{python sparce-python, message=FALSE,warning=FALSE}
n = 100
p = 400

Z = np.random.uniform( low = 0 , high = 1 , size = n) - 1/2 

W = ( np.random.uniform( low = 0 , high = 1 , size = n * p ) - 1/2 ).\
        reshape( n , p )
mean = 0
sd = 1

beta = ((np.random.normal( mean , sd, p )) * 0.2)
gX = np.exp( 4 * Z ) + (W @ beta)
X = np.concatenate( ( Z.reshape(Z.size, 1), Z.reshape(Z.size, 1) \
                     ** 2, Z.reshape(Z.size, 1) ** 3, W ) , axis = 1 )
random.seed(2)
Y = gX + np.random.normal( mean , sd, n )
```
:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
#
#
#
#
#
#
#
plot(gX,Y, xlab="g(X)", ylab="Y")    #plot V vs g(X)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
# We use package Glmnet to carry out predictions using cross-validated lasso, ridge, and elastic net
fig = plt.figure()
fig.suptitle('Y vs g(X)')
ax = fig.add_subplot(111)
plt.scatter( Y, gX)
plt.xlabel('g(X)')
plt.ylabel('Y')
plt.show()
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
print( c("theoretical R2:", var(gX)/var(Y)))
var(gX)/var(Y)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
print( f"theoretical R2:, {np.var(gX) / np.var( Y )}" ) 
np.var(gX) / np.var( Y ) #theoretical R-square in the simulation example
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
library(glmnet)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
# No libraries
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
fit.lasso.cv   <- cv.glmnet(X, Y, family="gaussian", alpha=1)  # family gaussian means that we'll be using square loss
fit.ridge   <- cv.glmnet(X, Y, family="gaussian", alpha=0)     # family gaussian means that we'll be using square loss
fit.elnet   <- cv.glmnet(X, Y, family="gaussian", alpha=.5)    # family gaussian means that we'll be using square loss
yhat.lasso.cv    <- predict(fit.lasso.cv, newx = X)            # predictions
yhat.ridge   <- predict(fit.ridge, newx = X)
yhat.elnet   <- predict(fit.elnet, newx = X)

```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python, message = FALSE}
# Reshaping Y variable
Y_vec = Y.reshape( Y.size, 1)
# Scalar distribution
scaler = StandardScaler()
scaler.fit( Y_vec )
std_Y = scaler.transform( Y_vec )
# Regressions
fit_lasso_cv = LassoCV(cv = 10 , random_state = 0 , normalize = True ).fit( X, std_Y )
fit_ridge = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.0001 ).fit( X, std_Y )
fit_elnet = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.5, max_iter = 100000 ).fit( X, std_Y )
# Predictions
yhat_lasso_cv = scaler.inverse_transform( fit_lasso_cv.predict( X ) )
yhat_ridge = scaler.inverse_transform( fit_ridge.predict( X ) )
yhat_elnet = scaler.inverse_transform( fit_elnet.predict( X ) )
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
MSE.lasso.cv <- summary(lm((gX-yhat.lasso.cv)^2~1))$coef[1:2]  # report MSE and standard error for MSE for approximating g(X)
MSE.ridge <- summary(lm((gX-yhat.ridge)^2~1))$coef[1:2]        # report MSE and standard error for MSE for approximating g(X)
MSE.elnet <- summary(lm((gX-yhat.elnet)^2~1))$coef[1:2]        # report MSE and standard error for MSE for approximating g(X)

```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
MSE_lasso_cv = sm.OLS( ((gX - yhat_lasso_cv)**2 ) , np.ones( yhat_lasso_cv.shape )  ).fit().summary2().tables[1].round(3)
MSE_ridge = sm.OLS( ((gX - yhat_ridge)**2 ) , np.ones( yhat_ridge.size )  ).fit().summary2().tables[1].round(3)
MSE_elnet = sm.OLS( ((gX - yhat_elnet)**2 ) , np.ones( yhat_elnet.size )  ).fit().summary2().tables[1].round(3)
# our coefficient of MSE_elnet are far from r output
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
fit.rlasso  <- rlasso(Y~X,  post=FALSE)      # lasso with plug-in penalty level
fit.rlasso.post <- rlasso(Y~X,  post=TRUE)    # post-lasso with plug-in penalty level

yhat.rlasso   <- predict(fit.rlasso)            #predict g(X) for values of X
yhat.rlasso.post   <- predict(fit.rlasso.post)  #predict g(X) for values of X

MSE.lasso <- summary(lm((gX-yhat.rlasso)^2~1))$coef[1:2]       # report MSE and standard error for MSE for approximating g(X)
MSE.lasso.post <- summary(lm((gX-yhat.rlasso.post)^2~1))$coef[1:2]  # report MSE and standard error for MSE for approximating g(X)

```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
fit_rlasso = hdmpy.rlasso(X, Y, post = False)
fit_rlasso_post = hdmpy.rlasso(X, Y, post = True)

yhat_rlasso = Y - fit_rlasso.est['residuals'].reshape( Y.size,  )
yhat_rlasso_post = Y - fit_rlasso_post.est['residuals'].reshape( Y.size ,  )

MSE_lasso = sm.OLS( ((gX - yhat_rlasso)**2 ) , np.ones( yhat_rlasso.size )  ).fit().summary2().tables[1].round(3)
MSE_lasso_post = sm.OLS( ((gX - yhat_rlasso_post)**2 ) , np.ones( yhat_rlasso_post.size )  ).fit().summary2().tables[1].round(3)
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
lava.predict<- function(X,Y, iter=5){
    
g1 = predict(rlasso(X, Y, post=F))  #lasso step fits "sparse part"
m1 =  predict(glmnet(X, as.vector(Y-g1), family="gaussian", alpha=0, lambda =20),newx=X ) #ridge step fits the "dense" part

    
i=1
while(i<= iter) {
g1 = predict(rlasso(X, Y, post=F))   #lasso step fits "sparse part"
m1 = predict(glmnet(X, as.vector(Y-g1), family="gaussian",  alpha=0, lambda =20),newx=X );  #ridge step fits the "dense" part
i = i+1 }

return(g1+m1);
    }
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
def lava_predict( x, y, iteration = 5 ):
    
    g1_rlasso = hdmpy.rlasso( x, y , post = False )
    g1 = y - g1_rlasso.est['residuals'].reshape( g1_rlasso.est['residuals'].size, )
    
    new_dep_var = y-g1
    new_dep_var_vec = new_dep_var.reshape( new_dep_var.size, 1 )
    
    # Scalar distribution
    scaler = StandardScaler()
    scaler.fit( new_dep_var_vec )
    std_new_dep_var_vec = scaler.transform( new_dep_var_vec )
    
    fit_ridge_m1 = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.0001, alphas = np.array([20]) ).fit( x, std_new_dep_var_vec )
    m1 = scaler.inverse_transform( fit_ridge_m1.predict( x ) )
    
    i = 1
    while i <= iteration:
        
        g1_rlasso = hdmpy.rlasso( x, y , post = False )
        g1 = y - g1_rlasso.est['residuals'].reshape( g1_rlasso.est['residuals'].size, )

        new_dep_var = y-g1
        new_dep_var_vec = new_dep_var.reshape( new_dep_var.size, 1 )

        # Scalar distribution
        scaler = StandardScaler()
        scaler.fit( new_dep_var_vec )
        std_new_dep_var_vec = scaler.transform( new_dep_var_vec )

        fit_ridge_m1 = ElasticNetCV( cv = 10 , normalize = True , random_state = 0 , l1_ratio = 0.0001, alphas = np.array([20]) ).fit( x, std_new_dep_var_vec )
        m1 = scaler.inverse_transform( fit_ridge_m1.predict( x ) )
        
        i = i + 1
        
    return ( g1 + m1 )
        
```

:::
::::::


:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
yhat.lava = lava.predict(X,Y)
MSE.lava <- summary(lm((gX-yhat.lava)^2~1))$coef[1:2]       # report MSE and standard error for MSE for approximating g(X)
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
yhat_lava = lava_predict( X, Y )
MSE_lava = sm.OLS( ((gX - yhat_lava)**2 ) , np.ones( yhat_lava.size )  ).fit().summary2().tables[1].round(3)
```

:::
::::::

:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
table<- matrix(0, 6, 2)
table[1,1:2]   <- MSE.lasso.cv
table[2,1:2]   <- MSE.ridge
table[3,1:2]   <- MSE.elnet
table[4,1:2]   <- MSE.lasso
table[5,1:2]   <- MSE.lasso.post
table[6,1:2]   <- MSE.lava

colnames(table)<- c("MSA", "S.E. for MSA")
rownames(table)<- c("Cross-Validated Lasso", "Cross-Validated ridge","Cross-Validated elnet","Lasso","Post-Lasso","Lava")
tab <- xtable(table, digits =3)
print(tab,type="latex") # set type="latex" for printing table in LaTeX
tab
```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
table2 = np.zeros( (6, 2) )

table2[0, 0:] = MSE_lasso_cv.iloc[0, 0:2].to_list()
table2[1, 0:] = MSE_ridge.iloc[0, 0:2].to_list()
table2[2, 0:] = MSE_elnet.iloc[0, 0:2].to_list()
table2[3, 0:] = MSE_lasso.iloc[0, 0:2].to_list()
table2[4, 0:] = MSE_lasso_post.iloc[0, 0:2].to_list()
table2[5, 0:] = MSE_lava.iloc[0, 0:2].to_list()



table2_pandas = pd.DataFrame( table2, columns = [ "MSA","S.E. for MSA" ])
table2_pandas.index = [ "Cross-Validated Lasso",\
                       "Cross-Validated Ridge", "Cross-Validated elnet",\
                       "Lasso", "Post-Lasso", "Lava" ]
table2_pandas = table2_pandas.round(3)
table2_html = table2_pandas.to_html()
table2_pandas
```

:::
::::::


:::::: {.columns}
::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{r}
plot(gX, gX, pch=19, cex=1, ylab="predicted value", xlab="true g(X)")

points(gX, yhat.rlasso,   col=2, pch=18, cex = 1.5 )
points(gX, yhat.elnet,  col=3, pch=17,  cex = 1.2  )
points(gX, yhat.lava,  col=4, pch=19,  cex = 1.2 )

legend("bottomright", 
  legend = c("rLasso", "Elnet", "Lava"), 
  col = c(2,3,4), 
  pch = c(18,17, 19), 
  bty = "n", 
  pt.cex = 1.3, 
  cex = 1.2, 
  text.col = "black", 
  horiz = F , 
  inset = c(0.1, 0.1))

```

:::
::: {.column width="1%" data-latex="{0.04\textwidth}"}
\ 
<!-- an empty Div (with a white space), serving as
a column separator -->
:::
:::::: {.column width="49.5%" data-latex="{0.48\textwidth}"}

```{python}
fig = plt.figure()

ax1 = fig.add_subplot(111)
ax1.scatter( gX, gX , marker = '.', c = 'black' )
ax1.scatter( gX, yhat_rlasso , marker = 'D' , c = 'red' , label = 'rLasso' )
ax1.scatter( gX, yhat_rlasso_post , marker = '^' , c = 'green' , label = 'Post-rLasso')
ax1.scatter( gX, yhat_lasso_cv , marker = 'o' , c = 'blue' , label = 'CV Lasso')
plt.legend(loc='lower right')

plt.show()
```

:::
::::::