Feature: Cor and Cov for Sparse Matrices

[jeffwong]

Hi, I am new to Julia and want to become a contributor. In my daily work I operate on sparse matrices every day, and need a sparse implementation for cor and cov. From what I can tell, there are only functions for computing cor and cov on dense matrices. I'd like to work on this functionality and contribute it to Julia. Were there any other plans that someone had in mind?

One of the challenges for computing cov is the fact that the input matrix needs to be centered and scaled. Centering a sparse matrix makes it dense again, which makes it sound like it defeats the purpose. However, we can write an implementation that operates on two data structures, the original sparse matrix X and a vector of means, m. We will be able to construct cov(X) by working on X-rep(m) and still be able to preserve some of the sparse operations. This is what happens in R's glmnet package.

[nalimilan]

I don't think anybody has plans about this, so your contribution would be welcome. But note the standard cov and cor methods should be included in Julia Base (in the sparse module): only the weighted methods are defined in this package. See JuliaLang/julia#7788, which seems to be more limited than what you propose since it only discussed optimizing mean.

BTW, just in case: you should avoid taking inspiration from the glmnet code since it's GPL2 and Julia Base is MIT. Using the same algorithm is fine, though.

[jeffwong]

Ah got it! Thanks again for your help @nalimilan

[nalimilan]

Closing since the issue should be handled in Base. Feel free to discuss the implementation on that issue.