rand vs. sample

[StefanKarpinski]

Base Julia provides the rand and rand! functions, which do random sampling over various domains. The StatsBase package provides sample and sample! with partially overlapping functionality. There have been various threads (e.g. this one) where people are discussion additional functionality for one or the other. I'm opening this issue to discuss whether these interfaces should be separate or if they should be merged and if they are to remain separate, to clarify the distinction between rand and sample.

[JeffBezanson]

Looking at the docs, I don't think I see a case where the functions couldn't be combined.

[StefanKarpinski]

Among others, there's this:

julia> using StatsBase

julia> rand(1:3)
1

julia> rand([1:3])
ERROR: no method rand(Array{Int64,1})

julia> sample(1:3)
3

julia> sample([1:3])
3

sample(a) where a is an array could just be a method of rand and since we already support rand(3:100) and a range is just an efficiently stored vector, there is precedent for this.

[StefanKarpinski]

Ah, I misread your comment as "could" rather than "couldn't". So you're in agreement :-)

[StefanKarpinski]

I wonder if the idea of a weighted collection shouldn't be generalized. There's a lot of overlap with multisets and dicts where the values are weights. The features that don't appear in Base's rand function are weights, replacement and ordering. It would be nice if we could arrange things so that there doesn't need to be a separate function just to add those features. We can either backports support for those things into Base or figure out a way to layer support onto the base rand functions in a package.

[andreasnoack]

cc: @lindahua

[lindahua]

I am all for migrating the sample function to base, and merging it with rand. The use of a separate function name in StatsBase is just a stop gap.

cc: @johnmyleswhite

[johnmyleswhite]

I wholeheartedly agree. I would love to see sample in Base.

[porterjamesj]

+1000000 for sample in Base.

[StefanKarpinski]

Well, it's more the other direction we're proposing: using rand in stats to sample things.

[johnmyleswhite]

I thought the idea was to merge the names rand and sample.

[StefanKarpinski]

Yes, that's the idea. It remains to be seen which name will be left at the end.

[johnmyleswhite]

Presumably you're going to keep randn, right? That seems sufficient reason to keep rand over sample?

[jiahao]

I was thinking about that. Samplen is pretty odd.

[lindahua]

Agree to use rand in the place of sample.

[StefanKarpinski]

Ok, so the naming issue is decided. It remains to determine what needs to be done to merge them.

[lindahua]

If you may wait a little bit, I can make a PR for this after the deadline in this weekend. I am also fine if someone else can take a lead to do that.

[StefanKarpinski]

No hurry on this. It's not going into 0.3.

[lindahua]

I have now get to the point to tackle this issue. I am proposing the API below:

The main idea is to reuse rand for sampling from population:

# randomly draw one sample from a
rand(a::AbstractArray) 

# randomly draw multiple samples from a, with or without replacement
# this subsumes the rand(a::Range, n) methods
rand(a::AbstractArray, n::Int; replace::Bool=true)
rand(a::AbstractArray, dim::Dims; replace::Bool=true)

The rand function will be extended in StatsBase to accept a weight-vector for weighted sampling.

With this implemented, the sample function in StatsBase will then be deprecated in favor of rand.

[JeffBezanson]

Sounds good. There's a slight potential for confusion with rand((m,n)), which constructs a random array instead of sampling from the tuple. But this doesn't seem like a huge problem.

[ViralBShah]

@lindahua Is this worth revisiting now in the 0.4 cycle?

[lindahua]

I may look into this again next week.

[tkelman]

Did not make it into 0.4, removing from milestone.

[StefanKarpinski]

Someone needs to own this to make it happen or it's going to slip again.

[StefanKarpinski]

It's also a bit unclear what still needs to be done at this point.

[simonbyrne]

The main difference now is that sample provides keyword arguments for replacement and ordering, e.g.

sample(1:10,8,replace=false)

The only reason we can't combine them is that we can't add more keyword arguments to existing method signatures without getting a method overwritten warning.

[simonbyrne]

and the problem @JeffBezanson mentioned above:

julia> rand([1,2])
1

julia> rand((1,2))
1×2 Array{Float64,2}:
 0.966145  0.590825

(sample throws a MethodError on the tuple one)

[simonbyrne]

After some discussion, the decision was made to leave it as is. rand and sample are indeed subtly different:

• rand samples independently (either uniform [0,1], uniform from a discrete set, or from a specific distribution if used with Distributions),
• sample allows for non-independent methods (e.g. without replacement), which are only applicable to finite sets.