add comprehension support to @threads

[StefanKarpinski]

For example:

@parallel [ f(i) | i=1:n ]

[ajdecon]

Is this particularly different from pmap(f, {i|i=1:10} )?

I.e., would it make sense to implement this kind of comprehension as just some syntactic sugar on pmap, or are you envisioning more to it than that?

[JeffBezanson]

The syntactic sugar part could be @parallel [ f(i) | i = 1:n ].

But, we want to handle multiple dimensions, and it would be nice for it to return a distributed array. It would also be nice for each processor to compute points evenly spaced over the whole range, since this usually gives better load balancing. Then you might want to be able to move those results into a simpler distribution.

[JeffBezanson]

Now implemented on the jb/multi branch!

[StefanKarpinski]

Sweet. Is this an indication that the new Darray model is more productive?

[JeffBezanson]

Yes, a bit, but the only real difference in the interface is that the core operation is constructing a piece of an array from its indexes. The old DArray constructor could have been changed to work that way, but many other changes are needed too (distributing in all dimensions, and doing most operations lazily).

I'm also not sure what to do about array references inside parallel comprehensions. What you want is for A[i,j] to be transformed to A_chunk[i-ioffs,j-joffs] where A_chunk is the part of A that each processor needs.

[JeffBezanson]

The basic thing is now on master, but it does not handle array references yet. Doing that rewrite may be too magical, but we should probably do it.

[amitmurthy]

@JeffBezanson , seems like you had already implemented this - which I have moved under @darray in #5512 .

What do you mean by "does not handle array references yet" ?

[StefanKarpinski]

This would be spelled @threads [ f(i) for i in 1:n ] these days. Not sure if we need to keep an issue open for it, but I don't want to deprive someone of the fun of closing a three-digit issue when they implement it 😁

[tkf]

FYI,

using Transducers
tcollect(2x for x in 1:10)

already does it in parallel. Filtering and flattening are also supported.