future.apply: Separate package for future_lapply(), future_apply(), ...(?) (req for feedback)

[HenrikBengtsson]

UPDATE 2017-12-06: See https://github.com/HenrikBengtsson/future.apply.

As some of you've already noticed, I've been holding back on announcing the future_lapply() function to the world. One of the reasons is that I'm not sure it should be part of the future package per se. It probably better suited in a separate package that provides such a higher-level API on top of the Future API (future() + value() and %<-%). This is also the reason for why there is yet no future_sapply(), future_apply() and so on.

Putting future_lapply() and friends in a separate package future.apply would be analogue to how foreach provides foreach(), plyr provides llply(), and so on;

Package	Functions	Backends
future.apply	Future-versions of common goto *apply() functions available in core R (the 'base' package and some 'stats' package): future_apply(), future_eapply(), future_lapply(), future_mapply(), future_rapply(), future_sapply(), future_tapply(), future_vapply(), future_aggregate(), ...	All future backends
foreach	foreach(), times()	All future backends via doFuture
plyr	**ply(..., .parallel = TRUE) functions: aaply(), ddply(), dlply(), llply(), ...	All future backends via doFuture (because it uses foreach internally)
BiocParallel	Bioconductor's parallel mappers: bpaggregate(), bpiterate(), bplapply(), and bpvec()	All future backends via doFuture (because it supports foreach) or via BiocParallel.FutureParam (direct BiocParallelParam support; prototype)

None of the above high-level APIs implement their own parallel backends, but they rather enhance existing ones (such as doParallel and doMC).

By having future_lapply() and friends in a separate package, it keeps the identity and the size of the future package much lighter. The core idea of the future package is not to replace foreach and other higher-level "parallel mapper" functions, but to provide a minimalistic parallel framework for such to built upon such that they can utilize all backends that implements the Future API.

Comments and feedback much appreciated.

[xiaodaigh]

Good idea

[ThoDuyNguyen]

That would be great

[HenrikBengtsson]

Just a very early heads up: I'm pretty sure this will happen at some point, i.e. future_lapply() will be moved to a future.apply package that works on-top of the future package. Such a transition will be done by making future.apply::future_lapply() available first giving enough time for downstream packages to migrate. Then future::future_lapply() can be deprecated and eventually be made defunct.

[wlandau-lilly]

Thank you for the heads up. Very important for ropensci/drake#42.

[HenrikBengtsson]

I've created a future.apply package which currently provides future_lapply() and a new future_sapply(). It'll be on GitHub-only for a while. Let's continue any future discussions/feature requests related to future_lapply() related topics there, i.e. https://github.com/HenrikBengtsson/future.apply.

[ThoDuyNguyen]

Thank you for the update.