Deprecated batches replacement?

[ari-e]

I need to batch several tasks to run together on a GPU and return their results. I found the now-removed celery.contrib.batches task class, but I am unable to get it to return the results of the tasks (which I assume is an artifact of it not being maintained in years). Is there some new replacement for this class or a good way to accomplish my goal by some other approach?

[ask]

In the current batches you will have to use backend.mark_as_done manually, as it doesn't store results. You should be able to simply copy+paste the 3.1 batches implementation and use that in 4.0.

I wanted to replace this with a built-in implementation, but haven't found the time to do so yet.

[ari-e]

Thanks for the reply. I did try using backend.mark_as_done manually, but the task just remains pending indefinitely. I think it's an issue with the SimpleRequest object that the Request is converted to. I tried using the mark_as_done on a non-Batched task and it worked correctly. Any thoughts on how to get it to work?

[jclark360]

For anyone else that reads this post, the above isn't entirely true. Batches in 3.1 form does not work with Celery 4.0.2 as far as we've seen. I had to make changes to get it functional.

https://github.com/celery/celery/blob/3.1/celery/contrib/batches.py#L199

request = Req(message, body=message.payload, on_ack=ack, app=app,

https://github.com/celery/celery/blob/3.1/celery/contrib/batches.py#L166

return cls(request.id, request.name, request.body[0],
                   request.body[1], request.delivery_info, request.hostname)

[nijel]

JFYI: There is also separate module maintaining the removed code: https://github.com/percipient/celery-batches

[mrsndmn]

https://github.com/closeio/tasktiger/ is also supports batch tasks

[robin-vjc]

The fix above by @jclark360 wasn't sufficient to get it to work with Celery 5+ on Python 3.8.

I created a gist that works for me: https://gist.github.com/robin-vjc/1a4676ccb055162082c5a061ab556f58

[clokep]

The fix above by @jclark360 wasn't sufficient to get it to work with Celery 5+ on Python 3.8.

I created a gist that works for me: gist.github.com/robin-vjc/1a4676ccb055162082c5a061ab556f58

I'm planning to release a new version of celery-batches with support for Celery 5 this week. Just finished up the PR for it over at clokep/celery-batches#21. (This is the same repo as percipient/celery-batches, just moved to my personal namespace.)