Improve documentation for process of converting existing tasks to jobtastic

[shadowing]

first, thank you for you brilliant work!

I just found your project, it really should've been a lifesaver, BUT now that I've implemented many many "ordinary" complex celery task classes, is there some convenient way for me to migrate to jobtastic easily?

Also, does it integrate with celery 3.0 well?

[winhamwr]

Hi Shadowing,

is there some convenient way for me to migrate to jobtastic easily?

I can't speak for the ease, but I can tell you that we created Jobtastic to help standardize a couple-dozen existing Celery tasks on a large project. In fact, I'm about to push a ticket to production that was basically just converting a task to use Jobtastic for the caching and thundering herd avoidance.

My general process:

• Switch from extending the Task class to extending JobtasticTask
• Rename your run method to calculate_result
• Switch any task _args to *_kwargs
• Decide which keyword arguments should be used for caching and set the significant_kwargs
• Take a guess on the herd_avoidance_timeout
• If it should be cached, set the cache_duration
• Switch to using delay_or_run or delay_or_fail

This part is usually very quick and gives really good bang for your buck. I make sure everything is still working correctly at this point.

Then, if I want progress display, I sprinkle in update_progress calls and then update my client-side code to actually handle the PROGRESS feedback. The client side part is the most time-consuming. There's still work to be done making jquery-celery as well-documented and easy to use as Jobtastic itself (issues and pull requests very welcome).

Maybe I should convert the documentation to more than just a README and expand this as a guide? If you do go through and convert some tasks, I'd really love any feedback on gotchas or tips beyond this. I know that as the author, I might be glossing over some important-to-know detail about usage.

As far as Celery 3.0, the answer is "I think so." Unfortunately, I haven't used Celery 3 in production yet, but the good news is that if it doesn't work, it's not going to be subtle. It will fail hard and you'll be able to give me a stack trace to fix :)

Thanks for the feedback! I hope you're able to use Jobtastic and it saves you some time.
-Wes

[shadowing]

Hi, Wes,
Thank you for your promptly response! The migrating process does seem easy enough, though when I tried, there are some problems:

1. when I started the celery app, it raises an exception: Jobtastic requires either Django or Flask + Memcached result backend. I checked jobtastic/task.py only to find out that it requires either django or werkzurg installed and has a configuration item for CELERY_CACHE_BACKEND. My original project does not have such settings ( I am using AMQP result backend and don't use cache)
2. Since I have a rather complex celeryconfig.py, thinking that it may involves unnecessary confusion for integrating with Jobtastic, I want to use a clean example project to test the functionality. But there is none in your source repo and the example in the doc requires a bit extra work to run. (Sorry I am deadly tight in my current work schedule and didn't try). Also there are no unittests.
3. The most important thing may be that Jobtastic is probably in tight bundle with Django, it even lists Django as a requirement in the install package. Unfortunately I am using Flask. You see, it'll be an important factor to consider using it on production environment.

I think your project, along with jquery-celery plugin, is very promising in terms of both problem scenario and design philosophy. Looking forward to your further work.