Include a new SqlParallel queue type that enables multiple queue runners to process in parallel

[artfulrobot]

This is a feature adding PR not a bugfix.

It adds a new Sql queue type which will allow multiple runners to fetch the next available job from a queue and run in parallel. e.g. if you offload email sending to a queue this might be useful; you don't care which task is run first so you could have 2 queue runners running at once and get through the queue in half the time as task 2 does not need to wait for task 1 to complete.

There's a test for this at https://github.com/artfulrobot/queuetest

Like my other queue PR it ensures queue safety in that despite multiprocessing, no task can be run twice.

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[mlutfy]

Just a very quick review:

• Would this be for extensions? or could core use this queue eventually?
• Just curious: would it be possible to implement as an extension?
• Would be good to have feedback from Tim

Nitpick: Before merging, could you trim the copyright header to use the new (shorter) comment?

[artfulrobot]

@mlutfy on your first 2 points

• core or extensions could use it, but the use case is multiple concurrent processes attacking a queue in parallel; CiviCRM doesn't do much of this at the mo - there's the max cron jobs going at once setting, but that's it. It's probably a tool for people who have specific needs to increase processing performance.

• I don't think it's possible to implement as an extension; I don't think the queue factory/service knows to look elsewhere.

PR rebased and license text updated 👍

[eileenmcnaughton]

test this please

[artfulrobot]

A lot of work has gone into this, but it is not currently needed in core. It was a byproduct of another pr and discussions with @totten - at the time it seemed like a good tool to add.

I would be ok with closing this for the sake of reducing forwards maintenance and saving on reviewer/merger time.

That's a personal view (I don't use it myself) but I haven't closed it myself because it's not just my work that has gone in. But from my pov: close/merge/meh!

[eileenmcnaughton]

I'm somewhat inclined to think this is a good thing to have in core & as extra code only it's low risk but I have not tested / reviewed as yet

[artfulrobot]

Fine, I just didn't want to be unnecessarily increasing core/voluntary workload. A low priority one.

[eileenmcnaughton]

I'm going to merge this as

1. it has no risk IHMO
2. I think it came out of a sprint & was well discussed & agreed