[6.x] Optimize next available job in database queue #34990
Conversation
Current getNextAvailableJob queue selects all columns in the jobs database. This can cause database performance issues if the jobs table has many rows and the payload column is large. Added subquery to find the next available job prior to selecting all columns in the database. Fully backwards compatible.
GrahamCampbell
changed the title
|
What kind of performance issues were you seeing on your application? |
| @@ -212,13 +212,18 @@ public function pop($queue = null) | |||
| protected function getNextAvailableJob($queue) | |||
| { | |||
| $job = $this->database->table($this->table) | |||
| ->lock($this->getLockForPopping()) | |||
There was a problem hiding this comment.
You'll need to retain the lock like this to avoid deadlocks. See #31287
There was a problem hiding this comment.
Although an optimization related to the database queue, this lock change appears to be a separate ongoing discussion unrelated to the changes I am recommending.
|
@taylorotwell Jobs database was 1.9gb with ~32,000 rows. We have a lot of jobs scheduled in the future (some 6 months out). The existing select required the entire database to be in memory, causing Job database queries to take well over 60 seconds. Temporary solution was to increase the memory available to the database server, but an optimized query is a better long-term solution of course. |
|
Your removal of the lock that @paras-malhotra mentions introduces even bigger problems to the database queue and will prevent more than just a few workers from running against it at the same time. Feel free to re-submit without that removal. |
Current getNextAvailableJob queue selects all columns in the jobs database. This can cause database performance issues if the jobs table has many rows and the payload column is large. Added subquery to find the next available job prior to selecting all columns in the database. Fully backwards compatible.