(dev/core#1304) Queues - Define schema for runner, run_count, lease_time, et al

[totten]

Overview

Expand the schema for civicrm_queue (et al). By configuring properties on civicrm_queue (et al), you can tell a generic queue-agent how to handle items from this queue.

Each civicrm_queue (general metadata; matches $queueSpec) may specify runner, lease_time, batch_limit, retry_limit, retry_interval.

Each civicrm_queue_item (data-storage for Sql/SqlParallel queues) may track the number of execution attempts (run_count). This facilitates implementation of retry_limit.

This PR only defines the schema so that these options may be passed around. For fuller implementations (with test-coverage), see #22762.

ping @eileenmcnaughton @artfulrobot

Cross-Reference

• https://lab.civicrm.org/dev/core/-/issues/1304: General discussion about issues goals for queues (esp wrt developer experience and performance)
• (dev/core#1304) Queues - Allow background worker to drive tasks via APIv4 #22690 - Prior draft for adding APIv4 support for claiming+running queued tasks. Discussion raised additional requirements, spawning replacement PRs.
• (dev/core#1304) Queues - Allow background worker to drive tasks via APIv4 #22762 - Updated draft for adding APIv4 support. It builds on the present commits (updated schema) and makes them operational.

Before

The civicrm_queue and $queueSpec support the following option:

• bool $is_autorun: Enable or disable automatic task-execution for this queue. This option was defined in the schema in a recent revision, but it has not been used in stable-public code. It has only been used by drafts+alphas.

After

The civicrm_queue and $queueSpec support several options:

• ?string $runner: Specify how the queue should execute tasks. This replaces the the is_autorun flag. Some example values:
  • null (default/status quo): Do not automatically run tasks.
    • (Equivalent to the prior is_autorun=FALSE.)
  • 'task': The queue stores Task records (eg new CRM_Queue_Task('my_callback', ['my_data']). This is the same as Civi's existing queue APIs. In effect, the generic queue-agent will run call_user_func_array('my_callback', ['my_data']).
    • (Equivalent to the prior is_autorun=TRUE.)
  • Other strings: If you use any other string, then you must implement hook_civicrm_queueRun_{$runner} to define how to evaluate queued tasks. (Pros+cons are later.)
• int $lease_time (eg 3600 sec): When a single worker looks to claim a new task, how long should the task be reserved to that worker? (If the lease expires without action by the worker, then it is assumed that the worker failed)
• int $batch_limit (eg 1): When a single worker looks to claim new tasks, what is the highest number of tasks they may take?
• ?int retry_limit (eg 3): If a task fails, it may be tried again later. What is the maximum number of times to retry?
• ?int retry_interval (eg 1200 sec): If a task fails, how long should you wait before trying to execute it again?

Technical Details: General declarations

Following on recent work, there are a few ways to define a queue (eg Civi::queue or hook_managed). Here are a few examples using the new options:

// Initialize with runtime OOP API - Civi::queue()
$queue = Civi::queue('my-stuff', [
  'type' => 'Sql',
  'runner' => 'task',
  'lease_time' => 3600,
  'batch_limit' => 1,
  'retry_limit' => 2,
  'retry_interval' => 600,
]);

// Initialize via hook_managed API
function queuex_civicrm_managed(&$entities): void {
  $entities[] = [
    'module' => 'queuex',
    'name' => 'my-emails',
    'entity' => 'Queue',
    'params' => [
      'version' => 4,
      'values' => [
        'name' => 'my-emails',
        'type' => 'Sql',
        'runner' => 'my_email_sender',
        'lease_time' => 300,
        'batch_limit' => 25,
        'retry_limit' => 4,
        'retry_interval' => 600,
      ],
    ],
  ];
}

Technical Details: runner declarations

This PR only defines the schema elements. However, it may help to describe where the runner element is going.

As with Civi's existing queue APIs, you may fill a queue with executable tasks. Set runner=>task to execute them automatically.

Civi::queue('my-tasks', [
  'type' => 'Sql',
  'runner' => 'task',
]);

Civi::queue('my-tasks')->createItem(new CRM_Queue_Task('some_callback_func', ['Alice'));
Civi::queue('my-tasks')->createItem(new CRM_Queue_Task('some_callback_func', ['Bob'));

function some_callback_func(CRM_Queue_TaskContext $ctx, $name) {
  Civi::log()->info("Hello $name");
  return TRUE;
}

This has some pros/cons. It has the benefit of consistency/precedent. You can enqueue many qualitatively different tasks (with different callbacks). It requires minimal boilerplate for each task. However, only trusted parties (ie local PHP code) may write to a task-queue. If you have 10 related items come into the queue at the same time, these will be executed separately (probably by different worker-threads).

Instead of using open-ended/executable tasks, you may alternatively define your own runner=>mygreeting.

Civi::queue('my-people-greeter', [
  'type' => 'Sql',
  'runner' => 'mygreeting',
  'batch_limit' => 20,
]);

Civi::queue('my-people-greeter')->createItem(['name' => 'Alice']);
Civi::queue('my-people-greeter')->createItem(['name' => 'Bob']);

// Implement hook_civicrm_queueRun_{$runner}
function hook_civicrm_queueRun_mygreeting(CRM_Queue_Queue $queue, array $items, array &$outcomes) {
  $names = implode(', ', array_map(\fn($item) => $item->data['name'], $items));
  Civi::log()->info("These are some fine people: {$names}");
  $queue->deleteItems($items);
  foreach ($items as $n => $item) {
    $outcomes[$n] = 'ok';
  }
}

This variant supports batched-queuing (eg you can receive batch_limit=>20 items at a time). It does not accept arbitrary callbacks -- queues which use runner=>foobar may be amenable to accepting work from outside sources. However, batched-queuing requires more thoughtfulness about error-detection and error-handling.

(For this PR, I don't want to dig too far into the implementation of different runners - that'll be another PR. I'm just trying to demonstrate the kind of choice is offered by the runner option.).

There is no direct test-coverage for declaring the schema. However, the functionality in #22762 uses this schema and introduces test-coverage.

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

outside the scope of this PR - but batch_limit is NOT int|string - it is int and " Note that values will be retrieved from the database as a string" is not true - this is coming from gencode from the looks

[totten]

Yeah, that is confusing.

FWIW, I modeled the XML for this field after Event.xml's max_participants (which seemed analogous). The max_participants has the same issue (ie it lists string as a type even though it's really only for numeric data).

[eileenmcnaughton]

I guess it's not required but it feels like we could have a callback to get the list of registered options

[totten]

I was thinking about that too...

If sysadmins were using a GUI to define new queues, then it would be important to have that list. But I don't personally see the use-case for it right now. AFAICS, the use-cases are developer-driven ("Copy-paste 8-line snippet from dev-docs.")

The flip-side is that adding any pseudoconstant for runner would mean it becomes an affirmative obligation to register new values (otherwise validation will fail). So any custom runner requires 2x registrations. Compare: Registering a queue with custom runner. Should it require 1x records (Queue) or 2x records (Queue+OptionValue).

[eileenmcnaughton]

@totten yeah - it does mean it becomes harder to add later - but I am ok with leaving for now

[eileenmcnaughton]

note I don't really see having to do 2 registrations as a downside

[eileenmcnaughton]

@totten most of my comments are superficial - but from my pre-r-run understanding of this PR the handling for the civicrm_queue table seems pretty consistent across definition/ upgrade but I'm not sure the queue_item table is.....

[eileenmcnaughton]

OK - my show create on queue_item after upgrade - I'll do a fresh install & compare

CREATE TABLE `civicrm_queue_item` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `queue_name` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Name of the queue which includes this item',
  `weight` int(11) NOT NULL,
  `submit_time` datetime NOT NULL COMMENT 'date on which this item was submitted to the queue',
  `release_time` datetime DEFAULT NULL COMMENT 'date on which this job becomes available; null if ASAP',
  `data` longtext COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Serialized queue data',
  `run_count` int(11) NOT NULL DEFAULT 0 COMMENT 'Number of times execution has been attempted.',
  PRIMARY KEY (`id`),
  KEY `index_queueids` (`queue_name`,`weight`,`id`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC;


CREATE TABLE `civicrm_queue` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Name of the queue',
  `type` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Type of the queue',
  `runner` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Name of the task runner',
  `batch_limit` int(10) unsigned NOT NULL DEFAULT 1 COMMENT 'Maximum number of items in a batch.',
  `lease_time` int(10) unsigned NOT NULL DEFAULT 3600 COMMENT 'When claiming an item (or batch of items) for work, how long should the item(s) be reserved. (Seconds)',
  `retry_limit` int(11) DEFAULT NULL COMMENT 'Number of permitted retries. Decreases with each retry. Zero (0) to disable. Null for system default.',
  `retry_interval` int(11) DEFAULT NULL COMMENT 'Number of seconds to wait before retrying a failed execution.',
  PRIMARY KEY (`id`),
  UNIQUE KEY `UI_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC
;

[eileenmcnaughton]

fresh install

CREATE TABLE `civicrm_queue_item` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `queue_name` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Name of the queue which includes this item',
  `weight` int(11) NOT NULL,
  `submit_time` datetime NOT NULL COMMENT 'date on which this item was submitted to the queue',
  `release_time` datetime DEFAULT NULL COMMENT 'date on which this job becomes available; null if ASAP',
  `run_count` int(11) NOT NULL DEFAULT 0 COMMENT 'Number of times execution has been attempted.',
  `data` longtext COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Serialized queue data',
  PRIMARY KEY (`id`),
  KEY `index_queueids` (`queue_name`,`weight`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC

CREATE TABLE `civicrm_queue` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Name of the queue',
  `type` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Type of the queue',
  `runner` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Name of the task runner',
  `batch_limit` int(10) unsigned NOT NULL DEFAULT 1 COMMENT 'Maximum number of items in a batch. Tip: If you expand batch_limit, then also consider expanding lease_time.',
  `lease_time` int(10) unsigned NOT NULL DEFAULT 3600 COMMENT 'When claiming an item (or batch of items) for work, how long should the item(s) be reserved. (Seconds)',
  `retry_limit` int(11) DEFAULT NULL COMMENT 'Number of permitted retries. Decreases with each retry. Zero (0) to disable. Null for system default.',
  `retry_interval` int(11) DEFAULT NULL COMMENT 'Number of seconds to wait before retrying a failed execution.',
  PRIMARY KEY (`id`),
  UNIQUE KEY `UI_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC

[eileenmcnaughton]

So the above looks right - it's just this bit I don't understand....

civicrm-core/sql/civicrm_queue_item.mysql

Lines 11 to 33 in cf97aee

	CREATE TABLE `civicrm_queue_item` (
	`id` int unsigned NOT NULL AUTO_INCREMENT ,
	`queue_name` varchar(64) NOT NULL COMMENT 'Name of the queue which includes this item',
	`weight` int NOT NULL ,
	`submit_time` datetime NOT NULL COMMENT 'date on which this item was submitted to the queue',
	`release_time` datetime COMMENT 'date on which this job becomes available; null if ASAP',
	`retry_interval` int NULL COMMENT 'Number of seconds to wait before retrying a failed execution. NULL to disable.',
	`retry_count` int NULL COMMENT 'Number of permitted retries. Decreases with each retry. NULL to disable.',
	`data` text COMMENT 'Serialized queue'
	,
	PRIMARY KEY ( `id` )
	, INDEX `index_queueids`(
	`queue_name`
	, `weight`
	, `id`
	)
	) ENGINE=InnoDB DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ;

[totten]

@eileenmcnaughton I removed the sql/civicrm_queue_item.mysql and ran a few upgrade-tests locally. Based on MINIMUM_UPGRADABLE_VERSION = '4.6.12', I thought these were relevant to try:

• 4.6.36, 4.7.0, 4.7.31: These ran fine and ended-up with civicrm_queue_item containing the new run_count.
• 4.6.0, 4.6.11: As expected, these refused to upgrade. They gave a well-formed error.

  CiviCRM versions prior to 4.6.12 cannot be upgraded directly to 5.48.alpha1. This upgrade will need to be done in stages. First download an intermediate version (the LTS may be a good choice) and upgrade to that before proceeding to this version.

[eileenmcnaughton]

OK - I think getting rid of that file is the right answer.

I think when we do version removals it might be good to suggest the specific release when upgrading stopped working as "the LTS may be a good choice" is pretty meaningless now.

It seems 4.7 is our lowest supported version

[totten]

Failures are unrelated.