From 8834d141b0a392294c6ed2e619a6ae71f8035c13 Mon Sep 17 00:00:00 2001 From: Steven Bal Date: Thu, 24 Oct 2024 15:54:32 +0200 Subject: [PATCH] :memo: Update delivery guarantee documentation this still used outdated information, the envvars are now only used by setup configuration and the config can be done via the admin --- docs/delivery_guarantees.rst | 55 +++++++++++++------ .../configuration/opennotifs_config_cli.rst | 2 + 2 files changed, 41 insertions(+), 16 deletions(-) diff --git a/docs/delivery_guarantees.rst b/docs/delivery_guarantees.rst index 2b1d7b1a..f9234f81 100644 --- a/docs/delivery_guarantees.rst +++ b/docs/delivery_guarantees.rst @@ -65,22 +65,45 @@ practice, Open Notificaties will consider any ``2xx`` response status code as **Retry mechanism** -On delivery failure, Open Notificaties will automatically retry delivery, with -exponential backoff. E.g. the first retry will be done after one second, the next after -two seconds and the third after 4 seconds. - -The parameters for this can be :ref:`configured `: - -* ``NOTIFICATION_DELIVERY_MAX_RETRIES``: the maximum number of automatic retries. After - this amount of retries, Open Notificaties stops trying to deliver the message. -* ``NOTIFICATION_DELIVERY_RETRY_BACKOFF``: if specified, a factor applied to the - exponential backoff. This allows you to tune how quickly automatic retries are - performed. -* ``NOTIFICATION_DELIVERY_RETRY_BACKOFF_MAX``: an upper limit to the exponential - backoff time. - -On top of that, via the admin interface you can manually select notifications to retry -delivery for. +By default, sending notifications to subscribers has automatic retry behaviour, i.e. if the notification +publishing task has failed, it will automatically be rescheduled/tried again until the maximum +retry limit has been reached. + +**Autoretry explanation and configuration** + +Retry behaviour is implemented using binary exponential backoff with a delay factor, +the formula to calculate the time to wait until the next retry is as follows: + +.. math:: + t = \text{backoff_factor} * 2^c + +where `t` is time in seconds and `c` is the number of retries that have been performed already. + +This behaviour can be configured using :ref:`setup_configuration ` +and also via the admin interface at **Configuratie > Notificatiescomponentconfiguratie**: + +* **Notification delivery max retries**: the maximum number of retries the task queue + will do if sending a notification has failed. Default is ``5``. +* **Notification delivery retry backoff**: a boolean or a number. If this option is set to + ``True``, autoretries will be delayed following the rules of binary exponential backoff. If + this option is set to a number, it is used as a delay factor. Default is ``3``. +* **Notification delivery retry backoff max**: an integer, specifying number of seconds. + If ``Notification delivery retry backoff`` is enabled, this option will set a maximum + delay in seconds between task autoretries. Default is ``48`` seconds. + +With the assumption that the requests are done immediately we can model the notification +tasks schedule with the default configurations: + +1. At 0s the request to send a Notification to a subscriber is made, the notification task is scheduled, picked up + by worker and failed +2. At 3s with 3s delay the first retry happens (``2^0`` * ``Notification delivery retry backoff``) +3. At 9s with 6s delay - the second retry (``2^1`` * ``Notification delivery retry backoff``) +4. At 21s with 12s delay - the third retry +5. At 45s with 24s delay - the fourth retry +6. At 1m33s with 48s delay - the fifth retry, which is the last one. + +So if the subscribed webhooks is up after 1 min of downtime the default configuration can handle it +automatically. **Open Notificaties message broker** diff --git a/docs/installation/configuration/opennotifs_config_cli.rst b/docs/installation/configuration/opennotifs_config_cli.rst index 7b1cfc75..5ef75a99 100644 --- a/docs/installation/configuration/opennotifs_config_cli.rst +++ b/docs/installation/configuration/opennotifs_config_cli.rst @@ -65,6 +65,8 @@ Make sure that the correct permissions are configured in Open Zaak Autorisaties for example, ``open-zaak``. Required. * ``OPENZAAK_NOTIF_SECRET``: some random string. Required. +.. _installation_configuration_cli_retry: + Notification retry configuration --------------------------------