diff --git a/changelog.d/+howto-notification-testing.added.md b/changelog.d/+howto-notification-testing.added.md new file mode 100644 index 000000000..5f14c9734 --- /dev/null +++ b/changelog.d/+howto-notification-testing.added.md @@ -0,0 +1 @@ +Added documentation on how to safely test notifications. diff --git a/docs/development.rst b/docs/development.rst index ea0f5b146..babbd91bb 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -4,7 +4,6 @@ Development .. toctree:: - :glob: development/notes development/management-commands diff --git a/docs/development/howtos/notification-system-testing.rst b/docs/development/howtos/notification-system-testing.rst new file mode 100644 index 000000000..c172ea4b3 --- /dev/null +++ b/docs/development/howtos/notification-system-testing.rst @@ -0,0 +1,157 @@ +=========================================== +Howto: Safely test sending of notifications +=========================================== + +By "safely test" we mean "not accidentally spamming anyone with test incidents". + +Automated tests are well and good, but filters especially can be complex things +and can behave in what for users is an unexpected way. (The problem sometimes +does exist between keyboard and chair.) + +The techniques here are suitable both for problems in production, new types of +filters, unexpected behaviours, refactors of the entire shebang, and working +out how things actually work in order to improve automated tests. + +Testing filters, profiles, timeslots regardless of destination type +=================================================================== + +When doing a test of everything *except* details of the destination type, use +the standard email medium +(``argus.notificationprofile.media.email.EmailNotification``) as the sole +destination type. Adapt or create a profile accordingly. + +Automated tests +--------------- + +Use the `override_settings`_ decorator to override the setting +``MEDIA_PLUGINS`` with the exact list of media the test will need. + +Redirect email to console +========================= + +Do not actually send those notification emails, catch them locally. + +There are two ways of doing this: + +1. Changing the EMAIL_BACKEND setting to + ``django.core.mail.backends.console.EmailBackend``. Sending an email will + then turn up in some terminal window somewhere. +2. Sending to an email server you control, the easiest is to send it to one + set up for the test. Set ``EMAIL_HOST`` to ``localhost``, ``EMAIL_PORT`` to + ``1025``, and run a dummy mailserver:: + + $ python -m smtpd -n -c DebuggingServer localhost:1025 + + Sent notifications will then be dumped to the console where the dummy server + runs. + +The former method works less well with docker since it can be tricky to find +the correct container that has the dump of the email, plus there's usually so +much going on in the console of the container that finding the message might be +tricky. With the latter method the email is dumped in the window the +DebuggingServer is running in, which allows a little more control. + +One of the two methods above is probably what you want for your +development-setup regardless. + +Automated tests +--------------- + +Use the `override_settings`_ decorator to override the setting +``EMAIL_BACKEND`` and use method 1 above. + +Testing something that fails in production +========================================== + +Copy the profiles to your debugging-setup. You can just reuse your +development-setup for this: move the dev-database out of the way (for instance +with the ``ALTER DATABASE .. RENAME`` statement), dump the production database, +then import the dump into your dev-database. + +Deactivate profiles you are not testing +======================================= + +Deactivating profiles prevents spam, makes the test go faster since there is +less to test, and makes it easier to find the result in the console since less +will be printed. (Don't do this in production regardless.) + +Find the ids of the profiles you want to test first. Let's assume you'll be +testing profiles "1" and "22". + +There are several methods: + +1. Deactivate all others in the admin. +2. If you have access to the database, deactivate via ``python manage.py dbshell``:: + + argus=> UPDATE argus_notificationprofile_notificationprofile SET active = false WHERE id NOT IN (1, 22); +3. You can deactivate via the python shell as well, launch with ``python + manage.py shell``:: + + > from argus.notificationprofile.models import NotificationProfile + > NotificationProfile.objects.exclude(id__in=(1, 22)).update(active=False) + +With option 1 you need an authenticated user in the system that has both +``is_staff`` and ``is_superuser`` set to ``True``. For all the others you need +database access. + +Automated tests +--------------- + +Use factories to create exactly what to test. You can recreate a scenario in +the production database exactly with the factories, then when the test works, +adjust the factory to only set exactly the subset of attributes needed. + +Testing specific destination types +================================== + +If the plugin has a dummy version that just dumps to the console, see if +testing with that dummy is sufficient. Otherwise you need to find a test +destination (in order to not spam the production destination). For instance +with the MS teams-plugin set it up to spam a test-channel or yourself. + +If the live test works but pushing to the production destination does not, it's +probably a permissions-problem. + +Email is not delivered +---------------------- + +If an email is not delivered even if the email server logs of the first hop +(the one set as ``EMAIL_HOST``) says the email was received, it is probably one +of three things: + +1. Something is wrong with the first hop, like a missing or outdated + certificate, wrong entry in DNS, missing DKIM/SPF/DMARC, missing reverse DNS + lookup, the IP address is on a blacklist. The possibilities are nearly + endless. +2. Something is wrong at the final hop (the email server receiving for the + domain in the ``To:``-address) +3. Something is wrong in between. Network, DNS, other email servers along the + route... + +This is not something you can solve alone. Send a fake incident to the failing +email address with the normal email backend in settings and use that to find +the exact: + +* date and time the non-delivered email was sent (do not forget the timezone) +* the ``To:``-address +* the ``From:``-address + +Give this to your postmaster (the person or persons responsible for/monitoring +the email system). That should be enough for them to sometimes solve case #1. + +Making test incidents to trigger the notification system +======================================================== + +While it is possible to sit in the python shell and create everything by hand +for 100% control, most of the time it is sufficient to use one of the below +methods: + +1. In the admin, go to the ``Incidents`` model in the "ARGUS_INCIDENT" section. + There's a button "FAKE INCIDENT" just left of the "ADD INCIDENT"-button. +2. From the command line use the ``create_fake_incident`` command with ``python + manage.py``. + +We do not recommend making an incident directly in the database, because the +event that triggers the notification is made by Python. + +.. _override_settings: https://docs.djangoproject.com/en/4.2/topics/testing/tools/#django.test.override_settings diff --git a/docs/development/notes.rst b/docs/development/notes.rst index a0b9169de..94eea1906 100644 --- a/docs/development/notes.rst +++ b/docs/development/notes.rst @@ -46,16 +46,3 @@ files later. .. warning:: Do not check your ``cmd.sh`` or ``settings.py`` files into version control, since they contain passwords and sensitive data. - - -Debugging tips --------------- - -To test/debug notifications as a whole, use the email subsystem (Media: Email in a -NotificationProfile). -Set ``EMAIL_HOST`` to ``localhost``, ``EMAIL_PORT`` to ``1025``, and run a dummy -mailserver:: - - $ python -m smtpd -n -c DebuggingServer localhost:1025 - -Sent notifications will then be dumped to the console on the dummy server.