This app provides an event timeout function for Seq. It will read an input stream and monitor for matching text strings on configured properties, during a configured start and end time.
It includes a timeout and suppression interval, which allows you to fine tune the way the timeout works, such as waiting 10 seconds or 4 hours for an event to occur, and then not alerting again for 1 hour.
When a timeout occurs, it will output the configured text and description back to the stream, which can be used as the basis for a signal
This is a highly configurable and powerful Seq app. Consider some of the possible usages:
- Monitor a signal for @Message containing "started" between 1am and 3am, and alert if not seen within 30 minutes
- Monitor a signal for @Message containing ANY value and ServerName containing "MYSERVER" between 12am and 12am (24 hour period), and alert if not seen within 30 minutes
- Monitor a signal for ServerName containing "MYSERVER" and Status containing "Succeeded" between 2am and 3am, and alert if not seen within 10 minutes
- Monitor a signal for @Message containing "started" and ServerName containing "MYSERVER" and JobName containing "Backup" and Component containing "SQL" between 12am and 6am, and alert if not seen within 1 hour
- Monitor a signal for @Message containing ANY value between 12am and 1am on Monday-Friday, and alert if not seen within 20 minutes
- Monitor a signal for @Message containing ANY value between 1am and 2am on Monday-Friday excluding public holidays, and alert if not seen within 10 minutes
- Monitor a signal for @Message containing ANY value between 2am and 3am on the first day of the month and alert if not seen within 20 minutes
- Monitor a signal for @Message containing "stopped" between 3am and 4am on the fourth Friday of the month and alert if not seen within 5 minutes
- Monitor a signal for Status containing "failed" between 4am and 5am on the last weekday of the month and alert if not seen within 45 minutes
- Monitor a signal for @Message containing ANY value between 5am and 8am on the first day, first weekday, second monday, fifth friday, last weekday, and last day of the month, excluding public holidays, and alert if not seen within 30 minutes
- Monitor a signal for @Message containing "success" between 8am and 12pm on Monday-Friday, excluding the third monday and last weekday, excluding public holidays, and alert if not seen within 2 hours
- Monitor a signal for @Message containing ANY value between 12am and 12am every day, and reset the timeout when an event is matched
There are many possible ways to configure Event Timeout!
Date/time is converted to UTC time internally, so that the start and end times are always handled correctly when considering local timezone and daylight savings.
Event Timeout includes the optional ability to retrieve public holidays using AbstractApi's Public Holidays API which can retrieve your local and national public holidays.
- You can configure Event Timeout to look for holiday types and locales, so that only (for example) National and Local holidays in Australia or New South Wales will be effective.
- Events with "Bank Holiday" in the name are excluded by default, but can be enabled
- Weekends are excluded by default, but can be enabled
- Retrieval of holidays occurs once per instance per day, at 12am (local time). If an event monitoring period ("Showtime") is in progress, it will only occur after an event monitoring period has ended. If one is scheduled, it will be delayed until holidays are retrieved.
- The Holidays API free tier limits requests to one per second, so a 10 second retry is configured for up to 10 attempts per instance
- This allows even the free Holidays API pricing tier to be used for most cases.
- Proxy configuration is included for Seq instances that do not have outbound internet access
A summary of the full feature set of Event Timeout follows:
| Feature | Description | Benefit |
|---|---|---|
| Monitor configured signals | Standard Seq functionality - required for Event Timeout to work effectively | Each instance can monitor one or more signals (intersecting) and action them based on the configured criteria |
| Configurable start and end times | Watch for events between two times, to detect that an expected event did not occur | Event Timeout can be configured to any interval in a 24 hour period, such as midnight to midnight, or 8am to 10am. This allows fine grained control over when alerts may be raised your critical application processes. |
| Configurable days of week | Control the days that the timeout monitoring will start | Events do not always occur on a 7 day basis - for example, a process that only occurs Monday to Friday, but not Saturday and Sunday. This avoids false positive alerts being raised. |
| Configurable "include" day expressions | Include specific days of the month - first,last,first weekday,last weekday,first/second/third/fourth/fifth/last sunday-saturday,1-31 | Fine grained control over how events are monitored; using this will allow you to ONLY monitor on specific days of the month. |
| Configurable "exclude" day expressions | Exclude specific days of the month - first,last,first weekday,last weekday,first/second/third/fourth/fifth/last sunday-saturday,1-31 | Fine grained control over how events are monitored; using this will allow you to EXCLUDE specific days of the month - even if it overrides an "Include" expression. |
| Up to 4 configurable properties to match | Case-insensitive partial string match against configured property names for the event text to monitor. Property 1 defaults to @Message and the other properties are optional. | This version of Event Timeout relies on matching between 1 and 4 properties with a configured text string that represents the event that is being watched for. If the match value is empty, ANY value will match. This allows filtering the input stream to ensure that only the specific event is matched. |
| Configurable timeout interval | Within the start and end times, an interval in which the event must be seen before an alert is output | Within the configured start and end times, there must be an interval in which the event is expected to be seen. This could be 60 seconds, 60 minutes, 12 hours - but this provides the timeout trigger for an alert to be raised. |
| Repeatable timeouts | After a match is made, optionally reset the timeout to match future events - a "heartbeat" type of functionality | Within the configured start and end times, there must be an interval in which the event is expected to be seen. This could be 60 seconds, 60 minutes, 12 hours - but this provides the timeout trigger for an alert to be raised. |
| Configurable suppression interval | An interval during which further alerts are suppressed | Event Timeout evaluates whether it has matched an event once per second, so without a suppression interval, 60 alerts per minute would be output. A suppression interval allows for an alert to be raised after the suppression interval expires, such as 1 minute, 1 hour, 6 hours. This is useful to detect an error that has not been corrected. |
| Configurable alert message and description | Detailed text that will be output as an event to the stream when a timeout occurs. | An alert "Message : Description" will be output to the stream, allowing this to be matched in multiple ways for signals, dashboards, and alerts. |
| Configurable alert level | An error level that will be assigned to the alert. | The default alert error level is "Error", but can be overridden to Verbose, Debug, Information, Warning, Error, or Fatal for filtering purposes. |
| Configurable tags | Tags added to the log entries for the app | The tags may be of use in configuring signals or queries to match specific events. |
| Optional diagnostic logging | Extensive diagnostic logging can be enabled to permit clarity on what's happening "under the hood" | A time-based event logging app can make it complex to understand what happened and when. Enabling the diagnostic logging provides a greater deal of clarity over the workings of Event Timeout, which can be particularly helpful if you suspect something is wrong. |
| UTC-based operation | Configured start/end times are converted to, and managed as, UTC times to avoid the complexity of time zone offsets. | Consistent handling of time intervals, and the start/end times will work across daylight saving time boundaries so that events continue to be monitored between the correct local times. |
| Public holiday automation | Using the AbstractApi Holidays API, retrieve a list of public holidays, which can be filtered on; * Type (eg. National, Local, Religious) - text insensitive partial string match * Location (eg. Australia, New South wales) - case insensitive full string match * Weekends - Disable public holidays falling on weekends by default * Bank Holidays - filter events with "Bank Holiday" in the name by default This implementation allows a test date to be configured, for simulation of a given holiday. It also includes configuration for proxy servers for Seq instances that do not have direct outbound accesss. |
AbstractApi offer a 'Holidays API' that can provide up to date public holidays for specific countries and locations (eg. states). Their REST API offering includes a free tier, which permits 1000 requests per month, 1 request per second. https://www.abstractapi.com/holidays-api Event Timeout implements Holidays API with an approach that will fit the free tier in most cases. Each instance of Event Timeout: * retrieves the current day's public holidays once per day (when the local date changes) * retries every 10 seconds, up to 10 times, to allow for cases where the 1 request per second limit has collisions between instances * optionally filters the holiday feed based on the configured Type case-insensitive partial string match, to allow filtering for only National and Local holidays (for example) * optionally filters the holiday feed based on the configured Location case-insensitive full string match, to allow filtering for only country and state events (for example). A full string match is required to avoid false positives. * filter weekends - by default, public holidays falling on Saturday or Sunday will be filtered out, but can be enabled. This does not override the configured days of week. * filter Bank Holidays - by default, a hard-coded case-insensitive partial string match on the holiday name is performed. If it matches "Bank Holiday" it is filtered out, but can be enabled. * "Test Date" - an ability to simulate a specific date to see the behaviour on a given public holiday. This has been deeply integrated so that the Start/End time will correctly reflect the "next UTC day" after the public holiday has ended. Operationally: * Retrieved public holidays do not interrupt an interval ("Showtime") that is already in progress - for example, a start time at 10:00pm local time which then ends at 2:00am on a public holiday. * Start and end times are recalculated after a UTC day rollover, once a "Showtime" interval is no longer in progress. * At recalculation, the active public holidays will be taken into consideration, so the above example would mean that the next start time, falling at 10:00pm on the public holiday, will be moved to the next day. |
The application has a large amount of configuration items. Here's an example instance config which will monitor a signal between 12:00pm and 12:05pm on Monday-Friday (except public holidays) for @Message containing "Test Event Text", and alert if not found.
This uses application logic that ensures that if Property 1 Name is not set, then @Message will be matched.
| Property | Value |
|---|---|
| Title | TestEvent |
| Stream Incoming Events | Enabled |
| Signal | TestSignal |
| Allow manual input | Disabled |
| Re-order input by timestamp | Disabled |
| Start Time | 12:00:00 |
| End Time | 12:05:00 |
| Timeout Interval (seconds) | 120 |
| Repeat timeout | Disabled |
| Days of Week | Monday,Tuesday,Wednesday,Thursday,Friday |
| Include Days of Month | |
| Exclude Days of Month | |
| Suppression Interval (seconds) | 60 |
| Log level for timeouts | Error |
| Property 1 Name | |
| Property 1 Match | Test Event Text |
| Property 2 Name | |
| Property 2 Match | |
| Property 3 Name | |
| Property 3 Match | |
| Property 4 Name | |
| Property 4 Match | |
| Alert message | Test Event Failure! |
| Alert description | The Test Event Text was not seen in the expected timeframe! |
| Alert tags | |
| Include instance name in alert message | Enabled |
| Use Holidays API for public holiday detection | Enabled |
| Country code | AU |
| Holidays API Key | Per Abstract API account |
| Match these holidays | National,Local |
| Local Holidays | Australia,New South Wales |
| Include weekends | Disabled |
| Include Bank Holidays | Disabled |
| Test Date | |
| Proxy address | |
| Proxy bypass local addresses | Disabled |
| Local addresses for proxy bypass | |
| Proxy username | |
| Proxy password |
An example using multiple properties over a 24 hour period, except for public holidays and the first day of the month:
| Property | Value |
|---|---|
| Title | TestEvent2 |
| Stream Incoming Events | Enabled |
| Signal | TestSignal |
| Allow manual input | Disabled |
| Re-order input by timestamp | Disabled |
| Start Time | 0:00:00 |
| End Time | 0:00:00 |
| Timeout Interval (seconds) | 3600 |
| Repeat timeout | Disabled |
| Days of Week | Monday,Tuesday,Wednesday,Thursday,Friday |
| Include Days of Month | |
| Exclude Days of Month | first |
| Suppression Interval (seconds) | 3600 |
| Log level for timeouts | Error |
| Property 1 Name | Status |
| Property 1 Match | Error |
| Property 2 Name | JobName |
| Property 2 Match | TestJob |
| Property 3 Name | Alerts |
| Property 3 Match | 1 |
| Property 4 Name | |
| Property 4 Match | |
| Alert message | Test Event 2 Failure! |
| Alert description | The Test Event 2 Text was not seen in the expected timeframe! |
| Alert tags | |
| Include instance name in alert message | Enabled |
| Use Holidays API for public holiday detection | Enabled |
| Country code | AU |
| Holidays API Key | Per Abstract API account |
| Match these holidays | National,Local |
| Local Holidays | Australia,New South Wales |
| Include weekends | Disabled |
| Include Bank Holidays | Disabled |
| Test Date | |
| Proxy address | |
| Proxy bypass local addresses | Disabled |
| Local addresses for proxy bypass | |
| Proxy username | |
| Proxy password |