Review parsed data and database schema
*/db_view/
While the Beat! Button only provides a limited result (up to 10 rows from each log), a log's data can be fully explored through the Database Query page.
Results are always sorted by time, with a specific start or end time specified.
They default to Starting : one week before now and Ending : now on page load.
A variety of filters, settings, and log-specific pre-assembled queries are provided. The image above shows the Basic options. The optional Location and IP Address filters can be toggled; IP Address form is not shown.
Detailed query descriptions follow the image.
Query Descriptions
| Log - Query | Description |
|---|---|
| ALL - Basic | Simply use the options specified at the top of the page. They default to all entries, going back from now, limited to 50 rows per page. |
| Access - Ignorable | Hits matching Home Ignorable specification, if set. |
| Access - Known Devices | Hits with user-agents matching Known Devices specification, if set. |
| Access - Filtrate | Outside hits with IPs not banned by fail2ban. Matches limited to one-week blocks. |
| Access - Regex2 | Log data processed by the default secondary regex method. They lack data for HTTP protocol version and request method. |
| Access - HTTP v X | Hits with the specified HTTP network protocol version. One grouping for 2.0 and another for 1.0, 1.1 |
| Access - HTTP Xxx | Hits with the specified HTTP response status code grouping. see Neat Reference |
| Error - IP v X | Internet Protocol address version. IPv4 or IPv6 |
| Error - Filtrate | Outside hits with IPs not banned by fail2ban. Matches limited to one-week blocks. |
| Error - Level | All available Error Log severity levels for entries are provided. Parsed data info |
| fail2ban - Ignores | fail2ban entries with the "Ignore" action. I have fail2ban ignore my local, "Home IP", to allow me to tailor filters. |
| fail2ban - Match Ignores | Attempt to match fail2ban ignores with home hits on the access log. Home Ignorable specification may improve matching. Same table as shown in report. |
| fail2ban - Filter | query for each fail2ban filter |
Coordinates, Cities, and Counties are added to the Results table if any of the results have location data. Unlike the report, the presented data is not styled in any way, providing a raw view of the database. Data styling is retained for Match Ignores, however, which attempts to combine associated entries from the fail2ban and access logs.
If the there are more results than the size limit, the page provides a link to view more data.
If the Next button is used, a Previous button is provided on the resulting page. This will only backtrack to its source page. For this reason, it may be beneficial to open Next results in a new tab or window.
The SQL statement used to generate the table can be viewed for each result.
Conveniently, it can be copied to the clipboard with a click.
This action may be disabled by web browser, SSL certificate may be required.
*/data_cleaning/
Data saved from log parsing can be deleted from the database, based on date.
Data removal must be confirmed after estimation. Canceling will allow another estimate.
*/failed_regex/
Any line that fails parsing will be saved and categorized according to log file. Currently, BeatLog provides limited interaction with the failed lines. You can clear a log's failed regex or view a sample of the failed lines. I have not had any parsing failures using my default regex methods, therefore I didn't specify a Secondary regex method to generate failed lines for an example.
Individual log parsing result indicates failed lines, investigate on Failed Regex page:
I have some ideas of things to add:
- Test Regex for log's failed lines
- provide more detailed information on how/where regex failed
- check Time Skip method only
- Attempt to parse again and save into database
- . . .
Please submit any parsing failures you encounter with the default methods and/or features you might want with the failed lines.