log-parser

Overview

Log-parser is a CLI tool used to calculate statistics based on logs from servers, e.g. Gunicorn. It was designed with a view to making it easy to extend with further commands, formats and statistics.

Quickstart

The package manager used in the project is Pipenv.

The installation of the log-parser is very simple. You should be in the root directory of the project and run the following command:

bash install.sh

Uninstalling is also easy.

pipenv uninstall log-parser

To run tests:

pytest tests

In the ./data directory in the project there is an example log file on which you can test the operation of the application.

Usage

Log-parser has one subcommand implemented, which is called stats. It is used to calculate statistics based on logs from the server.

The statistics currently implemented and calculated by CLI are:

• number of all requests
• number of individual request statuses
• number of requests per second
• average response size for 2xx

log-parser

---

Getting help for the main command:

log-parser -h

log-parser stats

---

Get help for the stats command:

log-parser stats -h

log-parser stats file

---

Calculating statistics for the entire log file.

log-parser stats data/gunicorn.log2

log-parser stats file --since

---

The log file can also be filtered by date. The first way is the --since flag.

An important issue is that the functionality is currently implemented in such a way that you must provide the full date in this format: 01/Dec/2019:05:07:05

log-parser stats data/gunicorn.log2 --since=01/Dec/2019:05:07:05

log-parser stats file --until

---

Another option for filtering by date is the --until flag:

log-parser stats data/gunicorn.log2 --until=01/Dec/2019:05:07:05

log-parser stats file --since --until

---

Flags can appear together:

log-parser stats data/gunicorn.log2 --since=01/Dec/2019:05:07:05 --until=01/Dec/2019:10:00:00

Project structure

log_parser
├── __init__.py
├── __main__.py
├── cli
│   ├── __init__.py
│   ├── args_parsers.py
│   └── log_parser.py
├── commands
│   ├── __init__.py
│   ├── icommand.py
│   └── stats
├── exceptions.py
├── parser
│   ├── __init__.py
│   ├── log_line.py
│   ├── parser.py
│   └── pattern_builder
├── renderers
│   ├── __init__.py
│   ├── error_renderer.py
│   ├── irenderer.py
│   └── stats_output_renderer.py
└── services
    ├── __init__.py
    └── date_service.py

CLI

In ./cli directory there is the main program skeleton and individual argument parsers. The addition of another subcommand can be started by creating a separate argument parser and adding a new method to the LogParser.

Commands

In ./commands there is a logic of the stats command, as well as individual statistics that are calculated within it.

Parser

Logic is implemented in this part of the project, which helps to parse individual records from the log file. Builder Pattern has also been used here, which aims to enable the creation of new patterns for other types of logs.

Renderers

In ./renderers there are classes that are responsible for displaying the results of the application's operation directly in the console.

TODO

1. Enable more user-friendly input of dates into the --since and --until filters.
2. Analyzing and refining the application in terms of algorithmics.