The JSalParser Cli creates a command line interface to parse Extended Log Format files generated by Apache HTTP Server, AWS S3, or AWS CloudFront (to name a few) into Java POJOs (Plain Old Java Objects), enhance the data and convert to other formats.
It extends the JSalParser Project (https://github.com/hiramsoft/jsalparser).
java -jar jsalparser.jar \
--store-engine excel \
--input /var/log/s3 \
--output /mnt/data/parser.xlsx \
--enhancers geoip \
--enhancers useragent \
--enhancer-options geoip-database-path=/mnt/maxmind/country.mmdb| argument | Description |
|---|---|
| help | list all commands |
| input | Path where all log files are stored |
| output | Path to file that will be created after the process |
| store-engine | Define which store engine will be used. Allowed values: json,excel |
| locale | If the date on your log files is not formatted in English you can overwrite the locale using it. |
| enhancers | Which data enhancers do you want to use. Allowed values: geoip and useragent |
| enhancer-options | Options required to initialize some enhancers. Allowed values: geoip-account-id, geoip-licence-key, geoip-database-path |
| Enhancer | Description |
|---|---|
| geoip | Add requester country using GeoLite 2 (https://dev.maxmind.com/geoip/geoip2/geolite2/) |
| useragent | Add more data about the user agent using (https://github.com/woothee/woothee) |
I was requested to convert multiple s3 log files to excel to allow some non-programmers to do some analysis on the requests. In the beginning, I did an ah doc implementation, but this request became more regular. I created this command-line interface to convert those logs to excel or json in an extremely easy way and also allowing me to enhance some information when required.
- S3 Server Access Log Format: http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html
- CloudFront Access Log Format: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html
- AWS SDK for Java: http://aws.amazon.com/sdk-for-java/
- JODA DateTime: http://www.joda.org/joda-time/
- ANTLR: http://www.antlr.org/
- JSalParser Project: https://github.com/hiramsoft/jsalparser
- Geolite 2: https://dev.maxmind.com/geoip/geoip2/geolite2/
- Woothee: https://github.com/woothee/woothee
I will add tests as soon as possible.
BSD