Skip to content

Log Aggregation

Jump to bottom
Wuyi Chen edited this page Jul 3, 2019 · 11 revisions

Overview

Log aggregation basics

There are 2 main log aggregation solutions:

  • Local: Send log to the local log server.
  • Cloud: Send log to the cloud log server.

Send log to Splunk (Local)

Step 1: Set up a data input point

Step 1.1: Login the Splunk web portal

Step 1.2: Click "Add Data" under "Explore Splunk Enterprise" dashboard

Step 1.3: Choose "Monitor" under "Or get data in with the following methods" section

Step 1.4: Select Source

  • On the left side, select "TCP / UDP".
  • For choosing TCP or UDP, if you want to send the log by syslog traffic, use "UDP". You can also send the log by TCP traffic for better transmission reliability.
  • Set the port number.

Step 1.5: Input Settings

  • Source type
    • Choose "Select".
    • Select Source Type: Choosing "Application" -> "catalina".
  • App context
    • By default, you can choose "Search and Reporting (search)".
    • If you have created your own app, you can select the app you created.
  • Host
    • Splunk can track the log sender by DNS (hostname) or IP (address).
  • Index
    • Choose "Default".

Step 1.6: Review

Step 1.7: Done

Step 2: Let Spring application send the log to Splunk

Option 1: Use UDP traffic

If you set the type of the data input point as "UDP" in Splunk, you need to change the logback-spring.xml file under the resources directory like this:

logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="SYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
        <syslogHost>127.0.0.1</syslogHost>                                      <!-- Splunk server is running in local  -->
        <facility>SYSLOG</facility>
        <port>50000</port>                                                      <!-- Use the port number when you set up the UDP data input point  -->
        <suffixPattern> [%thread] %logger %msg</suffixPattern>
    </appender>
    <root level="INFO">
        <appender-ref ref="SYSLOG" />
    </root>
</configuration>

Option 2: Use TCP traffic

(Need to add)

Step 3: Verification

After you start the Spring application with the updated logback-spring.xml, you should see the log coming into the Splunk (with the app you chose).

Send log to Papertrail

References

Clone this wiki locally