Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to configure ingest node to have geoip #2793

Merged
merged 2 commits into from
Oct 19, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion libbeat/docs/outputconfig.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,16 @@ output.elasticsearch:

===== pipeline

format string value configuring the ingest node pipeline to write events to.
A format string value that specifies the ingest node pipeline to write events to.

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
pipeline: my_pipeline_id
------------------------------------------------------------------------------

For more information, see <<configuring-ingest-node>>.

===== pipelines

Expand Down
4 changes: 2 additions & 2 deletions libbeat/docs/shared-config-ingest.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ After defining the pipeline in Elasticsearch, you simply configure your Beat to
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
parameters.pipeline: my_pipeline_id
pipeline: my_pipeline_id
------------------------------------------------------------------------------

For example, let's say that you've defined the following pipeline in a file named `pipeline.json`:
Expand Down Expand Up @@ -59,7 +59,7 @@ Then in the +{beatname_lc}.yml+ file, you would specify:
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
parameters.pipeline: "test-pipeline"
pipeline: "test-pipeline"
------------------------------------------------------------------------------

When you run {beatname_uc}, the value of `beat.name` is converted to lowercase before indexing.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packetbeat/docs/images/kibana-update-map.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions packetbeat/docs/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ include::./version.asciidoc[]
:filebeat: http://www.elastic.co/guide/en/beats/filebeat/{doc-branch}
:winlogbeat: http://www.elastic.co/guide/en/beats/winlogbeat/{doc-branch}
:elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch}
:logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch}
:securitydoc: https://www.elastic.co/guide/en/x-pack/5.0
:kibanadoc: https://www.elastic.co/guide/en/kibana/{doc-branch}
:plugindoc: https://www.elastic.co/guide/en/elasticsearch/plugins/{doc-branch}
:version: {stack-version}
:beatname_lc: packetbeat
:beatname_uc: Packetbeat
Expand All @@ -30,6 +33,8 @@ include::./packetbeat-filtering.asciidoc[]

include::../../libbeat/docs/shared-config-ingest.asciidoc[]

include::./packetbeat-geoip.asciidoc[]

include::./configuring-logstash.asciidoc[]

include::../../libbeat/docs/shared-env-vars.asciidoc[]
Expand Down
102 changes: 102 additions & 0 deletions packetbeat/docs/packetbeat-geoip.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
[[packetbeat-geoip]]
== Exporting GeoIP Information

You can use Packetbeat along with the
{plugindoc}/ingest-geoip.html[ingest geoIP processor plugin] in Elasticsearch
to export geographic location information about source IPs for incoming HTTP
requests. Then you can use this info to visualize the location of your
clients on a map in Kibana.

Prior to version 5.0, Packetbeat provided a `geoip` configuration option for
exporting geoIP information about the source IPs. Starting with 5.0, the
`geoip` configuration option in Beats is deprecated in favor of using the
ingest geoIP processor plugin. This plugin adds information about the
geographical location of IP addresses, based on data from the Maxmind GeoLite2
City Database. Because the plugin uses a geoIP database that's installed on
Elasticsearch, you no longer need to install a geoIP database on the
machines running Beats.

NOTE: If your use case involves using Logstash, you can use the
{logstashdoc}/plugins-filters-geoip.html[GeoIP filter] available in Logstash
instead of using the ingest plugin. However, using the ingest plugin is the
simplest approach when you don't require the additional processing power of
Logstash.

[float]
[[packetbeat-configuring-geoip]]
=== Configuring the Ingest GeoIP Processor Plugin

To configure Packetbeat and the ingest geoIP processor plugin:

1. {plugindoc}/ingest-geoip.html[Install the ingest geoIP processor plugin].
After installing the plugin, remember to restart the node.

2. Define an ingest node pipeline that uses a `geoip` processor to add location
info to the event. For example, you can use the Console in Kibana to create the
following pipeline:
+
[source,json]
-------------------------------------------------------------------------------
PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{
"geoip": {
"field": "client_ip",
"target_field": "client_geoip",
"properties": ["location"],
"ignore_failure": true
}
}
]
}
-------------------------------------------------------------------------------
+
This pipeline adds a `client_geoip.location` field of type `geo_point` to the
event. The ID of the pipeline is `geoip-info`. `client_ip` is the output field
in Packetbeat that contains the IP address of the client. You set
`ignore_failure` to `true` so that the pipeline will continue processing events
when it encounters an event that doesn't have a `client_ip` field.
+
See
{plugindoc}/using-ingest-geoip.html[Using the Geoip Processor in a Pipeline]
for more options.

3. In the Packetbeat config file, configure the Elasticsearch output to use the
pipeline. Specify the pipeline ID in the `pipeline` option under
`output.elasticsearch`. For example:
+
[source,yaml]
-------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
pipeline: geoip-info
-------------------------------------------------------------------------------

4. Run Packbeat, passing in the configuration file that you updated earlier.
+
[source,shell]
-------------------------------------------------------------------------------
./packetbeat -e -c packetbeat.yml
-------------------------------------------------------------------------------
+
The event that's sent to Elasticsearch should now include a
`client_geoip.location` field.

[float]
[[packetbeat-visualizing-location]]
=== Visualizing the Location of Your Packetbeat Clients

To visualize the location of your Packetbeat clients, you can either
<<packetbeat-sample-dashboards,import the Kibana dashboard for Packetbeat>> (if
you haven't already), or create a new {kibanadoc}/tilemap.html[Tile map] in
Kibana and use the `client_geoip.location` field as the Geohash.

image:./images/kibana-update-map.png[Update Packetbeat client location map in Kibana]

TIP: If the map in the dashboard reports "no results found", and you don't see
`client_geoip.location` in the list of available Geohash fields, try refreshing
the field list in Kibana. On the Management tab, select the `packetbeat-*`
index pattern, and refresh the field list to pick up any fields that were added
by the ingest geoIP processor.