missing ECS fields

[emalzer]

Logstash information:

1. Logstash version

# /usr/share/logstash/bin/logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 7.15.1

2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
   DEB / Ubuntu 18.04

3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
   systemd

4. How was the Logstash Plugin installed
   core plugin

JVM (e.g. java -version):

OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Linux minion-em-elk01 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behaviour:
The current ECS scheme (1.12) uses the field geo.region_iso_code field, the geoip filter plugin actually uses the geo.region_code field, which does not comply with the ECS scheme. Also it inserts the country_code2 and country_code3 (which is fine for me :) ), but not the geo.country_iso_code.

Steps to reproduce:

root@minion-em-elk01:/usr/share/logstash# ./bin/logstash -e 'input { stdin { codec => json } } filter { geoip { source => "ip" target => "client" } } output { stdout {} }'
....
{ "ip": "52.168.174.229" }
{
            "ip" => "52.168.174.229",
      "@version" => "1",
    "@timestamp" => 2021-12-06T09:32:26.081Z,
          "host" => "minion-em-elk01",
        "client" => {
              "latitude" => 37.9254,
              "location" => {
            "lat" => 37.9254,
            "lon" => -76.8591
        },
          "country_name" => "United States",
              "dma_code" => 556,
             "longitude" => -76.8591,
           "postal_code" => "22560",
         "country_code3" => "US",
         "country_code2" => "US",
           "region_code" => "VA",
        "continent_code" => "NA",
             "city_name" => "Tappahannock",
                    "ip" => "52.168.174.229",
              "timezone" => "America/New_York",
           "region_name" => "Virginia"
    }
}
^C[WARN ] 2021-12-06 10:32:40.735 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2021-12-06 10:32:40.964 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2021-12-06 10:32:41.931 [LogStash::Runner] runner - Logstash shut down.

[kaisecheng]

@emalzer ECS is not enabled by default in Logstash 7.15
The event layout you shared is not ECS compatible.
To enable it, go to pipeline.yml

- pipeline.id: geoip_test
  pipeline.ecs_compatibility: v1
  config.string: input { stdin { codec => json } } filter { geoip { source => "ip" target => "client" } } output { stdout {} }

and start logstash.

[emalzer]

@kaisecheng yes, you are right, I'm in the middle of mirgating to ECS and forgot the compatibility mode while trying to recreate the issue as simple as possible. I thus tried it again with the following configuration:

root@gs-dev00:/usr/share/logstash# ./bin/logstash -e 'input { stdin { codec => json } } filter { geoip { source => "ip" target => "client" } } output { stdout {} }' --pipeline.ecs_compatibility v1
...
{ "ip": "52.168.174.229" }
{
            "ip" => "52.168.174.229",
        "client" => {
          "ip" => "52.168.174.229",
        "mmdb" => {
            "dma_code" => 556
        },
         "geo" => {
                 "region_name" => "Virginia",
                 "postal_code" => "22560",
                    "location" => {
                "lon" => -76.8591,
                "lat" => 37.9254
            },
                   "city_name" => "Tappahannock",
                "country_name" => "United States",
                    "timezone" => "America/New_York",
              "continent_code" => "NA",
                 "region_code" => "VA",
            "country_iso_code" => "US"
        }
    },
    "@timestamp" => 2022-01-10T17:58:47.301Z,
         "event" => {
        "original" => "{ \"ip\": \"52.168.174.229\" }\n"
    },
          "host" => {
        "hostname" => "gs-dev00"
    },
      "@version" => "1"
}
^C[WARN ] 2022-01-10 19:03:14.274 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2022-01-10 19:03:14.457 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2022-01-10 19:03:15.388 [LogStash::Runner] runner - Logstash shut down.

I attached the --pipeline-compatibility v1 parameter. Then the fields are ECS compatible - except for the field region_code which should be region_iso_code.

[kaisecheng]

@emalzer I see your point! Looking at the example in ECS schema, geo.region_iso_code: CA-QC, geo.region_name: Quebec, the region_code that this plugin gives does not fit to any of them, but it leans toward region_iso_code.
From the Maxmind database, the value belongs to iso_code as well.

I will try to make it to region_iso_code and seek feedback from others. Thank you for reporting the issue!

[kaisecheng]

Fixed in #206