Facet filter for panel "term" give incomplete list

[jueadams]

Dear all,

first of all, many thanks for that execellent piece of software. I really love it, because it is really basd on an execellent (software) design and functionality.

I'm facing a very strange behaviour using "term" panels. The bool filter (from elastic.js ?) use a "must" for a "match_all" like:

{
  ...
    "filter": {
      "bool": {
----->  "must": [
          {
            "match_all": {}
          },
  ...
}

This query results for me an incomplete list and wrong numbers for the facets - a lot of fields for that facet query are missing and the facet is miscalcuated. I played a little and changed the "must" to a "should" and all fields are returned correctly (checked using ES head). Suprise, suprise.

It becomes more strange, because this is not for all facets of all term panels the case.

I'm not sure if this is an ES issue or a problem of Kibana. I won't/can't change the JAVA code - to complex for a beginner using Kibana.

Do you see a change to find out the reason more specific if this is an ES or Kibana issue?

Many thanks for any help,
Juergen

BTW: I'm using a Graylog2 index to analyse with Kibana. I reconfigured Kibana to use that mappings/timestamps. It took a while, but now I can't see any issues using GL2 and Kibana - all data is processed fine.

Except that strange thing ...

Details follows ...

[jueadams]

Dear all,

I found the issue. The mapping field (from Graylog2) for the timestamp is misinterpreted by Kibana. The timestamp comes with filed "histogram_time" like this:

2013-09-07 19-29-51

with no ticks and no time zone info. The timestamp is in UTC, so I've simply added 2 hours (my time zone is Berlin +100, Summertime) using the timepicker/absolute (Time). Strange, after adding 2 hours for To(now) all lists/facests are complete/correct.

Any idea?

Regards,
Jurgen

[spalger]

That sounds really strange. Can you copy/paste the mappings?

[jueadams]

Thanks to spenceralger. Yes, I'll add the missing mappings - tomorrow. It's 1am im Germany. Too much monkey work, today.

Thanks a lot,
Juergen

[jueadams]

Hi all,

the base mapping for the GELF data of an Graylog2 index is as follows:

{
"graylog2_recent": {
"message": {
"dynamic_templates": [
{
"store_generic": {
"mapping": {
"index": "not_analyzed"
},
"match": "_"
}
}
],
"_ttl": {
"enabled": true
},
"source": {
"compress": true
},
"properties": {
"created_at": {
"type": "double"
},
"full_message": {
"type": "string",
"analyzer": "standard"
},
"histogram_time": {
"type": "date",
"format": "yyyy-MM-dd HH-mm-ss"
},
"message": {
"type": "string",
"analyzer": "standard"
}
}
}
},
"kibana-int": {
"dashboard": {
"properties": {
"dashboard": {
"type": "string"
},
"group": {
"type": "string"
},
"title": {
"type": "string"
},
"user": {
"type": "string"
}
}
}
},
"graylog2_0": {
"message": {
"dynamic_templates": [
{
"store_generic": {
"mapping": {
"index": "not_analyzed"
},
"match": ""
}
}
],
"_ttl": {
"enabled": true
},
"_source": {
"compress": true
},
"properties": {
"created_at": {
"type": "double"
},
"full_message": {
"type": "string",
"analyzer": "standard"
},
"histogram_time": {
"type": "date",
"format": "yyyy-MM-dd HH-mm-ss"
},
"message": {
"type": "string",
"analyzer": "standard"
}
}
}
}
}

As I mentioned already a single mapping for a timestamp is given with "histogram_time "

      "histogram_time": {
      "type": "date",
      "format": "yyyy-MM-dd HH-mm-ss"
    }

My impression is a more general issue of Kibana to handle that format of a timestamp with no ticks and time zone info as used with logstash @timestamp:

2013-07-03T08:48:40.000Z

"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
}

I found that all facets are not calculated correctly. If I add 2 hours for the time period, all results are correct. Less than 2 hours added, the calculation went wrong as well.

The logstash mapping look like:

{
"logstash-2013.04.22": {
"logs": {
"properties": {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"@Version": {
"type": "string"
},
"Line": {
"type": "date",
"format": "dateOptionalTime"
},
"MessageEventTime": {
"type": "string"
},
"OpcodeValue": {
"type": "long"
},
"short_message": {
"type": "string"
},
"tags": {
"type": "string"
},
"type": {
"type": "string"
},
"version": {
"type": "string"
}
}
}
}
}

[jueadams]

Wrong button :)

[jueadams]

Dear all,

I found the reason why my data is incomplete - very simple. Lohstash/Kibana use timestamps like this 2013-07-03T08:48:40.000Z. All timestamps of Kibana are are expected in UTC. I use Kibana for a Graylog2 index. Graylog2 use a very simple time format like this yyyy-MM-dd HH-mm-ss.

If no time zone info is given with a timestamp, the tme is treated by Kibana as UTC. In my timezone MESZ = +02:00 the is always 2 hours ahead UTC. If the timerpicker is used for relativ periods of time (e.g. last 24 hours), this will cause to miss data for the differenrence between UTC and MESZ: 2 hours.

There is nothing wrong with Kibana. All timestamps of all messages in an index has to be in UTC to use the timepicker relativly. That is not that bad, because all timestamps in a central event database has to be in UTC.

Regrads,
Juergen

[spalger]

Glad you figured it out! Still trying to wrap my head around the way Kibana does/doesn't use timezones.