WIP: Change ES tag schema to support tag search in Kibana

[pavolloffay]

Resolves #906

Example tag mapping

tags: {
  "http:method": {
     "value": "GET",
     "type": "string"
  }
}

With this mapping default ES configuration stores ~183 unique tag keys - see also #980 (comment)

[yurishkuro]

what about backwards compatibility with the data people already have stored in ES?

[pavolloffay]

what about backwards compatibility with the data people already have stored in ES?

I haven't been thinking about it yet. We could provide a migration to new model or support query for old model.

[pavolloffay]

Performance results https://github.com/pavolloffay/jaeger-perf-tests, 300k spans different limit parameter for jaeger-query

QUERY_FROM=jaeger-query JAEGER_QUERY_ASYNC=true NUMBER_OF_SPANS=50000 NUM_OF_TRACERS=6 JAEGER_QUERY_LIMIT=50000 mvn clean package exec:java

limit 50000 https://pastebin.com/axmkbmjH
limit 1500 https://pastebin.com/nX0SQQXP

Query time of
http://localhost:16686/api/traces?service=perf-test-thread-0&limit=50000&lookback=1h&tags={"fooo.bar*?%http.d6cconald":"hehuhoh$?ij","fooo.ba2sar":"true","fooo.ba4342r":"1","fooo.bar1":"fobarhax*+??","fooo.bar*?%http.do**2nald":"goobarRAXbaz","fooo.bar*?%http.don(a44ld":"goobarRAXbaz","fooo.ba24r*?%":"hehe"}

limit 50000: mean = 13533.82 milliseconds
limit 1500: mean = 681.32 milliseconds
limit 20: mean = 41.09 milliseconds

Report time with multiple quries
260223.00 milliseconds = 4.3 min

The number of tags does not seem to affect query time

[Monnoroch]

I think #980 is a better solution. Types are just noise, given that ES already infers them. If you have untyped tags this is not true any more, but it is a bad practice and you always have the option of typing them as strings as an ad-hoc fix.

[pavolloffay]

Most important piece of my proposal is that only *specified tags would be stored as direct object. Other tags would still be stored as nested object like it is right now.

• standard OpenTracing tags https://github.com/opentracing/specification/blob/master/semantic_conventions.yaml#L9 and configured tags.