Use a single index with wildcard in Elasticsearch reader

[pavolloffay]

Resolves #1361

Use a wildcard (jaeger-span-*) in queries instead of the concrete list of indices (jaeger-span-2018-12-24, jaeger-span-2018-12-25...).

The motivation for this change:

• simplified index management and usage - no --es.max-span-age parameter
• search screen and trace detail screen will show the same results.
• align with rollover (a single read index) and later deprecate rollover lookback option (similar to --es-max-span-age) https://medium.com/jaegertracing/using-elasticsearch-rollover-to-manage-indices-8b3d0c77915d

Possible negative impacts:

• Performance - although kibana also uses wildcard pattern therefore time range queries (on span timestamp) should be well optimized.

Docker images to test:

pavolloffay/jaeger-query:es-wildcard
pavolloffay/jaeger-collector:es-wildcard

• GetTrace, GetServices and GetOperatios used --es.max-span-age to compose a list of historical indices for the query - now query all indices via the wildcard
• FidTraceIDs and FindTraces use times from query parameters to compose the list of historical indices - now query all indices but they use timestamp range query.

[codecov]

Codecov Report

Merging #1969 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1969      +/-   ##
==========================================
- Coverage   96.99%   96.99%   -0.01%     
==========================================
  Files         203      203              
  Lines       10061    10034      -27     
==========================================
- Hits         9759     9732      -27     
  Misses        264      264              
  Partials       38       38

Impacted Files	Coverage Δ
plugin/storage/es/factory.go	100% <ø> (ø)	⬆️
plugin/storage/es/spanstore/reader.go	100% <100%> (ø)	⬆️
plugin/storage/es/dependencystore/storage.go	82.6% <100%> (-3.11%)	⬇️
plugin/storage/es/options.go	100% <100%> (ø)	⬆️
plugin/storage/es/spanstore/service_operation.go	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1bc28d...8797957. Read the comment docs.

[pavolloffay]

@jaegertracing/elasticsearch we are looking for volunteers to test this functionality especially if the query performance changed. Any feedback is appreciated. The docker images are listed in the PR description.

[annanay25]

Should we also change the esCleaner.py script as part of this PR and allow users to specify a max TTL for span data?

[annanay25]

This seems confusing. Can we add the message Now all indices are searched in the parenthesis?

[pavolloffay]

Should we also change the esCleaner.py script as part of this PR and allow users to specify a max TTL for span data?

I am not sure if I follow you. The purpose of esCleaner.py script is to clean old data. Elasticsearch does not allow to specify TTL in any form.

[annanay25]

@pavolloffay Sorry I didn't frame that properly.

Elasticsearch does not allow to specify TTL in any form.

Yes. What I meant was that should we modify the esCleaner.py script as part of this PR as well, to include the change made in index naming format, so that users can continue to specify how long they want to store data.

[pavolloffay]

There is no change in index name format in this PR. The writer still writes to daily indices.

[pavolloffay]

@jkandasa run performance tests to measure query time with this patch. The results prove increased query time by on average 300-400%. Therefore I am closing this PR.

https://docs.google.com/spreadsheets/d/1rwTzDvnHHhssNdZibhFuHBZDf6njt6b1DrvRiJrkt14/edit?usp=sharing