[Maps] Use ES mvt #114553

thomasneirynck · 2021-10-11T19:11:31Z

Replace Kibana's mvt generation with Elasticsearch ES-mvt generation

Vector tile drawing of Elasticsearch data is going GA in Kibana Maps

This PR removes the Kibana implementation of mapbox vector tiles (" mvt") (https://docs.mapbox.com/vector-tiles/reference/), in favor of the mvt-implementation in Elasticsearch( cf. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)

Since 7.11, the Kibana implementation of mvt was a beta-feature, and being used for:

vector tile scaling option in document layers
super-fine grid resolution for cluster layers

With this change to using Elasticsearch, both these capabilities are now going GA.

Increase geo-data volume on screen

This change significantly enhances the amount of geo-data that can be shown on screen.

Displaying more individual documents

Each tile contains up to the max_result_window of features. This is a per-index setting https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html. A single view contain will conist of 4-9 tiles.

Displaying clusters at high resolution

The super-fine resolutions displays up to 128x128 cells per tile, creating high-res cluster maps

Other changes

The client-side data-mapping is no longer possible for ES document layers

This switch is always on by default for scaling with vector tiles. This means the domain-range (ie. min & max) of a value is determined by querying the min-max of the underlying data in Elasticsearch, and not by only getting the min-max of the features on screen.

top-term aggregation is disabled

Top-terms agg is not supported by the vector tile search API of Elasticsearch.

too many features outline

This PR introduces a orange dotted outline of the bounding-box of the features int he tile. This is only used if the number of features exceeds the maximum allowed number of features per tile.

e.g.: zoomed out, cannot show all features at this scale level. Users would need to filter or zoom-in further to see all data.

Field formatters are not applied on map labels

When using vector tiles, field formatters of the data view are not applied for labeling on the map.

…o handling undefined _index. MVT now provides _index

nreese · 2021-10-25T13:55:51Z

@elasticmachine merge upstream

elasticmachine · 2021-10-25T13:56:05Z

Pinging @elastic/kibana-gis (Team:Geo)

thomasneirynck

scope exceeds original goals. I'm +1 on this change. thx!

nreese

LGTM, code review and tested in chrome. This is such a huge improvement. Elasticsearch MVTs have such an amazing performance boost.

Anything that I am still unhappy with is tracked in #116150 and can be fixed at a later time.

jsanz

Tested with Chrome and Firefox and I haven't got a single glitch. I will open a separate issue for removing labels in super-fine grid since they don't make any sense with such small squares but other than that, I've tested medium/large datasets with point, lines, and polygon geometry types and they render fast, panning now around is a great experience. Beautiful job everyone!

kibanamachine · 2021-10-25T16:08:52Z

💛 Build succeeded, but was flaky

Test Failures

[job] [logs] Default CI Group #7 / dashboard feature controls dashboard feature controls security no dashboard privileges "before all" hook for "doesn't show dashboard navLink"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`maps`	839	837	-2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`maps`	201	203	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`maps`	2.7MB	2.7MB	+3.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`maps`	42.1KB	41.7KB	-470.0B

Unknown metric groups

API count

id	before	after	diff
`maps`	202	204	+2

References to deprecated APIs

id	before	after	diff
`maps`	1697	1707	+10

History

💚 Build #1217 succeeded 0dc5411
💔 Build #1216 failed 2909005
💔 Build #1215 failed 01ecad3
💔 Build #1209 failed 0e4e68a
💚 Build #1170 succeeded a383be1
💔 Build #1150 failed bb1448b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* tmp * tmp * tmp * tmp * tmp * use es naming * typo * organize files for clarity * plugin for hits * tmp * initial styling * more boilerplate * tmp * temp * add size support * remove junk * tooltip * edits * too many features * rename for clarity * typing * tooltip improvements * icon * callouts * align count handling * typechecks * i18n * tmp * type fixes * linting * convert to ts and disable option * readd test dependencies * typescheck * update yarn lock * fix typecheck * update snapshot * fix snapshot * fix snapshot * fix snapshot * fix snapshot * fix test * fix tests * fix test * add key * fix integration test * move test * use centroid placement * more text fixes * more test fixes * Remove top terms aggregations when switching to super fine resolution (#114667) * [Maps] MVT metrics * remove js file * updateSourceProps * i18n cleanup * mvt labels * remove isPointsOnly from IVectorSource interface * move get_centroid_featues to vector_layer since its no longer used in server * labels * warn users when selecting scaling type that does not support term joins * clean up scaling_form * remove IField.isCountable method * move pluck code from common to dynamic_style_property * move convert_to_geojson to es_geo_grid_source folder * remove getMbFeatureIdPropertyName from IVectorLayer * clean up cleanTooltipStateForLayer * use euiWarningColor for too many features outline * update jest snapshots and eslint fixes * update docs for incomplete data changes * move tooManyFeatures MB layer definition from VectorLayer to TiledVectorLayer, clean up VectorSource interface * remove commented out filter in tooltip_control add api docs for getMbLayerIds and getMbTooltipLayerIds * revert changing getSourceTooltipContent to getSourceTooltipConfigFromGeoJson * replace DEFAULT_MAX_RESULT_WINDOW with loading maxResultWindow as data request * clean up * eslint * remove unused constants from Kibana MVT implemenation and tooManyFeaturesImage * add better should method for tiled_vector_layer.getCustomIconAndTooltipContent jest test * fix tooltips not being displayed for super-fine clusters and grids * fix check in getFeatureId for es_Search_sources only * eslint, remove __kbn_metadata_feature__ filter from mapbox style expects * remove geoFieldType paramter for tile API * remove searchSessionId from MVT url since its no longer used * tslint * vector tile scaling option copy update * fix getTile and getGridTile API integration tests * remove size from _mvt request body, size provided in query * eslint, fix test expect * stablize jest test * track total hits for _mvt request * track total hits take 2 * align vector tile copy * eslint * revert change to EsSearchSource._loadTooltipProperties with regards to handling undefined _index. MVT now provides _index * clean up * only send metric aggregations to mvt/getGridTile endpoint * update snapshot, update getGridTile URLs in tests * update request URL for getGridTile * eslint Co-authored-by: Nathan Reese <reese.nathan@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-10-27T16:43:26Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 114553 or prevent reminders by adding the backport:skip label.

thomasneirynck added 30 commits July 13, 2021 19:43

tmp

1fbc4a7

Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt_poc

6c339e1

tmp

33bce67

tmp

9da4dde

Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt_poc

d052824

tmp

04c8264

Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt_poc

0b923cf

tmp

254595c

Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt_poc

d2d5448

use es naming

4fdf57d

typo

0ec6b77

organize files for clarity

f8d202c

plugin for hits

a4d969e

tmp

6762ec7

initial styling

ad1b192

more boilerplate

f8c9e67

tmp

fb69717

temp

d20bd79

add size support

c831c38

Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt_poc

c9e0fb7

remove junk

832529d

tooltip

453b32f

edits

1396751

too many features

d16fbf0

rename for clarity

94c88a0

typing

dea461f

tooltip improvements

72936ae

icon

bd4ad7d

callouts

a765eb7

align count handling

dfe86d5

nreese removed the WIP Work in progress label Oct 22, 2021

nreese added 6 commits October 21, 2021 20:03

revert change to EsSearchSource._loadTooltipProperties with regards t…

a383be1

…o handling undefined _index. MVT now provides _index

clean up

e003e97

only send metric aggregations to mvt/getGridTile endpoint

0e4e68a

update snapshot, update getGridTile URLs in tests

01ecad3

update request URL for getGridTile

2909005

eslint

0dc5411

Merge branch 'master' into feature/maps/use_es_mvt

ddb5d29

nreese marked this pull request as ready for review October 25, 2021 13:56

nreese requested a review from a team as a code owner October 25, 2021 13:56

thomasneirynck commented Oct 25, 2021

View reviewed changes

nreese mentioned this pull request Oct 25, 2021

[maps] clean-up ES mvt migration tech debt #116150

Closed

5 tasks

nreese approved these changes Oct 25, 2021

View reviewed changes

jsanz approved these changes Oct 25, 2021

View reviewed changes

jsanz mentioned this pull request Oct 25, 2021

[Maps] Disable labels for super fine aggregation #116159

Closed

nreese merged commit 33fd1bd into master Oct 25, 2021

thomasneirynck mentioned this pull request Oct 26, 2021

[Maps] Use elasticsearch _mvt implementation #109954

Closed

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 27, 2021

thomasneirynck added the backport:skip This commit does not require backporting label Oct 27, 2021

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 27, 2021

nreese mentioned this pull request Oct 29, 2021

[maps] Unable to pluck style meta from local data for geo_grid vector tiles for any property other then count #116782

Closed

This was referenced Oct 29, 2021

[Maps] Display millions of points on a map #75104

Closed

[Maps] Show incomplete data bounds #66605

Closed

nreese mentioned this pull request May 6, 2022

Get mb field name #131766

Closed

spalger deleted the feature/maps/use_es_mvt branch May 8, 2022 22:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] Use ES mvt #114553

[Maps] Use ES mvt #114553

thomasneirynck commented Oct 11, 2021 •

edited

Loading

nreese commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

thomasneirynck left a comment

nreese left a comment

jsanz left a comment

kibanamachine commented Oct 25, 2021

API count

References to deprecated APIs

kibanamachine commented Oct 27, 2021

[Maps] Use ES mvt #114553

[Maps] Use ES mvt #114553

Conversation

thomasneirynck commented Oct 11, 2021 • edited Loading

nreese commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

thomasneirynck left a comment

Choose a reason for hiding this comment

nreese left a comment

Choose a reason for hiding this comment

jsanz left a comment

Choose a reason for hiding this comment

kibanamachine commented Oct 25, 2021

💛 Build succeeded, but was flaky

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

References to deprecated APIs

History

kibanamachine commented Oct 27, 2021

thomasneirynck commented Oct 11, 2021 •

edited

Loading