Use ray picking to query extrusions in "queryRenderedFeatures"

[1ec5]

With 3D polygon extrusions (mapbox/mapbox-gl-style-spec#456) on a tilted map, feature eventing will continue to return the same results regardless of any extrusion. That is, even if the query point visually falls upon a building wall, the results will reflect whatever would happen to be rendered at ground level had the building wall been absent (which may be the footprint of a different building). Ideally, querying a point on screen would return the building (due to any of its walls), the building across the street, and so on until reaching the ground.

Typically, interactive 3D environments implement ray picking. Ray picking would probably be more straightforward to implement if GL JS would position each component of the map in a 3D coordinate space. For better or worse, GL JS instead treats the entire map as a flat surface, modulo any transforms. So I don’t know that we’ll be able to take advantage of usual approaches to ray picking.

/cc @lbud @peterqliu @ansis

[samanpwbb]

Just ran into this on a demo map:

[1ec5]

Ray picking would need to be optional, because there are also use cases that call for the current behavior, for example finding the height of a building at a particular geographic coordinate (which would be mapped to a screen coordinate before querying).

[holmesal]

+1 - I'm running into this issue while trying to create hoverable extrusions.

Users generally hover over the extruded body, but queryRenderedFeatures only returns features when the cursor is inside the "footprint" polygon.

This doesn't return the desired feature (pink arrow == mouse position):

This does return the desired feature:

[andrewharvey]

Typically, interactive 3D environments implement ray picking. Ray picking would probably be more straightforward to implement if GL JS would position each component of the map in a 3D coordinate space. For better or worse, GL JS instead treats the entire map as a flat surface, modulo any transforms. So I don’t know that we’ll be able to take advantage of usual approaches to ray picking.

Is this where you render another pass with each object a unique colour, then simply lookup the x/y pixel in that buffer and map back from the color to the object id?

UPDATE: As pointed out by @pbabik this would only return the "topmost hit", Some applications might want the features behind at that point too (eg. if you have transparent buildings, clicking might let you cycle through which one you want to select).

[pbabik]

That's how feature selection works in OpenLayers when Canvas renderer is in use. But as I understand this will allow for "topmost hit" selection only (still better than current functionality)

[cmarfil]

From what I've been able to investigate, mapbox dosn't work if you need to select buildings/extrusions or add hover effects on them... It's a shame..

[billyc]

It's very counterintuitive to click an extrusion and have the building behind it get selected. Or three our four behind it, depending on the pitch! It breaks any semblance of "realness" to the 3D objects displayed on the map.

[korywka]

Hi guys. Please tell, will this be planned for the nearest time? Or this feature is not a priority? Thanks.

[nicooprat]

@Bravecow Looks like it's coming in the next few months: https://www.mapbox.com/mapbox-gl-js/roadmap/

[mushon]

It's been a few months now of this issue pending in the roadmap, any chance we can get an updated ETA? Will this bug be fixed together with the #6022 enhancement?

[andrewharvey]

OSMBuildings appears to implement this with the render to a framebuffer with each building in a unique color then just lookup the pixel color and map back to the feature.

https://github.com/OSMBuildings/OSMBuildings/blob/ea70e1f1708c678fdb8301e55660d1b9d0f26736/src/render/Picking.js

[chloekraw]

Hi all - wanted to reassure folks that this is a priority for us and we have been looking into it; however, the solution is not straightforward and may require several more months of effort. Thank you for your patience.

[wangxiaoyu]

Does the ray picking change come with an on/off switch?

The proposed change was the feature I expect however after read related issues now I have a second thought. Basically the change is to opt for selection on screen visuals instead of selection on a (tilted) map.

By the function name queryRenderedFeatures(0 the change is pretty straightforward. However this change the behavior and could cause unexpected results. Besides how would it interact with fill-extrusion-opacity?

It seems the query result would be an array with the front object as the first element. However if the front feature is visually larger and blocks other features behind it then none of them could be selected, unless the function also return something like "angle-of-view" so that the user could customize selection based on small AOV values. Otherwise in order to select those blocked features the user has to constantly rotate the map to find the right pitch and bearing which would be quite a nuisance, especially in dense areas where small features are surrounded by large features.

kepler.gl has the wanted behavior in its example "New york city population by census tract" and playing with it you can see blocked objects are visible but not selectable. OSM has similar issues for OSM 3D buildings in NYC.

After playing with both I feel the current mapbox implementation is actually ok or even preferred for many use cases such as building selections. In the end the proposed change is a nice enhancement, but not a fix so I wish to have the option to keep the current implementation.

[ansis]

@wangxiaoyu thanks for your questions

queryRenderedFeatures will return all features under the mouse pointer, not just the front one. They are sorted in order of closest to furthest but the developer could implement a way of selecting further ones (first click selects the closest, second goes to the next, click again to cycle through all the features).

Or if you want the old behavior (based on the footprints) you can create a regular fill layer under the fill-extrusions and query that instead of the 3d buildings.

While I understand the challenges this API change adds I don't think we want to add an "angle-of-view" option because of the complexity that would introduce.

Would either of the two workarounds work for you?

[stevage]

Awesome! Really excited to see this in action!

[KravMaguy]

I am just seeing a post I made was referenced here. I am now seeing its an issue thats been discussed for a while, I followed peters instructions on that thread to create this :

fill extrusion hover

As you can see in the gif the tooltip is following the cursor and displaying the extrusion properties correctly, but only on the ground floor. So this is the issue apparently everyone is having.

queryRenderedFeatures will return all features under the mouse pointer, not just the front one. They are sorted in order of closest to furthest but the developer could implement a way of selecting further ones (first click selects the closest, second goes to the next, click again to cycle through all the features).

Yes this is an acceptable solution. I am looking forward to using it, and looking forward to seeing this fixed in ten days. As I was unable to use master branch.