EXT_mesh_features: Features and Properties for structured data

[donmccurdy]

EXT_mesh_features defines a means of storing structured metadata associated with geometry and subcomponents of geometry within a glTF 2.0 asset.

In most realtime 3D contexts, performance requirements demand minimizing the number of nodes and meshes in an asset. These requirements compete with interactivity, as applications may wish to merge static objects while still supporting interaction or inspection on those objects. Common performance optimizations for GPU rendering — like merging geometry or GPU instancing to reduce CPU overhead — may destroy references to distinct objects, their attributes, and their behaviors.

By defining a representation of conceptual objects ("features") distinct from rendered geometry, and a means of associating structured metadata ("properties") with those features, this extension allows applications to preserve important details of 3D assets for inspection and interaction without compromising runtime performance and draw calls.

This extension will be used by embedded glTF content in 3D Tiles for loading large, tiled geospatial datasets. The extension's design is intended to be similarly useful for individual glTF 2.0 assets outside of geospatial use cases, like large AEC designs, CAD models, or photogrammetry captures.

Disambiguation: glTF has other methods of storing details that could similarly be described as metadata or properties, including KHR_xmp_json_ld, Extras, and Extensions. While those methods associate data with discrete objects in a glTF asset — nodes, materials, etc. — EXT_mesh_features is uniquely suited for properties of more granular conceptual features in subregions composed of vertices or texels.

Markdown Preview:

• README.md

[donmccurdy]

Some earlier context on the use cases this extension addresses:

• Scene complexity limits #1699

[lilleyse]

Talked with @wallabyway and others today and there were some interesting interesting takeways.

There is a strong need for global feature IDs. In some systems these are 64-bit uints; after all 32-bit uints can only represent up to 4,294,967,295 discrete features and collisions could start to occur for a variety of reasons.

The FEATURE_ID attribute can be used as a global feature ID but since it's a vertex attribute it's constrained by the allowable glTF accessor component types - 32-bit integer component types are not allowed and 64-bit integer component types do not exist. FLOAT can only provide up to 2²⁴ safe integers.

In order to support 64-bit IDs you'd need to store these in a property table with the UINT64 component type. The FEATURE_ID would be an index into the table, so there'd be one level of indirection to get the global ID.

An example of this might be worth including in the extension writeup.

[wallabyway]

Good point on the UINT24 (safe float)...

We render to a target buffer RGBA (using MRT) to get UINT32 featureID (equivalent), but that complicates implementation for others. Hmmm🤔 what to do?

[wallabyway]

RE: The section in the spec on 'Schema-definitions'

https://github.com/CesiumGS/glTF/tree/proposal-EXT_mesh_features/extensions/2.0/Vendor/EXT_mesh_features#schema-definitions

Is it possible to limit this spec to just a featureID and not define how the ID is used?

Currently, the spec has a schema requirement, which is a bit too limiting for our AEC use-cases. Having just an featureID requirement, and not a schema definition, broadens the use for this extension.

Or perhaps I mis-interpreted the "Schema-definitions" section of the spec.
If so, perhaps we could word it differently ?

[javagl]

Currently, the spec has a schema requirement, ...

The schema definition itself is not required in the top-level extension object

Is it possible to limit this spec to just a featureID and not define how the ID is used?

I think that the case that you are referring to is already covered in the "Specifying Feature IDs" section. It says

Every propertyTables index must have an associated featureIds definition, but feature IDs may be defined without a property table. [...] As a result, the length of the featureIds array must be greater than or equal to the length of the propertyTables array.

So it is valid to say

// In primitive:
"extensions": {
  "EXT_mesh_features": {
    "featureIds": [
      {"offset": 0, "repeat": 3 }
    ]
  }
}

to assign Feature IDs to the triangles of a primitive, without specifying property tables, and therefore without assigning any internal meaning to these IDs. These IDs could then be picked up and resolved externally by an application.

(Maybe I overlooked some other requirement - so if there is a reason why this wouldn't be possible based on the specification, then this should indeed be clarified).

[wallabyway]

@javagl - Ah, thank you !

I think this is worth explicitly mentioning "what you said" somewhere in the spec.

ie.
"A minimal example of connecting FeatureIDs stored in triangle vertices, that could refer to IDs in an external database of properties"

Basically "What you said" 'verbatim', along with your json example.

[zeux]

to assign Feature IDs to the triangles of a primitive, without specifying property tables, and therefore without assigning any internal meaning to these IDs. These IDs could then be picked up and resolved externally by an application.

FWIW my understanding of the spec is that this only works for un-indexed geometry.

[lilleyse]

Yeah, if you assign a unique feature ID to each triangle like with {"offset": 0, "repeat": 3 } then each vertex would be unique and it would be redundant to have an indices buffer.

But more often groups of triangles will have the same feature ID and you can still benefit from indexed geometry. Maybe the example should be {"attribute": 0} since that's the more common case I think.

[zeux]

@lilleyse Hmm I'd like to clarify this to make sure we're on the same page.

My understanding is that using offset/repeat doesn't actually make all vertices unique by itself. What I meant is that typically in indexed geometry, there's no obvious mapping between the vertices and triangles (well, short of the index buffer). When you specify offset=0 repeat=3, my understanding of the spec is that you're taking the existing vertices - whatever the number and order - and assigning feature ids to vertices, the same one to each 3 consecutive ones. If the geometry is not indexed, this in effect will result in all 3 vertices of each triangle having a unique feature id. If the geometry is indexed, you're going to get the same mesh with the same indexing, but the feature ids are not going to map to triangles.

As a positive example, imagine a tool that batches N instances of the same mesh together by transforming each instance, generating a new vertex buffer, concatenating all vertex buffers and index buffers together. E.g. gltfpack does that on request. In this case, if each mesh had V vertices and T triangles, you can specify {"offset": 0, "repeat": V}, and you'll get each copy of the mesh in the resulting merged vertex buffer to have its own feature id, essentially resulting in instance ids being specified as part of each vertex. This would only work if you are merging the same mesh (V is the same for each mesh), but this doesn't depend on T or indexing.

[lilleyse]

@zeux yes exactly, everything you wrote is my understanding too.

[ptrgags]

Just wanted to cross-link a couple relevant issues/PRs from our end for better visibility.

[ptrgags]

Note that we've had some back-and-forth about the best way to describe the different types of feature IDs in JSON schema, see CesiumGS/3d-tiles#508.

The issue is that this very flexible definition of implicit feature IDs causes problems when used in this oneOf clause, as a valid featureIdTexture is valid here as well.

[javagl]

The latest update that has been integrated here separates the definition of feature IDs for elements of a glTF asset from the actual storage and structure of metadata.

This extension now focusses on the definition of feature IDs for vertices and texels in a glTF asset.

The definition of the structure and storage of metadata is covered with the EXT_structural_metadata extension that is proposed in #2151

[emackey]

This extension is already in use by published software, right? Is this ready to be merged?

[javagl]

^{(Disclosure: I'm an independent Khronos contributor, but have been involved in the development of this extension as part of contracted (freelancer) work for Cesium. I do not speak on behalf of Khronos or Cesium here, but just try to summarize the state)}

Support for reading the extension is implemented in CesiumJS. Basic structures for the extension are part of cesium-native. An internal implementation of the extension for glTF-Transform is part of the 3d-tiles-tools. There is also an "experimental" implementation in JglTF. The actual specification seems to be "settled", insofar that there have not been significant changes for quite a while now.

I think that further reviews could be worthwhile. But as for all extension PRs, there's the question: Who of the WG could allocate the time that is necessary for that...?