Revise EXT_primitive_voxels

[j9liu]

❗ WORK IN PROGRESS ❗

This revisits the specification for the EXT_primitive_voxels extension. The goal is to achieve same functionality as the original spec, while redefining elements that may make it more intuitive for simpler use cases. (e.g., a Minecraft-style chunk in a box voxel primitive).

I have not finished updating the README, because there's still things in the spec I'm not as confident about, and I don't want to put in the effort to make diagrams for things that will change 😃 I'm listing the biggest changes here, and will draw attention to the ones that are open to feedback.

Primitive Mode

Previously, this extension added three new primitive modes, each indicating some type of voxel geometry.

Now, the extension only adds one new primitive mode, representing a VOXEL constant. The actual geometry of the grid is indicated in the properties of the extension (see below).

Bounds

Previously, the bounds property was used by all three voxel geometry modes to specify a subsection of the shape in its unit shape space.

Now, bounds has been removed. The grid geometry (and their extents) are indicated in the respective properties: box, cylinder, and ellipsoid. This removes the need to shoehorn the grid bounds into the 3-element arrays shared by all shape types.

Shape Definitions

Previously, the unit shapes (in which the voxel grid is actually defined) were in a space of [-1, 1].

For example, consider a box voxel grid. The unit box is between [-1, 1] in all dimensions. The bounds would specify a subsection, say [-0.5, 0.25], within that [-1, 1] space. It was up to the node transform to scale this primitive to the desired world-space size.

In this revision, geometry is now no longer confined to that unit space. The geometry may be specified like so:

Box

box.min and box.max define the corners of the box-shaped voxel grid. These positions are given in the local space of the primitive. So, a box grid may be defined with values like (-1, -2, -3) and (3, 0, 1).

The translation and scale from the unit space box can be inferred from the given min and max. So hopefully this extra transform can be incorporated into existing implementations without too much trouble.

Cylinder

The same three "axes" in the previous version of the spec are still used for this one: radius, height, and angle. height is listed second because it goes along the y-axis of the glTF, so this attempts to mirror the XYZ order of the box.

radius is now a 2-element array specifying the min and max radial extents for the cylindrical grid. This is now given in the local space of the primitive, so the extents may be defined with values like [1, 2]. This can be scaled back down to the unit range [0, 1] by following the max radius. Minimum value = 0.

Similarly, height is now a 2-element array specifying the min and max height extents for the grid. Also now given in the local space of the primitive, so the extents may be defined with values like [-1, 3]. This can be scaled / translated back to the unit range [-1, 1].

Finally, angle is now a 2-element array specifying the min and max angular extents for the grid. Still given in the [-pi, pi] range.

❓ QUESTIONS ❓

Part of me feels like this definition of a cylinder is rather complex. (My simple mind would expect a cylinder to be simply defined by a radius and height.) So could the following suggestions have merit?

• radius and height can be potentially one item arrays. The default min radius could be 0, so the grid fills the entire cylinder volume up to the given max radius. Height could be interpreted as the cylinder ranging from [-height / 2, height / 2]
• angle can be optional
• If a box may be "translatable" via its min / max, should the cylinder have a center property so you can specify where the cylinder originates?

Ellipsoid

Previously, the ellipsoid was defined in rather geospatial terms. Here, it still kind of is, but it'd be great to make it more intuitive for non-geospatial people.

An ellipsoid is defined by the radii, a 3-element array specifying the values along the XYZ axes. The longitude and latitude properties specify the long / lat ranges across the ellipsoid where the voxel grid spans.

I'm still debating how someone would specify a "region" of the ellipsoid based on height. Previously the height range was included as part of the bounds, but as @lilleyse pointed out, this is is rather specific to geospatial so it may not be intuitive to make it a prominent part of the spec. But people may still want to specify subdivisions along the radial axes. So maybe there can be two ways to define this:

• height: a 2-element property representing the height above / below the ellipsoid where the region spans
• ratio: a 2-element property representing the ratio of the ellipsoid that the grid spans. For example, [0.5, 1] represents an ellipsoidal grid with a hollowed out middle.

❓ QUESTIONS ❓

Again, this definition of an ellipsoid is a bit complex. Could longitude, latitude, and ratio / height be optional?

Additionally, if a box may be "translatable" via its min / max, should the ellipsoid have a center property so you can specify where the ellipsoid originates?

And finally, should a region be a formal property in the main body of the extension (designating it as entirely different from the ellipsoid grid)? Or perhaps it should be an optional sub-property of ellipsoid?

[lilleyse]

Based on some of the inconsistencies you pointed out I'm starting to think the bounds approach might be cleaner.

@j9liu what do you think of this?

box
	size (meters) (size: 3)

	bounds (optional):
		min (0 to 1) (size: 3)
		max (0 to 1) (size: 3)

cylinder
	radius (meters)
	height (meters)

	bounds (optional):
		min radius (0 to 1)
		max radius (0 to 1)
		min angle (radians)
		max angle (radians)
		min height (0 to 1)
		max height (0 to 1)

sphere
	radius (meters)

	bounds (optional):
		min radius (0 to 1)
		max radius (0 to 1)
		min angle (radians) (size: 2)
		max angle (radians) (size: 2)

ellipsoid
	radii (meters) (size: 3)

	bounds (optional):
		min radius (0 to 1)
		max radius (0 to 1)
		min angle (radians) (size: 2)
		max angle (radians) (size: 2)

region
	semi-major axis radius (meters)
	semi-minor axis radius (meters)
        height from surface (meters)

	bounds (optional):
                min height (0 to 1)
		max height (0 to 1)
		min angle (radians) (size: 2)
		max angle (radians) (size: 2)

Some things I like about this approach:

• All shape types are centered at the origin, inspired by MSFT_collision_primitives
• Bounds are consistent across shape types, defined as either fractions or angles

I'm treating region as its own shape type since the intersection math would be different and runtimes can heavily optimize if there are two radii instead of three.

Also I don't think we need to support all these shape types up front, I just wanted to mock them out.

One final meta idea I had was to split this extension into two:

• Extension that defines basic shapes and implicit surfaces
• Extension that provides voxel data for these shapes

That would allow people to use the first extension in a raytracer or SDF renderer even if they don't have voxel data.

[javagl]

There are many possible questions, alternatives, or related extension proposals (and maybe I'll write a longer comment with further thoughts here a bit later). Many of these questions revolve about the concept of ~"equivalent representations" (down to the level of https://en.wikipedia.org/wiki/Basis_(linear_algebra) ...).

But one specific question is: What's the purpose of "ellipsoid"? Assuming that the structure of the voxel cells (i.e. the division into lat/lon/height cells) is not affected by the radii, and assuming that the voxels will be affected by the transform of the node that they are attached to, the different radii of the ellipsoid could be represented with a (non-uniform) node scale...

(Yes, the radius itself could be represented with a (uniform) node scale. And ... one could make a case for just omitting all "parameters" (radii of spheres/ellipsoids, radius/height of cylinders, sizeX/Y/Z of boxes), and always use the unit-representation and tweak it into the right shape with node transforms. And this could make sense, because it could simplify the definitions and maybe even the implementations, tremendously: A "box" voxel definition would just be something that defines how to subdivide a certain space (into dimensions cells), and the question of which space this is only depends on the transform of the node. But ... now I started to unfold these "further thoughts", and I'm sure that much of this has already been brought up, and there are technical reasons to not consider this further...)

[j9liu]

And ... one could make a case for just omitting all "parameters" (radii of spheres/ellipsoids, radius/height of cylinders, sizeX/Y/Z of boxes), and always use the unit-representation and tweak it into the right shape with node transforms.

This was the initial idea for the extension (as in #48 ). But in offline discussions with @lilleyse, we thought that it might not be intuitive to typical glTF users or use cases. IMO, the "unit" shapes in that PR (box from [-1, 1], or cylinder with height [-1, 1]) contains a hidden scale of 2 that is also not expected of a "unit" shape.

Plus the other extensions I've seen for defining implicit geometry in some way, have all included some ability to specify shape-relative dimensions anyway. e.g. here and here. (And I think you actually linked these, though maybe under different context 😅 )

Ultimately I would push back and say that, if we can just use node transforms for everything, why doesn't every 3D model exist in a [0, 1] space? It's more intuitive, at least to me, to have a mesh that exists as-is in its own local space, with the dimensions that it wants to have. Then, it can be transformed by nodes as needed. While math-wise this may be the same, conceptually I think this makes more sense. I don't think having a radius or size parameter for the sphere or box hurts the extension in any way, especially if we go with @lilleyse 's idea of splitting this into two extensions (which I like, and am currently prototyping):

One final meta idea I had was to split this extension into two:

Extension that defines basic shapes and implicit surfaces
Extension that provides voxel data for these shapes

[j9liu]

@lilleyse based on your last post, I mocked up what a separate EXT_implicit_geometry extension might look like here. (still WIP)
https://github.com/CesiumGS/glTF/tree/ext-primitive-voxels-revisions/extensions/2.0/Vendor/EXT_implicit_geometry

I had some thoughts:

• The unit box / cylinder is currently defined to be [-1, 1]. But this has an implicit scale of 2, as opposed to 1, which I would anticipate for a "unit" shape. Could it make sense to say the unit box is from [-0.5, 0.5] instead? And similar for the unit cylinder's height? In that case, the box size and cylinder height would actually indicate size and height.
• I think the bounds verbiage is a little confusing still... would it be okay to rename it to slice? So you can optionally specify a slice of the box, a slice of the cylinder, etc. For me, "bounds" indicates something that is outer (relative to the shape) rather than inner, like a bounding volume.

[javagl]

(I would have brought up that "unit cube" in my world means (0,0,0)-(1,1,1), but I couldn't decide whether a "unit cylinder" should have radius 1.0 or 0.5, even though a "unit sphere" definitely has radius 1.0, ... which is odd, because a "unit cube" could just be a "unit sphere with an ∞-norm", ... 🤪 ... I'll skip that for now...)

To be clear: I did not want to "propose" the unit-shape-based approach.

I was just wondering about pros and cons of different representations for the different possible application cases of such a 'extension that only defines geometric shapes'. These are roughly: Rendering, physics (rigid bodies, collision detection), and things like voxel bounds.

I thought that, for example:

• any sub-part of a 'box' (as defined by the 'bounds') would be a 'box' again (and therefore, just a scaled unit cube)
• any implementation that will do things like (for example) checking for intersections between a ray and a box((3,4,5)-(7,8,9)) will have to take into account the transform of the node that this box is attached to anyhow, so the (3,4,5)-(7,8,9) could be baked into the transform of that node, and the intersection test would always boil down to ray*nodeTransform^-1-vs-unitCube

But I know nothing about the practical implementation (for voxel, or for stuff like physics). I'm sure that there are drawbacks to some of the approaches that "look nice" when scribbling with pencil+paper, and advantages to the more elaborate ones that explicitly say, for example, "how large a box is". (It's certainly more intuitive, but there may be further advantages as well).

[j9liu]

@javagl sorry if my previous response sounded dismissive. I thought you raised good points, and they were ones I + @lilleyse considered. I meant to respond with the reasons I used to talk myself out of the same concerns

any sub-part of a 'box' (as defined by the 'bounds') would be a 'box' again (and therefore, just a scaled unit cube)

I do agree that the bounds for a box are a bit redundant. I also wonder if the concept of bounds may not be as widely used / needed as the simple "sphere" or "box". I guess this is an issue with trying to generalize some functionality for broader use, that we ultimately need to alter with specific regard to voxels. My comment about renaming to slice intends to make the idea more accessible, but perhaps even still it is misplaced. Still... maybe someone would want to do SDF modeling with this extension, and would want that ability to define subsections of the implicit primitives?

Worst comes to worst the bounds / slice property could be moved to the EXT_primitive_voxels extension instead. So EXT_implicit_geometry just defines the shape and its dimensions, and only the EXT_primitive_voxels has to deal with the concept of those "sub-volumes".

any implementation that will do things like (for example) checking for intersections between a ray and a box((3,4,5)-(7,8,9)) will have to take into account the transform of the node that this box is attached to anyhow, so the (3,4,5)-(7,8,9) could be baked into the transform of that node, and the intersection test would always boil down to ray*nodeTransform^-1-vs-unitCube

Yeah I get what you're saying. Though, not to be stubborn about it, but I also think a client could just account for the extra size properties in the math. (e.g., convert box.size to another scale transform). I guess for me it seems like a nice bit of modifiability that I don't perceive as that expensive, or unable to be worked around.

[lilleyse]

The unit box / cylinder is currently defined to be [-1, 1]. But this has an implicit scale of 2, as opposed to 1, which I would anticipate for a "unit" shape. Could it make sense to say the unit box is from [-0.5, 0.5] instead? And similar for the unit cylinder's height? In that case, the box size and cylinder height would actually indicate size and height.

That's clearer to me. It would also simplify the descriptions in the spec. The spec wouldn't need to define what the unit shape is. It would just say the box has this size and the cylinder has this radius and height.

Do you know what conventions other glTF extensions are using?

I think the bounds verbiage is a little confusing still... would it be okay to rename it to slice? So you can optionally specify a slice of the box, a slice of the cylinder, etc. For me, "bounds" indicates something that is outer (relative to the shape) rather than inner, like a bounding volume.

I agree, slice is clearer.

Any sub-part of a 'box' (as defined by the 'bounds') would be a 'box' again (and therefore, just a scaled unit cube)

Yeah good point @javagl. It's probably clearly to omit the bounds property for box, even if there's less consistency among shape types.

I do agree that the bounds for a box are a bit redundant. I also wonder if the concept of bounds may not be as widely used / needed as the simple "sphere" or "box". I guess this is an issue with trying to generalize some functionality for broader use, that we ultimately need to alter with specific regard to voxels. My comment about renaming to slice intends to make the idea more accessible, but perhaps even still it is misplaced. Still... maybe someone would want to do SDF modeling with this extension, and would want that ability to define subsections of the implicit primitives?

Worst comes to worst the bounds / slice property could be moved to the EXT_primitive_voxels extension instead. So EXT_implicit_geometry just defines the shape and its dimensions, and only the EXT_primitive_voxels has to deal with the concept of those "sub-volumes".

Maybe there should be distinct shape types for sub-volumes? That way the extension provides a super simple definition for boxes, cylinders, ellipsoids, and more complex variants for cylinder sub-volume, ellipsoid sub-volume.

[j9liu]

I started fleshing out more of the README + spec for EXT_implicit_geometry. I haven't included slice in the README, yet, because it's still up in the air.

Maybe there should be distinct shape types for sub-volumes? That way the extension provides a super simple definition for boxes, cylinders, ellipsoids, and more complex variants for cylinder sub-volume, ellipsoid sub-volume.

I'm open to this. As it currently stands, this is how the slice might look like:

"EXT_implicit_geometry": {
   "cylinder": {
      "radius": 2,
      "height": 2,
      "slice": {
         "minRadius": 0.25,
         "maxRadius": 0.75,
         "minAngle": 0, // half of the cylinder
         "maxAngle": pi // could be optional, since its defined against the original full cylinder
      }
   }
}

If we bring this out into a separate cylinderSlice geometry:

"EXT_implicit_geometry": {
   "cylinderSlice": {
      "minRadius": 0.5, // equal to 0.25 * 2
      "maxRadius": 1.5, // equal to 0.75 * 2
      "minAngle": 0, // half of the cylinder
      "maxAngle": pi, // required, non-optional
      "height": 2 // min / max height might not be necessary ?
   }
}

I could go either way. I like the cleanliness of having the base geometry types, and simply adding fancy modifiers to it (like slice) if desired. But making the "slices" a separate shape may reduce the obligation of clients to implement them. I also think using real values instead of [0, 1] ratios may be more intuitive -- for instance, being able to define the min / max radius of the cylinder slice in meters, instead of as a ratio. Though the ellipsoidSlice would still have to define a [0, 1] ratio to avoid implications from uneven min / max radii.

I also had some questions about region:

• First, I assume that heightFromSurface means that the region starts on the ellipsoid surface, and then is extruded upwards. Is that correct?
• semiMajorAxisRadius = X/Z radius, semiMinorAxisRadius = Y radius?
• Would we want to incorporate the slice properties into the base region definition itself? The base region goes around the entire ellipsoid, almost like at atmosphere. By itself it's just another way of defining an ellipsoid. But including angle bounds in long/lat makes it more clear what the difference is supposed to be. Something like:

"region": {
   "semiMajorAxisRadius": 4,
   "semiMinorAxisRadius": 2,
   "heightFromSurface": 1.5,
   "minAngle": [-pi, 0],
   "maxAngle": [0, pi/2]
}

[j9liu]

Also, thinking about how EXT_implicit_geometry would be paired with EXT_primitive_voxels... I wonder if it would be more or less clean if the attributes were included underneath the EXT_primitive_voxels extension, since they are ignored / unused by EXT_implicit_geometry. Example:

Outside	Inside
{ "attributes": { "_TEMPERATURE": 0 }, "extensions": { "EXT_implicit_geometry": { "cylinder": { "radius": 2, "height": 3 } }, "EXT_primitive_voxels": { "dimensions": [8, 8, 8], } } }	{ "extensions": { "EXT_implicit_geometry": { "cylinder": { "radius": 2, "height": 3 }, "EXT_primitive_voxels": { "dimensions": [8, 8, 8], "attributes": { "_TEMPERATURE": 0 }, } } }

The one area where it may be a little tricky / confusing is when you also introduce EXT_structural_metadata... but I still think it could look like this:

{
  "extensions": {
    "EXT_structural_metadata": {
      "schema": {
        "classes": {
          "voxels": {
            "properties": {
              "temperature": {
                // ...
              }
            }
          }
        }
      }
      propertyAttributes: [
        {
          "class": "voxels",
          "properties": {
            "temperature":{
              "attribute": "_TEMPERATURE"
            }
          }
        }
      ]
    }
  },
  "meshes": [
    {
      "primitives": [
        {
          "extensions": {
            "EXT_implicit_geometry": {
              "box":{
                "size": [2, 2, 2]
              }
            },
            "EXT_primitive_voxels": {
              "dimensions": [8, 8, 8],
              "padding": {
                "before": [1, 1, 1],
                "after": [1, 1, 1]
              },
              "attributes": {
                "_TEMPERATURE": 0
              },
            },
            EXT_structural_metadata: {
              propertyAttributes: [0]
            }
          }
        }
      ]
    }
  ]
}

[j9liu]

Synced about the status of these extensions offline.

• EXT_implicit_geometry has been moved from the mesh primitive to the root level glTF. It now contains an array called geometries, which contains geometry instances. The geometry is just a wrapper around box, sphere, etc.
• EXT_primitive_voxels now has a shape property, which is an index into EXT_implicit_geometry.geometries.

TODO:

• Add noData value to EXT_primitive_voxels
• Experiment with meshopt and Draco compression

Please nitpick names and organization as you see fit 😃

[j9liu]

Add noData value to EXT_primitive_voxels

noData seems out of place on the EXT_primitive_voxels extension itself. Voxels can have multiple attributes, each of which could have (or could omit) a different noData value.

It seems like this should really exist on the accessor instead? Similar to min/max on the accessor, but:

        "noData": {
            "type": "array",
            "description": "A sentinel value that indicates there is no or missing data wherever it appears.",
            "items": {
                "type": "number"
            },
            "minItems": 1,
            "maxItems": 16
        },

Otherwise, I guess you could map attribute names to noData values in the EXT_primitive_voxels extension, but that seems messy.

Perhaps we could just create a dependency of EXT_primitive_voxels on EXT_structural_metadata for this purpose?

[javagl]

Perhaps we could just create a dependency of EXT_primitive_voxels on EXT_structural_metadata for this purpose?

I'm concerned about the amount of complexity that this would imply for possible implementors.

[j9liu]

I'm concerned about the amount of complexity that this would imply for possible implementors.

That's a fair point. I was rushing to leave that comment and didn't think through the implications.

This is my picture so far for how EXT_primitive_voxels can be used in 3D Tiles...

in tileset.json:

{
  "asset": {
    "version": "1.1"
  },
  "schema": {
    "classes": {
      "voxel": {
        "properties": {
          "temperature": {
            "type": "SCALAR",
            "componentType": "FLOAT32",
            "noData": 999.9
          },
          "salinity": {
            "type": "SCALAR",
            "componentType": "UINT8",
            "normalized": true,
            "noData": 255
          }
        }
      }
    }
  },
  "root": {
    "boundingVolume": {
      "box": [0, 0, 0, 100, 0, 0, 0, 100, 0, 0, 0, 100],
    },
    "content": {
      "uri": "{level}/{x}/{y}/{z}.gltf",
      "extensions": {
        "3DTILES_content_voxels": {
          "dimensions": [8, 8, 8],
          "class": "voxel"
        }
      }
    },
    "implicitTiling": {
      "subdivisionScheme": "OCTREE",
      "subtreeLevels": 6,
      "availableLevels": 14,
      "subtrees": {
        "uri": "{level}/{x}/{y}/{z}.subtree"
      }
    },
    "transform": [0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 6378137, 0, 0, 1],
    "geometricError": 100.0,
    "refine": "REPLACE",
  }
}

in a given glTF:

{
  "extensions": {
    "EXT_structural_metadata": {
      "schema": {
        "classes": {
          "voxel": {
            "properties": {
              "temperature": {
                // same as tileset definition
              },
              "salinity": {
               // same as tileset definition
              }
            }
          }
        }
      }
      propertyAttributes: [
        {
          "class": "voxels",
          "properties": {
            "temperature": {
              "attribute": "_TEMPERATURE"
            },
            "salinity": {
             "attribute": "_SALINITY"
            }
          }
        }
      ]
    },
    "EXT_implicit_geometry": {
      "box":{
        "size": [2, 2, 2]
      }
    }
  },
  "meshes": [
    {
      "primitives": [
        {
          "extensions": {
            "EXT_primitive_voxels": {
              "shape": 0,
              "dimensions": [8, 8, 8],
              "attributes": {
                "_TEMPERATURE": 0,
                "_SALINITY": 1
              },
            },
            "EXT_structural_metadata": {
              "propertyAttributes": [0]
            }
          }
        }
      ]
    }
  ]
}

For a voxel tileset it would be expected / required / implied that:

• EXT_primitive_voxels.dimensions = 3DTILES_content_voxels.dimensions (and same for padding)
• EXT_structural_metadata contains same class + definitions as tileset schema
• EXT_structural_metadata only references the one property attribute corresponding to the 3DTILES_content_voxels.class
• shape is the same type as the tileset's bounding volume (box, cylinder, ellipsoidal region)

Perhaps shape also needs to match the slice of that tileset bounding volume that it occupies in implicit tiling, but the transform is implicitly applied by the tiling scheme (and therefore not included in the glTF itself?)

[j9liu]

It's been a while, but I've updated the PR with the following:

• Removed EXT_implicit_geometry, in favor of KHR_implicit_shapes as written here: KHR_implicit_shapes extension Draft Proposal KhronosGroup/glTF#2370
• Added EXT_implicit_ellipsoid_region.
• Reduced VOXEL constant from 0x80000000 to 0x7FFFFFFF, the maximum number possible with an int32_t.

Still TODO:

• Find a good place to put noData
• Complete READMEs

[lilleyse]

The latest changes look good, though I think we need a "cylinder region" shape defined by min radius, max radius, min angle, max angle, and height, in order to support tiled cylinders like below (where each red area is a glTF in a larger tileset).

[j9liu]

Whoops, forgot about the cylinders. Will work on that, thanks for pointing it out @lilleyse!