[x3d-public] X3D meeting minutes 1 February 2018: strategy for mapping glTF 2.0 to X3Dv4

Fri Feb 1 09:55:08 PST 2019

Attendees:  John Carlson, Anita Havele, Michalis Kamburelis, Vince Marchetti, Nicholas Polys, Dick Puk, Don Brutzman.

Major progress today, building on recent meetings.  Group approach is really productive.

----------------------------------------------------------------------

1. Wakeup eye candy and immediate deep dive: interesting discussion on

On 2/1/2019 6:13 AM, Andreas Plesch wrote:
> The exploding gltf's scene below works also pretty well on an older mid range cell phone which I just tested. -Andreas
> 
> On Wed, Jan 30, 2019 at 12:08 PM Andreas Plesch <andreasplesch at gmail.com <mailto:andreasplesch at gmail.com>> wrote:
> 
>     Since we are getting closer to x3dom merging in gltf2 inline support in its dev. version, here a first example of multiple gltf inlines, using an older x3dom scene:
> 
>     https://rawcdn.githack.com/andreasplesch/x3dom/b34506419c8877884501143b122cbcbd287abb43/test/regression-suite/test/cases/routes/routes.html

We think these models are essentially glTF meshes (though duck avocado boom-box etc. might have originated in prior lifetimes as nurbs).

Vince notes that there are two ways to integrating glTF into X3D.  First is to treat glTF as "black box" of hidden data that simply displays all or part of a glTF model, with the content of the node inaccessible to the rest of the scene graph (no ROUTEs or inspection, etc.).  The other approach is to convert/rewrite/transcode the glTF content as native X3D scene-graph content.

References:
	X3DOM documentation and node index (somewhat out of date)
	https://doc.x3dom.org/developer
	https://doc.x3dom.org/author/nodes.html

	ExternalShape and ExternalGeometry (from SRC era)
	https://doc.x3dom.org/author/Shape/ExternalShape.html
	https://doc.x3dom.org/author/Geometry3D/ExternalGeometry.html

	BinaryGeometry
	https://doc.x3dom.org/author/Geometry3D/BinaryGeometry.html

More recently, the mappings are changing, actively merging dev webvr branch into master, tracking on the github mailing list. Many moving parts, no deeper links provided here today.

	https://github.com/x3dom/x3dom

Michalis elaborated that conversions of glTF to X3D can occur internally in the player, X3DOM is using those nodes, and every player will likely perform this conversion internally.  So the key question might be whether we advance those nodes to X3Dv4.

Nicholas observed that some things might be some things to expose, for example transcoding to an X3D PBRMaterial.  There is a run-time namespace issue, and a conversion correspondence issue.  This might be especially important for animation.

Michalis reported that he has accomplished similar types of tradeoffs when importing .spine JSON format into Castle Game Engine.

	http://esotericsoftware.com/spine-json-format

----------------
Scroll down to an incredibly informative message from Andreas. 1st example shows static model conversion,
https://github.com/michaliskambi/x3d-tests/wiki/Binary-meshes
- for glTF materials: PhysicalMaterial
- for glTF mesh: BufferGeometry and BufferGeometryAccessor

2nd example has animations (of Transform), so it's just standard PositionInterpolator, OrientationInterpolator etc., but instead of explicit numbers in key="..." and keyValue="...", it refers to a binary data, using again the BufferGeometryAccessor node inside.

Note that we have TimeSensor with name "clockANI0" inside -- this is how you would run glTF animation from the outside.
----------------

John shared gltf JSON Schema (v4) link for the below:
https://github.com/KhronosGroup/glTF/blob/c02fd64e1fc15985d84cbbfacb536709f7c22e28/specification/2.0/schema/mesh.primitive.schema.json

----------------

Confirming no surprises geometry constructs... Existing glTF meshes modes:

https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#primitivemode
Allowed values:
0 POINTS
1 LINES
2 LINE_LOOP
3 LINE_STRIP
4 TRIANGLES
5 TRIANGLE_STRIP
6 TRIANGLE_FAN

Continuing: a big issue with PBR material (and indeed meshes) is that a scene author might certainly want to use those advanced materials again.  So this might be a strong motivator for having some kind of path to capture PBR material within an X3D scene graph.  However mandating such a correspondence could have a negative impact on performance.  So we don't want to be heavy-handed about it.

However we do need to provide expressibility of glTF 2 capabilities in X3Dv4, making a variety of import and support capabilities possible.

Dick notes that if there is insufficient definition, then variable results might occur.

Wondering if progress has advanced sufficiently to define what nodes are needed in X3Dv4 to express glTF capabilities: mesh, material, animation, humanoid skeletons.  X3D has much of that already.  The deltas are significant and probably our "short set" of what is needed.

----------------------------------------------------------------------

2. Lighting.  Of note is that glTF does not define lighting sources, but it does use lighting equations to define their characteristics.  The glTF lighting equations appear as part of the glTF material definitions.

 From Michalis: There are two extensions to glTF (outside of glTF specification) to define light sources. One of them defines simple point/directional light sources (largely compatible with X3D), one defines environment light source.

Materials in glTF:
https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#materials

Lights in glTF: Links from
https://github.com/michaliskambi/x3d-tests/wiki/How-to-add-PBR-to-X3D%3F#what-about-lights

- https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Khronos/KHR_lights_punctual
. It defines point, directional and spot lights, largely compatible with X3D lights but has a different recommended attenuation law, and required non-rendering/culling beyond the max. distance. The extension is supported by the glTF blender exporter. There is no ambient light. (Thanks go to Andreas Plesch for this information, posted mail on x3d-public.)

- https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Vendor/EXT_lights_image_based
. This defines image-based lights. Yeah, basically some numbers and a cubemap (which we can already express in X3D).

It thus appears possible that we could formally extend X3D Lighting Model to match these equations.

Cross-check: often we see differences in how glTF players render the same content.  Why?  Is that due to different lighting arrangements/locations by different players, or a fundamental problem?

Michalis thinks that differences many of us have observed might easily be explained by different lighting approaches in model portrayal, not in lighting equations per se.

----------------------------------------------------------------------

3. Inventory:  what do we need to add to to X3D v4 Specification to fully support glTF capabilities?

Here are links to X3D v3.3 specification, we would start modifying v4 specification in github.

* 11 Rendering component for meshes, define mappings or directly supporting glTF geometry buffers for high performance.  Look at Coordinate node and lines.

* 13 Geometry3D component includes IndexedFaceSet, deserves close inspection/comparison.

* 12 Shape component: likely add PhysicalMaterial (or upgrade Material/TwoSidedMaterial), possibly ExternalShape.

* 17 Lighting component: 17.2.2 Lighting model equations, existing lighting nodes, possibly a prior-compatibility mode with earlier lighting model.
	(Michalis points out that new lighting nodes are likely not needed)

No immediate need yet seen for modifying animation, since X3D has a rich animation model already.  Perhaps future work on geometry buffers, if glTF starts doing that.

* 26  Humanoid animation (HAnim) component: desirable to continue mapping to HAnim, especially since X3D will soon support HAnim 2.0 and BVH-style motion animation.

No other work noted in the rest of the 41-total X3D components.

All of the above X3D specification work seems do-able and driven by X3D glTF implementation work in progress: X3DOM, X_ITE, Castle Game Engine.  Any others - FreeWrl perhaps?

----------------------------------------------------------------------

4. Next steps for X3Dv4, prioritized:

a. X3Dv4 Specification (github)
b. Examples: Andreas scene, glTF spec examples.
c. Implementations: iterate until glTF-TO-X3D mappings, X3D specification and examples are stable.
d. Encoding and validation work will wait until any new nodes are stable.

Open question: is it possible to accomplish for the X3D Working Group to accomplish these tasks by Web3D 2019 and SIGGRAPH?

----------------------------------------------------------------------

Wow.  So much work (years actually) coming to a focused point that proposes exactly where X3Dv4 is going with respect to glTF.

Summary: looks like we have a fully actionable strategy for mapping glTF 2.0 to X3Dv4.

No doubt more discussion and analysis will continue for us to correctly achieve these worthy goals.

Thanks to many many people getting us to this point!  The work continues...

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman