[x3d-public] X3D meeting minutes 1 February 2018: strategy formapping glTF 2.0 to X3Dv4 (Brutzman, Donald (Don) (CIV))

[John Carlson]

Use cases: glTF examples
Requirement: pass glTF standards process to be called “glTF.”

John

Sent from Mail for Windows 10

From: Andreas Plesch
Sent: Friday, February 1, 2019 5:30 PM
To: X3D Graphics public mailing list
Subject: Re: [x3d-public] X3D meeting minutes 1 February 2018: strategy formapping glTF 2.0 to X3Dv4 (Brutzman, Donald (Don) (CIV))

Thanks for the great summary of an accurate discussion around glTF 2.0.

The eye candy is all glTF 2.0 meshes and PBR materials. They are all available from the Khronos glTF sample models repo, and were converted from Collada. The Duck looks like the Nurbs duck but the others are likely authored in one of the main authoring systems.

In addition, the X3D box is regular X3D, to show how both shapes coexist in the same light (head light, lighting PBR material and X3D material).

The x3dom inline glTF support is focused on the black box model. But as Michalis points out there is not that much difference between the black box and the transcoded model since the black box often ultimately relies on transcoding anyways,

In x3dom, PBRMaterial is already available as an exposed node and could work with regular X3D geometries. But this is not tested. ( I should try it. )

Similarly, BufferGeometry (replacing ExternalGeometry) is already available as an exposed node for binary geometry and could work with regular X3D appearance material. But this is also not tested. ( I should try it. )

ExternalShape is not updated to glTF 2, and likely will not.

glTF animations are just regular TimeSensor, Interpolator, ROUTE combos. This works well. As earlier explained (have to dig up the list messages) the Interpolator nodes are extended to allow binary key and keyValue values, to be able to directly use glTF binary data. This is done via BufferAccessor nodes, renamed from BufferGeometryAccessor because they are now used for interpolators as well, not just Geometry (and could be used for other binary data). Also, for easier implementation, 'linear', 'cubicspline', and 'step' modes were added to Interpolators in a field.

glTF animations are not formally exposed but can be accessed via the DEF names of the involved nodes (which have to be discovered). So there is the issue of how to define control over embedded animations, perhaps similar to video media control.

glTF PBR materials should not have a huge performance impact these days over regular X3D materials. It is all in the shader and texture (various maps) use which is very efficient. It would be a component in a profile, no ?

The only item to add to the lighting discussion is that it is unclear if a environment light with PBR should be combined with other light sources since the PBR environment (cubemap) light implementations typically assume that it is the only light to consider for energy balance as I understand it. x3dom currently does not allow this combination but AFAIK most other engines do not care and just add up contributions of all light sources. Maybe that is a question to ask in the three.js forum.

Also, the difference in rendering of the same glTF model in different engines is (mostly) due to use of different lighting. The idea is that the glTF spec. is tight enough to get very similar renderings under the same conditions. But there are enough differences in the implementations of PBR to get slight variations, and pixel by pixel equivalence is not a goal, I believe. For example, for environment lights, aka image based lights, a BRDF lookup map is typically involved to determine contributions of viewing angle and roughness to specular color.The map differs slightly between shaders. Three.js does not use such a lookup map but a functional form that it believes is better suited. But all other engines use a map. It is all in the open.

I believe the exploding glTF scene could also work in X-ITE. It would be interesting to find out and what adjustments may be necessary.

We should be able to converge on a naming convention for a TimeSensor derived from a glTF. The x3dom names changed a bit, I am curious what the castle names are.

oops that became long, thanks for reading,

Cheers,

-Andreas


> Date: Fri, 1 Feb 2019 17:55:08 +0000
> From: "Brutzman, Donald (Don) (CIV)" <brutzman at nps.edu>
> To: X3D Graphics public mailing list <x3d-public at web3d.org>
> Subject: [x3d-public] X3D meeting minutes 1 February 2018: strategy
>         for mapping glTF 2.0 to X3Dv4
> Message-ID: <30083b0e-7156-296c-ea2d-b5296f1ae702 at nps.edu>
> Content-Type: text/plain; charset="utf-8"
>
> Attendees:  John Carlson, Anita Havele, Michalis Kamburelis, Vince Marchetti, Nicholas Polys, Dick Puk, Don Brutzman.
>
> Major progress today, building on recent meetings.  Group approach is really productive.
>
> ----------------------------------------------------------------------
>
> 1. Wakeup eye candy and immediate deep dive: interesting discussion on
>
> On 2/1/2019 6:13 AM, Andreas Plesch wrote:
> > The exploding gltf's scene below works also pretty well on an older mid range cell phone which I just tested. -Andreas
> >
> > On Wed, Jan 30, 2019 at 12:08 PM Andreas Plesch <andreasplesch at gmail.com <mailto:andreasplesch at gmail.com>> wrote:
> >
> >     Since we are getting closer to x3dom merging in gltf2 inline support in its dev. version, here a first example of multiple gltf inlines, using an older x3dom scene:
> >
> >     https://rawcdn.githack.com/andreasplesch/x3dom/b34506419c8877884501143b122cbcbd287abb43/test/regression-suite/test/cases/routes/routes.html
>
> We think these models are essentially glTF meshes (though duck avocado boom-box etc. might have originated in prior lifetimes as nurbs).
>
> Vince notes that there are two ways to integrating glTF into X3D.  First is to treat glTF as "black box" of hidden data that simply displays all or part of a glTF model, with the content of the node inaccessible to the rest of the scene graph (no ROUTEs or inspection, etc.).  The other approach is to convert/rewrite/transcode the glTF content as native X3D scene-graph content.
>
> References:
>         X3DOM documentation and node index (somewhat out of date)
>         https://doc.x3dom.org/developer
>         https://doc.x3dom.org/author/nodes.html
>
>         ExternalShape and ExternalGeometry (from SRC era)
>         https://doc.x3dom.org/author/Shape/ExternalShape.html
>         https://doc.x3dom.org/author/Geometry3D/ExternalGeometry.html
>
>         BinaryGeometry
>         https://doc.x3dom.org/author/Geometry3D/BinaryGeometry.html
>
> More recently, the mappings are changing, actively merging dev webvr branch into master, tracking on the github mailing list. Many moving parts, no deeper links provided here today.
>
>         https://github.com/x3dom/x3dom
>
> Michalis elaborated that conversions of glTF to X3D can occur internally in the player, X3DOM is using those nodes, and every player will likely perform this conversion internally.  So the key question might be whether we advance those nodes to X3Dv4.
>
> Nicholas observed that some things might be some things to expose, for example transcoding to an X3D PBRMaterial.  There is a run-time namespace issue, and a conversion correspondence issue.  This might be especially important for animation.
>
> Michalis reported that he has accomplished similar types of tradeoffs when importing .spine JSON format into Castle Game Engine.
>
>         http://esotericsoftware.com/spine-json-format
>
> ----------------
> Scroll down to an incredibly informative message from Andreas. 1st example shows static model conversion,
> https://github.com/michaliskambi/x3d-tests/wiki/Binary-meshes
> - for glTF materials: PhysicalMaterial
> - for glTF mesh: BufferGeometry and BufferGeometryAccessor
>
> 2nd example has animations (of Transform), so it's just standard PositionInterpolator, OrientationInterpolator etc., but instead of explicit numbers in key="..." and keyValue="...", it refers to a binary data, using again the BufferGeometryAccessor node inside.
>
> Note that we have TimeSensor with name "clockANI0" inside -- this is how you would run glTF animation from the outside.
> ----------------
>
> John shared gltf JSON Schema (v4) link for the below:
> https://github.com/KhronosGroup/glTF/blob/c02fd64e1fc15985d84cbbfacb536709f7c22e28/specification/2.0/schema/mesh.primitive.schema.json
>
> ----------------
>
> Confirming no surprises geometry constructs... Existing glTF meshes modes:
>
> https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#primitivemode
> Allowed values:
> 0 POINTS
> 1 LINES
> 2 LINE_LOOP
> 3 LINE_STRIP
> 4 TRIANGLES
> 5 TRIANGLE_STRIP
> 6 TRIANGLE_FAN
>
> Continuing: a big issue with PBR material (and indeed meshes) is that a scene author might certainly want to use those advanced materials again.  So this might be a strong motivator for having some kind of path to capture PBR material within an X3D scene graph.  However mandating such a correspondence could have a negative impact on performance.  So we don't want to be heavy-handed about it.
>
> However we do need to provide expressibility of glTF 2 capabilities in X3Dv4, making a variety of import and support capabilities possible.
>
> Dick notes that if there is insufficient definition, then variable results might occur.
>
> Wondering if progress has advanced sufficiently to define what nodes are needed in X3Dv4 to express glTF capabilities: mesh, material, animation, humanoid skeletons.  X3D has much of that already.  The deltas are significant and probably our "short set" of what is needed.
>
> ----------------------------------------------------------------------
>
> 2. Lighting.  Of note is that glTF does not define lighting sources, but it does use lighting equations to define their characteristics.  The glTF lighting equations appear as part of the glTF material definitions.
>
>  From Michalis: There are two extensions to glTF (outside of glTF specification) to define light sources. One of them defines simple point/directional light sources (largely compatible with X3D), one defines environment light source.
>
> Materials in glTF:
> https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#materials
>
> Lights in glTF: Links from
> https://github.com/michaliskambi/x3d-tests/wiki/How-to-add-PBR-to-X3D%3F#what-about-lights
>
> - https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Khronos/KHR_lights_punctual
> . It defines point, directional and spot lights, largely compatible with X3D lights but has a different recommended attenuation law, and required non-rendering/culling beyond the max. distance. The extension is supported by the glTF blender exporter. There is no ambient light. (Thanks go to Andreas Plesch for this information, posted mail on x3d-public.)
>
> - https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Vendor/EXT_lights_image_based
> . This defines image-based lights. Yeah, basically some numbers and a cubemap (which we can already express in X3D).
>
> It thus appears possible that we could formally extend X3D Lighting Model to match these equations.
>
> Cross-check: often we see differences in how glTF players render the same content.  Why?  Is that due to different lighting arrangements/locations by different players, or a fundamental problem?
>
> Michalis thinks that differences many of us have observed might easily be explained by different lighting approaches in model portrayal, not in lighting equations per se.
>
> ----------------------------------------------------------------------
>
> 3. Inventory:  what do we need to add to to X3D v4 Specification to fully support glTF capabilities?
>
> Here are links to X3D v3.3 specification, we would start modifying v4 specification in github.
>
> * 11 Rendering component for meshes, define mappings or directly supporting glTF geometry buffers for high performance.  Look at Coordinate node and lines.
>
> * 13 Geometry3D component includes IndexedFaceSet, deserves close inspection/comparison.
>
> * 12 Shape component: likely add PhysicalMaterial (or upgrade Material/TwoSidedMaterial), possibly ExternalShape.
>
> * 17 Lighting component: 17.2.2 Lighting model equations, existing lighting nodes, possibly a prior-compatibility mode with earlier lighting model.
>         (Michalis points out that new lighting nodes are likely not needed)
>
> No immediate need yet seen for modifying animation, since X3D has a rich animation model already.  Perhaps future work on geometry buffers, if glTF starts doing that.
>
> * 26  Humanoid animation (HAnim) component: desirable to continue mapping to HAnim, especially since X3D will soon support HAnim 2.0 and BVH-style motion animation.
>
> No other work noted in the rest of the 41-total X3D components.
>
> All of the above X3D specification work seems do-able and driven by X3D glTF implementation work in progress: X3DOM, X_ITE, Castle Game Engine.  Any others - FreeWrl perhaps?
>
> ----------------------------------------------------------------------
>
> 4. Next steps for X3Dv4, prioritized:
>
> a. X3Dv4 Specification (github)
> b. Examples: Andreas scene, glTF spec examples.
> c. Implementations: iterate until glTF-TO-X3D mappings, X3D specification and examples are stable.
> d. Encoding and validation work will wait until any new nodes are stable.
>
> Open question: is it possible to accomplish for the X3D Working Group to accomplish these tasks by Web3D 2019 and SIGGRAPH?
>
> ----------------------------------------------------------------------
>
> Wow.  So much work (years actually) coming to a focused point that proposes exactly where X3Dv4 is going with respect to glTF.
>
> Summary: looks like we have a fully actionable strategy for mapping glTF 2.0 to X3Dv4.
>
> No doubt more discussion and analysis will continue for us to correctly achieve these worthy goals.
>
> Thanks to many many people getting us to this point!  The work continues...
>
> all the best, Don
> --
> Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman

-- 
Andreas Plesch
Waltham, MA 02453

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20190201/6551f396/attachment-0001.html>