[x3d-public] Minutes of X3D WG call for Jan 18 2019

[vmarchetti at kshell.com]

X3D WG Call held Jan 18 2019 11 am - 12 noon EST.

Present: Vince Marchetti, Michalis Kamburelis

Read-only Google Drive Agenda & minutes document at 
https://docs.google.com/document/d/1nEyMTbyLH18SwyFPNgJWa46ECnxbe3FQx5mCF8Uvf34/edit?usp=sharing



Agenda 
1. The glTF Mesh, and what it can do for X3D

Short Summary of  discussion  as prepared by VM (corrections welcomed!)

1. Identified two primary motivations for supporting glTF mesh (and other glTF features) in X3D:
1.a There are good exporters available, particularly in Blender, for exporting content in glTF and glb. This means more useful 3D content is available in these formats. This is particularly true for animated 3D content.

1.b Using glTF offers the possibility of greater performance, if the buffer data in the glTF content is directly loaded into the GPU. However, some implementations for supporting glTF in X3D convert the buffer data into X3D - based structures (creating spearate lists for coordinates, colors, and textures) and so may even make using glTF data less efficient.

2. There are  two complementary approaches for supporting glTF meshes in X3D
2.a Load glTF mesh data into the currently defined IndexedFaceSet (and/or the IndexedTriangleSet nodes). It is possible to do a 100% lossless conversion from glTF meshes, including texture, normal, and per-vertex color data, into an X3D IndexedFaceSet node. This is the approach used in the CastleEngine support of glTF and glb.

2.b Introduce new X3D nodes specifically for "binary meshes", with explicit node fields which support indexing into binary data buffers. These nodes could be efficiently loaded from glTF , but can also support other binary mesh formats. This approach is used in X3DOM's support for gltf (see https://doc.x3dom.org/author/Geometry3D/ExternalGeometry.html ). See https://github.com/michaliskambi/x3d-tests/wiki/Binary-meshes, which includes A. Plesch's ideas on how to do this.


---------- End of Summary, what follows is stream of consciousness flow of the call ----------

See M Kamburelis notes at http://web3d.org/pipermail/x3d-public_web3d.org/2019-January/009815.html

You can convert glTF mesh to IndexedFaceSet in X3D, and get 100%
correct rendering, as far as I know. In fact, that's what Castle Game
Engine does now.

However, it is not the most efficient solution, because it requires
interpreting and "unpacking" binary buffers from glTF. For example
glTF binary mesh data may be "interleaved", which means that vertex
positions, normals, texture coordinates are all "mingled" in one
binary blob. Here's a pseudo-code how a mesh in glTF may look like in
memory:

(The fact that this pseudo-code looks like Pascal and uses Castle Game
Engine types is a pure coincidence! :) ) :

""
TPerVertexData = record
 Position: TVector3;
 Normal: TVector3;
 TexCoord: TVector2;
end;
TBinaryMesh = array [1...1000] of TPerVertexData;
"""

To convert this to an IndexedFaceSet, X3D browser must read it and
convert into 3 separate lists:

"""
// place this in Coordinate node, and place the Coordinate node in
IndexedFaceSet.coord
TVertexPositions = array [1...1000] of TVector3;

// place this in Normal node, and place the Normal node in IndexedFaceSet.normal
TVertexNormals = array [1...1000] of TVector3;

// place this in TextureCoordinate node, and place the
TextureCoordinate node in IndexedFaceSet.texCoord
TVertexTextureCoordinates = array [1...1000] of TVector2;
"""

This conversion is perfectly possible... but it's a waste of time.
More efficient player could just read TBinaryMesh from file, and pass
this binary data straight to GPU (e.g. to OpenGL API). OpenGL has an
API to say that "this binary blob contains positions and normals and
tex coords mixed together". This way rendering can be more efficient.
And (if you convert glTF -> X3D) this way conversion from glTF to X3D
is more efficient.

For this reason, a node like BufferView (from X3DOM) could be
introduced, as a new geometry node (to be used instead of
"IndexedFaceSet" or other existing geometry nodes in X3D).

My thoughts on this subject, along with excellent description from
Andreas Plesch about how the "BufferView" looks in X3DOM, are on
https://github.com/michaliskambi/x3d-tests/wiki/Binary-meshes .

Terminology: glTF has buffers, mesh is stored in buffers
In glTF, Node is like a Transformation.

In glTF:
	A glTF can be converted into a IndexedTriangleSet
	
	For conversion from IndexedTriangleSet into glTF:
		IndexedTriangleSet can have a Color node.
		glTF allows placing supplementary per vertex information in the buffer, including normal direction, RGB color, and RGBA color. Texture coordinates are also supported.
Conclusion: glTF with normal, position, color, texture coordinates can be converted without loss of information into an IndexedFaceSet

glTF can be loaded efficiently


1.1 Review status of current implementations in CastleEngine, X3DOM -- any others?
Efficient load of glTF: Load binary stream into GPU
CastleEngine does not yet do this, at this stage CE the unpacks the glTF stream into separate data for position, color, …
X3DOM is doing something more efficient through the “BufferView” node (A Plesch is knowledgeable about this)
See https://github.com/michaliskambi/x3d-tests/wiki/Binary-meshes , """Example how does glTF mesh translates to BufferGeometry in X3D""" .


1.2  Some questions
1.2.1 Is the glTF mesh equivalent to the IndexedFaceSet (or some other X3d node), or does full bilateral conversion require additional X3D nodes and/or additional glTF entities

1.2.2 What is the best way to support glTF meshes in X3D; through extension of existing Inline node, through a new Geometry node, ...(others).

I guess this depends on the use-case. I allow to "Inline" glTF in X3D
in Castle Game Engine, and my long-term goal is to support 100% of
glTF features by converting glTF to an X3D node graph. But I'm fully
aware that this is my use-case, I welcome others view of this :)

An alternative approach is to only import mesh from glTF, like
ExternalGeometry in X3DOM:
https://doc.x3dom.org/author/Geometry3D/ExternalGeometry.html .

In *both* cases, having a node like "BufferView" in X3D would make
this easy, and efficient.

1.2.3 Is there a justification for supporting glTF meshes as a feature by itself, without, or independent of, supporting other glTF features such as the rendering (PBR), animation. Or, does supporting glTF meshes only make sense as part of a broader support of most or all of glTF in X3D?

https://docs.google.com/document/d/1nEyMTbyLH18SwyFPNgJWa46ECnxbe3FQx5mCF8Uvf34/edit



There is an initial evaluation of using glb (in glTF 1) as a container for multiple meshes which would be loaded in as the CADFace geometry for a CAD model in X3DOM. This re
http://www.kshell.com/pages/x3dom_gltf_demo/


Important reasons for X3D to support glTF meshes.
glTF popularity, many available authoring tools and exporters. Example, Blender already has quality exporters (to glTF) for animations created in Blender
Efficiency of loading (not all implementation yet take advantage of efficiency)

Is there any use case for mixing (mashing) glTF with X3D , example import glTF mesh for geometry and apply an X3D defined Texture

glTF does not support cube maps or 3D Textures , so this might be an important use case for mixing X3D and glTF


Additional Comments, suggestions welcomed for the call

Vince Marchetti




Contribution from Michalis
------
There are strong cases for both formats:

- Both open standards, lots of software to view/convert etc.

- glTF is more efficient to deliver to GPU.

  X3D (non-binary encodings) have to be parsed, which means your bottleneck is StrToFloat().

  X3D binary encoding isn't very popular (not so many software to view/convert) and it's not perfect to deliver to GPU anyway. X3D binary removes the parsing problems (all those StrToFloat), but it doesn't add many ways to express a mesh data (per-vertex values, with interleaving etc.). (? Probably? Not knowing X3D binary, but probably no interleaving?)

- glTF is more modern (PBR).

- glTF seems to have more exporter support.

  Blender:
  - glTF 2.0 exporter is developed by Khronos,
  - there is a version for => 2.79 and for >= 2.80,
  - it's even included in Blender by default,
  - it supports animations, normalmaps.

  In contrast:
  - X3D exporter is only for => 2.79, until someone fixes it,
  - it was included in Blender by default, but not in 2.80,
  - it doesn't support animations or normalmaps (CGE exporter adds normalmaps, and can export to own animation format).

- glTF doesn't offer any interaction.

  No sensors (touch sensor, proximity sensor),
  no scripting.

- glTF doesn't offer some graphic features.

  No cubemap textures,
  no 3d textures,
  no texture coordinate generation modes,
  less ways to mix textures (although MultiTexturing in X3D comes with a large bad of problems, but they can be overcome by fixing various spec points, see CGE docs).

  No lights (image-based on not),
  at least in standard.
  There is extension for "punctual" (i.e. not image-based) lights:
  https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Khronos/KHR_lights_punctual
  and (less stable, cause only EXT) image-based lights:
  https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Vendor/EXT_lights_image_based

  No fog.
  No background.

- glTF doesn't offer some other features, I mention below some important ones:

  No easy Text writing
  (leaving conversion Text->mesh to 3D exporters is *not* enough, it doesn't allow to dynamically modify the text during game.)

  No sound.

  No NURBS.

  No shaders. (although X3D shaders need extending, to be composable -- see CGE extension.)

  No composition of many models using Inline?

  Although (in defense of glTF) most X3D exporters don't use these features either.
  E.g. Blender text is exported as a mesh to X3D,
  Blender sound objects are not exported to X3D,
  Blender NURBS are exported as a mesh to X3D.
  Merely having these concepts in X3D standard is not a magic wand to make them used.

  But at least APIs exist to use NURBS, sound etc.
  E.g. CGE Spine conversion converts Spine paths -> X3D NURBS.

- glTF doesn't offer scene graph?
  This is important as an engine API to build and modify what is rendered
  using X3D nodes.
  Although one can argue that applying glTF concepts in a straightforward way
  also allows you to define a reasonable API to operate on a scene.


--------------

Indexing problem:
glTF2 supports names in nodes, so it may be possible to implement referring to meshes by fragment URLs

Idea:
(The CAD assembly issue): Convert the many meshes into a single large mesh , and use Texture coordinates to differentiate faces. (in X3D, related to the hitTexCoord_change output event in TouchSensor node.