[x3d-public] gltf inspired binary storage

Thu Jan 20 10:50:39 PST 2022

I think interest and implementation of alternative data structures is fine, as long as we recall that we retain the idea that an x3d scene can be composed entirely using a highly abstracted Authoring interface gui and using very simple text editor, and joining the metame using a widely available and free runtime using all available resources of the platform and the me.

So, I am always going to want to guarantee that I can get an entirely human readable view of all controls and data so I can
easily add details to my interactions. All this boils down in Hanim where we expect a complex mesh and detailed textures, many accessories and careful animations. For skeletal animation of the skin, we need to define each vertex of the deformable skin to at least one joint of the skeleton and then specify the ratio of such element’s movement, the weight,  rotation of that joint has upon that specific vertex. In practice the animation of each vertex depends upon several skeletal motions and all that has to be accounted for. 

The ‘standard’ representation of these bindings for hanim is we have a mesh and we use the order of appearance in the user code to number each point 0 to whatever. Next we use the Joint object to list the points that are animated by that joint, as well as the weight, 0 to 1 for how much that vertex should move on its radius distance from the joint center. So the user code has a Joint object that includes the list of vertices that are affected by rotation of that Joint, and the weights. This gives us not only a relatively compact and readable data structure where all me human has to do is make sure I get the entire mesh parented by the Humanoid, then figure how rotation of various skeleton joints will produce desired animation of skin vertex. I might make a table that lists each joint along the mesh index and weight. 

So that gives us the canonical and validateable authortime user code that we can use several analytical styles to predict that it will probably be able to actually run, and now for the runtime. Since long time, we know that every vrml and x3d runtime is going to take that skeleton and skin apart and put it together in the best way for the platform. Since parallel high speed graphics, this means the that at some point in runtime scene generation, when we actually move a vertex, the runtime metal probably cares more about which joints are bound to a particular vertex than about which vertices are bound to a particular joint. The thing figures out how the skeleton has moved then processes each vertex by how its controlling joints have moved. 
Just because the authortime and the runtime data structures appear different, that does not mean that are not the same. Before some recent developments in sharing data, people had to figure out how to transport important harshly learned lessons about character rigging and animation between various development and production tools. For development it works with a gui and hardy ever needs to worry about human-readable user code. Especially for production they don’t need to know details, they care about getting data structures they can run. 
So, of course any x3d authoring tool should be able to import ‘standard’ data structures that make it easier to compose interactive systems like Hanim. Finally, since the data is the same, we should be able to express any of these in the standard canonical form. If I get a skeleton binary and a skin binary and a bindings binary and some animation binary or whatever standard forms recognized by x3d, the x3d authoring system in use should be able produce canonical xml that can be validated and, as well, be able to produce gltf forms if requested. The x3d is open and transparent all the way down and the human-readable code is main feature of the platform. Even in our standard binary forms, the content is converted to canonical form for scene content and structure validation.  Without this an x3d authoring tool is just another tool, nothing special. 

Great for pushing this to the limit.
Joe

From: Andreas Plesch
Sent: Wednesday, January 19, 2022 3:33 PM
To: Michalis Kamburelis
Cc: X3D Graphics public mailing list
Subject: Re: [x3d-public] gltf inspired binary storage

Thanks, Michalis.

Yes, this tracks very closely the glTF specification. But there are probably tweaks that can be introduced to better integrate with x3d. 

I would expect some discussion around the current description. A first order question is that it may be more appropriate for the Interpolators to have new nodes such as BufferCoordinateInteprolator rather than an additional 'buffer' field. It was more convenient in x3dom to just add fields but it leads to competition between the key/keyValue fields and the BufferAccessor fields requiring clarifications.

Also, referring to nodes by index into an array of nodes tracks glTF but may be unusual for x3d.

In a way, the wiki description is a bit of initial spec. work but I am not really sure it is the best design.

For example, typically there is only one buffer file. Should therefore Buffer become a node rather than a field so there is a way to DEF/USE it ?

Thanks, Michalis, for your summary wiki. I honestly did not remember that I had put some thoughts into that before. Glad to see that the thinking stayed largely consistent.

Right, MorphTargets are the equivalent. The expectation is that all targets can be uploaded to the GPU and that then the interpolation can take place on the GPU, very fast. But it places a limit on the number targets/frames. x3dom currently does all on the CPU, with javascript. That is quite slow but fast enough for simpler models. I think for real GPU processing it would become necessary to migrate to Three.js, or Babylon.js, and write a x3d interpreter around those which is a massive task.

Best , Andreas

On Wed, Jan 19, 2022 at 3:58 PM Michalis Kamburelis <michalis.kambi at gmail.com> wrote:
Excellent, this is very valuable.

In general, I absolutely agree that in future X3D we should copy the
glTF approach of binary accessors+views. It's a flexible mechanism,
that maps nicely to GPU operations, and makes a huuuge speedup when
reading real-life models. (By "real-life models" I mean here "large 3D
models that are typically exported from 3D authoring software like
Blender", not coded by hand in a  text editor.)

I have a page on my wiki
https://github.com/michaliskambi/x3d-tests/wiki/Binary-meshes to
remind me of that :)

It's great that X3DOM takes a lead on this. Would you be willing to
work on PR to X3D specification to incorporate accessors+views in the
X3D standard? I would gladly help with review and implementing it on
my side (CGE/view3dscene) too. Because if you don't do it, then I'll
have to do this specification work sooner or later :) And from what I
see, X3DOM approach maps nicely to glTF, so it's just reasonable to
move X3DOM accessors+views idea to X3D specification.

P.S. Of note is that glTF actually has one more animation type (on top
of simple translation/rotation/scale that you mention), and this
animation type corresponds to X3D CoordinateInterpolator idea
more-or-less: glTF allows to define a number of "morph targets" (which
are equivalent to "Shape Keys" in Blender) and then to animate their
weights, using channel.path = weights in animation. See
https://www.khronos.org/registry/glTF/specs/2.0/glTF-2.0.html#animations
. This is how in glTF they do something equivalent to
CoordinateInterpolar: define a number of "morph targets" and animate
between them. X3D CoordinateInterpolator is similar to a "morph
targets animation" with particular morph targets already premultiplied
with weights for the current animation.

Regards,
Michalis

śr., 19 sty 2022 o 20:05 Brutzman, Donald (Don) (CIV)
<brutzman at nps.edu> napisał(a):
>
> Tweeted @Web3dConsortium with some nice screenshots.  Bravo!
>
>
>
> Improved #X3DOM binary storage of field values on GPU using (inspired by @glTF3D techniques) can efficiently store exceptionally large arrays such as #X3D CoordinateInterpolator, useful for large keyframe animations.
>
> http://web3d.org/pipermail/x3d-public_web3d.org/2022-January/016533.html
>
>
>
> https://twitter.com/Web3DConsortium/status/1483819271016824835
>
>
>
> all the best, Don
>
> --
>
> Don Brutzman  Naval Postgraduate School, Code USW/Br        brutzman at nps.edu
>
> Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA    +1.831.656.2149
>
> X3D graphics, virtual worlds, Navy robotics https:// faculty.nps.edu/brutzman
>
>
>
> From: x3d-public <x3d-public-bounces at web3d.org> On Behalf Of Andreas Plesch
> Sent: Tuesday, January 18, 2022 7:40 PM
> To: X3D Graphics public mailing list <x3d-public at web3d.org>
> Subject: [x3d-public] gltf inspired binary storage
>
>
>
> Let me announce x3dom support for efficient support of large CoordinateInterpolator (and other) nodes by adopting gltf inspired binary storage of field values.
>
>
>
> Here is an example:
>
>
>
> regular CoordinateInterpolator (ca. 12MB):
>
>
>
> https://61e75f82d2db93000729e6a7--x3dom.netlify.app/examples/cases/coordinateInterpolator/coordinateInterpolator.html
>
>
>
> binary CoordinateInterpolator (ca. 5MB):
>
> https://61e75f82d2db93000729e6a7--x3dom.netlify.app/examples/cases/coordinateinterpolator/coordinateinterpolatorbuffer
>
>
>
> The binary version should load faster. There is no visual difference. There is also no performance difference after initial loading.
>
>
>
> This is useful for large keyframe animations.
>
>
>
> Background:
>
>
>
> glTF stores all data in buffers, eg. binary storage which can often be used directly on the GPU. The details on how the data is stored is provided by 'views' into the buffer and 'accessors' of views.
>
> x3dom implements glTF support not by translating these data structures into x3d field values but by implementing additional x3d nodes and fields. There is a 'BufferGeometry' node and additional fields such as BufferAccessor for Interpolators. This has the advantage that these additions can also be used outside of glTF.
>
> glTF only has support for translation, scale and rotation interpolation. In x3dom, buffer storage can now also be used for other interpolators. Here is a fuller description:
>
>
>
> https://github.com/x3dom/x3dom/wiki/Buffer-interpolators
>
>
>
> It is possible to convert regular Interpolators to buffer Interpolators. Here is a sample converter for CoordinateInterpolator:
>
>
>
> https://observablehq.com/@andreasplesch/x3dom-interpolator-buffer-converter
>
>
>
> In x3dom, there is also a BinaryGeometry node which is a precursor to the BufferGeometry node used for glTF support. Though non-standard, the BinaryGeometry node is very popular since it allows for faster and more efficient loading of large IFS. There is a converter (aopt) which generates BinaryGeometry. It should be possible to write another converter to BufferGeometry. This binary format is quite different from x3db, the Fast Infoset based binary encoding of x3d which requires heavy processing. With glTF support in x3d, there may be now more interest in glTF type binary storage of heavy models, perhaps leading to standardization.
>
>
>
> Any feedback or comment always welcome,
>
>
>
> Andreas
>
> --
>
> Andreas Plesch
> Waltham, MA 02453
>
> _______________________________________________
> x3d-public mailing list
> x3d-public at web3d.org
> http://web3d.org/mailman/listinfo/x3d-public_web3d.org

-- 
Andreas Plesch
Waltham, MA 02453

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20220120/61838a2a/attachment.html>