[x3d-public] glTF - HAnim conversion

Thu Dec 13 10:25:21 PST 2018

inline

From: Andreas Plesch
Sent: Wednesday, December 12, 2018 7:49 PM
To: Joe D Williams
Cc: X3D Graphics public mailing list; Michalis Kamburelis; Humanoid Animation (H-Anim) Working Group
Subject: Re: [x3d-public] glTF - HAnim conversion

Hi Andreas

➢ I agree that using center is more natural and appropriate. glTF compensates for the lack of center by using the InverseBindMatrix. HAnim in return can represent an IBM with an inner Joint.

Maybe you are on to something as a comparison of why to not use translation. What if you put a scale in there? Similar problem

➢ Using translation should not be illegal ?

I agree, but it just won’t work as expected. When you use translation in a stack of transforms, what are you really doing? Where will the joint be with respect to root or a previous joint. I think you will see that while translation will work if you engineer it to work in simple widgets,  using center keeps the skeleton in the humanoid root space. I think you will find that the examples you are using will work using either translation or center, but using center and no translation will position the joint action where you want it to be.
In the example I sent, try changing the center(s) to translation(s) and see what you get.

➢ Perhaps the way skincoordindex is used in the joint hierarchy is unusual.

I don’t understand unusual. The index associates a vertex with a joint. The skin is best when it is in the same space as the skeleton. When you start translating joints, then you are changing the (root) space. Try and see, even using that simple mainly useless except for showing skinning and actually somewhat misleading example.  

➢ The transformation defined by a joint is relative to its parent, which could be the root.

Again, try some stuff. You will see that as soon as you translate a joint you are changing the space away from the root or previous joint. 

➢ The way I have come to think about glTF joints is that there are three steps:
  - the InverseBindMatrix moves vertices to the local origin of the joint
- the joint rotates the vertices
- the joint translates them back

Sorry, I don’t understand. A vertex is not rotated, it is translated according to joint rotation by a formula that may invlude more than one joint. 
We don’t care about gltf joints even if there was such a thing, just the data that might be available and usable in a gltf package. 

➢ Let's see how that works with the RiggedSimple example. 

Fine, experiment some with the thing. Look at the difference when you actually make an x3d model although please look at something that has to do with complex joint structures. 

➢  Ok, it does not really as it turns out that the example has a somewhat bizarre way of defining the skeletal structure. This must be how blender set it up. Node_2_IBM controls the lower, mostly fixed portion and Node_3 the upper, animated portion. Both are children of Node_2, which is effectively the root.

Right, skip the bizarre stuff and get to a reasonable example. I think the I

The root node is the first joint in the structure and can be translated or rotated to move the complete object. Rotating any joint will move stuff parented by it. 

➢ My impression is that the animation works only because the physical joint is already at the origin.
➢ I guess it is time to look for another example.

I agree, why not try a real one instead of just a very minor simple example of a useless wiggling widget. 
Anyway, you could try using center in your example vs translation to position the joint and see what the difference is. The gltf you are using is aimed at some relatively simple cases like you are working with. 

➢ By the way, there was no intent to pollute HAnim. It may just turn out for implementation purposes it is convenient to use more of a glTF type data structure. This is unrelated to authoring.

Intent does not matter, it matters what you are actually getting. This is not unrelated to authoring, Not in the x3d user code. The IBM will work either way, just not correctly as expected in hanim. You would be far from implementing x3d hanim if you use translations like you are thinking. That will lead to madness and groups of humanoids that could not use similar animations and that is a main purpose of hanim – to allow transport of animations between humanoids. Nobody is concerned about convenient implementation purposes. It should work as it always has and if that is not convenient for your implementation, then too bad because it is proven to work in many applications, as you will see as you experience  more. 

Your questions are directly related to authoring and it overlooks an important inherent feature of x3d and any scenegraph and I believe you will see the wisdom sooner or later😊. Regardless of what that simple example shows, you want all joints in the root space. And really, if you ae working with something related to x3d, then maybe you ought to use an x3d hanim example and a compliant x3d player as reference.  

➢ Best, Andreas

Good Luck,
Joe

---on the phone---

On Tue, Dec 11, 2018, 9:30 AM Joseph D Williams <joedwil at earthlink.net wrote:
From: Andreas Plesch
Sent: Monday, December 10, 2018 4:54 PM
To: Joe D Williams
Cc: X3D Graphics public mailing list; Michalis Kamburelis; Humanoid Animation (H-Anim) Working Group
Subject: Re: [x3d-public] glTF - HAnim conversion

Great feedback.

On Mon, Dec 10, 2018, 12:10 PM Joseph D Williams <joedwil at earthlink.net wrote:

➢ X3D xml
Some nice work and analysis and glad to see some success and understanding, however please label your “X3D xml” example as experimental or proposed, because the code is not current x3d hanim and will not work in any that I have found.

• Which browser would you recommend for best conformance ? I know it is working with x3dom  but HAnim was implemented there based on the somewhat limited number of available models.

The best one is BS Contact or Instant, I think. 

• Can you be more specific which construct is most problematic in the experimental xml ?

Well, using translation instead of center will not work that I have seen. 
See, it is different when you try to translate each joint instead of using center where all joints are really in the same space, the humanoid space.
When you use translation for each joint, then where are you translating one joint with resect to another? 
I will try and get this running from the data you have given. 
I think, as you add more joints to produce a realistic skeleton and skin, you can see the need to add some non-hanim nodes in basic x3d to produce simple joint-driven animations like you are showing. It should not be necessary to create the humanoid container in order to use freestyle joint structures, add some skin, and have a nice smooth animation of something. And, of course, Displacer also.
Yes, joint driven animation is universally supported in 3d engines. I think a boundary to HAnim would be somewhat arbitrary, the LOA levels then becoming the main feature.
 I think there is no boundary, either you want to build a humanoid out of joints or build a general purpose free-style animation. For the second case, It would be nice to build it out of objects that do not require the humanoid container. 
• One of the bigger differences is that glTF associates a joint or
multiple joints to each vertex of a mesh while HAnim associates
vertices to joints. If there are many vertices (here 96) it becomes
hard to invert that mapping manually but it is actually pretty
straightforward to do with a few lines of code which are outlined at
the end of the page.

• glTF associates a joint or
multiple joints to each vertex of a mesh 

yes, gltf apparently associates ‘all’ or always at least 16(?) joints (and weights) with each vertex. That means there is feature called a vertex that has a list of joints and weights associated with it. There is one of these for each vertex, all listing all joints and weights.

Yes. For glTF, the minimum a compliant viewer needs to support is 4 joints (out of many) per vertex. A viewer is free to ignore additional joints, with the lowest weights.

Hanim does not have any max number of joints to control a vertex. The code is run as authored with no freedom to drop stuff. I think we have already seen that in the gltf each vertex gets filled with with 16 or so by default, right?  

>while HAnim associates vertices to joints. 

Yes, the each vertex and weight is associated with one or more joints. That means there is a feature called a joint that contains a list of vertices and weights. There is one of these for each joint, each listing only the associated vertices and weights. 

Check the case where you have 96  joints and 2000 vertices.

• Is there a situation where that many weighted joints need to affect a vertex ?

No, except for the humanoid to which the entire skin is bound. I think we can see that max 4 or 5 joints will control a vertex, and in many cases, only one.  

• The reason why glTF requires joints for each vertex is that the joint transforms are best applied in the vertex shader which is executed  for each vertex anyways. Skipping is more complicated than some matrix math. Logic is more expensive than math on the GPU.

Fine, but this has nothing to do with the user code for x3d. Have you looked at the example I sent. Nothing so special except the authoring for the way hanim does it is much simpler to read and check because it tells you exactly what needs to be done without repetition and duplication. Who cares about the logic required to reform the data to this one particular gltf form. When you take in the user code and put it in whatever form your runtime needs, then it is done. 

• If there are many vertices (here 96) it becomes
hard to invert that mapping manually but it is actually pretty
straightforward to do with a few lines of code which are outlined at
the end of the page.

Check the case where you have 96  joints and 2000 vertices. 
One very important part is to know that every vertex is assigned to at least one joint
So, at least you want to see it both ways. The three arts are joint(s) vertices, and weights. 
Objective is compact (as possible), complete user code.
• Understood. If we can manage to put the animation in the shader for x3dom, it may become attractive to then convert the other way, eg. look which joints are affecting a given vertex and use 0 weights for the other joints.

Please do not ask me to pollute my user code with this month’s guess on what is the best way to do it. 

• So perhaps a HAnim to glTF conversion is next.
Please. For skinned, I think the example I sent was the best yet.  
Please have a look at the attached.
• Let me see if x3dom understands this scene.

If it does not, then x3dom does not do x3d hanim. Pleae make it work, we need it. 

Cheers

Andreas

All Best, 
Joe

From: Andreas Plesch
Sent: Sunday, December 9, 2018 2:24 PM
To: Joe D Williams
Cc: X3D Graphics public mailing list; Michalis Kamburelis; Humanoid Animation (H-Anim) Working Group
Subject: Re: [x3d-public] glTF - HAnim conversion

In order to learn more about glTF skeletal animation versus HAnim, I
went through a slightly more realistic example conversion, manually.

https://github.com/andreasplesch/x3dom/wiki/RiggedSimple-glTF-conversion

documents this process and has both the initial glTF data and the
resulting X3D HAnim scene:
http://andreasplesch.github.io/Library/Models/HAnim/glTF/RiggedSimple.html

While the example is still very simple, the source model was
originally modelled in Blender, then exported to Collada, then
converted to glTF and then slightly cleaned up in glTF.

This is a more typical work flow. It turns out that overall it is
quite possible to convert to HAnim.

The wiki page has all binary data shown as text which is then used in
the X3D xml. VSCode has a glTF plugin which conveniently shows the
binary accessor data.

One of the bigger differences is that glTF associates a joint or
multiple joints to each vertex of a mesh while HAnim associates
vertices to joints. If there are many vertices (here 96) it becomes
hard to invert that mapping manually but it is actually pretty
straightforward to do with a few lines of code which are outlined at
the end of the page.

I think x3dom currently does not deal with normals during HAnim
deformation, eg. they are not updated. I may look into this next since
it could be a reachable target.

-Andreas

On Mon, Dec 3, 2018 at 11:09 PM Andreas Plesch <andreasplesch at gmail.com> wrote:
> 
> Hi Joe,
> 
> thanks for your insights. I will look into using the center field and have these immediate reflections.
> 
> The exercise on the wiki page is about how to go about translating a gltf skeletal animation into HAnim, in the most direct way. The glTF example is synthetic and used to test glTF renderers against each other.
> 
> glTF does not have a center field for transforms, only TRS. For centering rotations, it is necessary to use stacked transforms.
> 
> So for import purposes, I think, it may be necessary to use the translation field of Joint, rather than center, if the glTF uses the translation key as well. This may be rare.
> 
>  A more realistic but still simple glTF example would be the next step.
> 
> best,
> 
> Andreas
> 
> ---on the phone---
> 
> 
> On Mon, Dec 3, 2018, 5:03 PM Joseph D Williams <joedwil at earthlink.net wrote:
>> 
>> Hi Andreas and All,
>> 
>> 
>> 
>> Please check this out.
>> 
>> 
>> 
>> https://github.com/andreasplesch/x3dom/wiki/HAnim-and-glTF-skins
>> 
>> 
>> 
>> the example program,
>> 
>> This is actually a common learning leap, but once you try it you will see how it works.
>> 
>> 
>> 
>> You show a skeleton Joint as:
>> 
>> 
>> 
>>       <HAnimJoint DEF='node_2' name='secondglTFjoint' skinCoordIndex='' skinCoordWeight=''
>> 
>>        translation='0 0 0.2' rotation='-1 0 0 0.5235'>
>> 
>> 
>> 
>> The Joint is not positioned by its translation, but by specifying its center, thus its center of rotation. Thus
>> 
>> 
>> 
>>       <HAnimJoint DEF='node_2' name='secondglTFjoint' skinCoordIndex='' skinCoordWeight=''
>> 
>>        center='0 0 0.2' rotation='0 0 1 0'>
>> 
>> 
>> 
>> There may be tools that seem to use the translation field, but in the end, in x3d hhanim and all, you want to deal with a joint center rather than its translation. If done right, it all takes care of itself thanks to the basic features of vrml/x3d.
>> 
>> 
>> 
>> Also, why do you want to initialize the joint with some rotation? Start the thing with 0 0 1 0 default and program it from there. What will happen with a ‘standard’ animation sets you model to default pose? What will happen when your segment geometry is set to all joints 0 0 1 0?
>> 
>> 
>> 
>> Except to demo skeleton malfunction or special (non-rotary) operation if a Joint, the user never needs send translation to Joint except for the ROOT. (script may move a joint center) All translation(s) of joint(s) are the result of the parent hierarchy. I would say to actually check this out both ways and I hope you are more comfortable with Joint center.
>> 
>> 
>> 
>> And, please remember, that if J1 is a parent of J2 and you have geometry between joints J1 and J2, then that geometry must be the child of J1 or it won’t work right. You might also try that choice.
>> 
>> 
>> 
>> Thanks for Your work on this and hope for demoable results.
>> 
>> Joe
>> 
>> 
>> 
>> From: Joseph D Williams
>> Sent: Monday, December 3, 2018 9:07 AM
>> To: X3D Graphics public mailing list; Humanoid Animation (H-Anim) Working Group
>> Subject: Re: [x3d-public] HAnim and glTF skins
>> 
>> 
>> 
>> For now, we have seen ogl, vrml/x3d, collada, webgl, and now gltf, and all with the goal of providing reliable creation and transport of data. While all these provide certain features, none is directly aimed at simulation quality realtime interactive scenegraph interactions for the WWW, that I have seen. .
>> 
>> 
>> 
>> Please, I want to get this right, did I miss any? And, I meant
>> 
>> 
>> 
>> “… that I have seen, except  vrml/x3d.”
>> 
>> 
>> 
>> Thanks,
>> 
>> Joe
>> 
>> 
>> 
>> 
>> 
>> From: Joseph D Williams
>> Sent: Sunday, December 2, 2018 10:25 PM
>> To: Andreas Plesch; X3D Graphics public mailing list; Humanoid Animation (H-Anim) Working Group
>> Subject: Re: [x3d-public] HAnim and glTF skins
>> 
>> 
>> 
>> 
>> 
>> However, the tradeoff is that this makes implementations harder or perhaps completely unfeasible in some cases.
>> 
>> 
>> 
>> No author or specwriter cares about how complicated the implementation might be, they just want to stuff to work.
>> 
>> I try not to forget, but please to acknowledge while all this stuff may look new, the only thing I have ever seen that x3d contributes as new is the fact that humans can read the documentation then actually run the stuff. For instance, look at that topic of gltf transport of a type of vertex animation. This is the sort of stuff you may use to transport a pose or a series of poses, most clearly aimed at transport of video frames. It is not documentation of the process used to create the data, but just a set if frames for part of a scene. That is the typical export of tools aimed at video, they just send you so many frames and it is up to you about how this data gets used, and that is it.
>> 
>> 
>> 
>> Now, if you want to aim at realtime, then you probably don’t need all those frames with all that time resolution everywhere because you can use interpolators and other tools to author the intervals and be much more efficient. But hey, unless you know the majik words, then they won’t show you the timers and interpolators used under the covers, all they want to show you is the fixed fps stuff you need to capture a video.
>> 
>> 
>> 
>> If any authoring system is not exporting to x3d, tell ‘em ok, I can do it a style sheet if you just show me the simple common data sets I just authored. I’ll put in x3d containers and add some x3d boilerplate? Sure, export of a feature not included in x3d is obviously impossible, but what is feature documented by data not compatible with x3d? (spoiler alert: the list is long for some fancy stuff)? Just like the old days before all the majors of the time agreed to support export of vrml for the stuff that vrml did, which is mainly the basic stuff that x3d does, including the humanoidanimation, but now a new group of folks that may have forgotten that the same or compatible data is under the covers regardless of the interface exposed to the author or the target runtime and may have forgotten or are ignoring the value pf easy transport for consumer and hobby use.
>> 
>> 
>> 
>> It is the objective of x3d to directly expose the pertinent data in suitable form to have a realtime authortime and a realtime runtime. If you are authoring in a system that won’t let you export geometries, various textures, maps, lights, basic time- and event- driven animations, and cameras to vrml or x3d, then you may be dealing with a company evolved over the last 5 or so years. These fairly new tools are based on more or less the ogl and vrml and the authoring capabilities of the late 90s and 00s then big 4 or 5 technical and entertainment digital animation toolmakers of the world. See, at the time, in the mid 90’s and early 00s, vrml and then web3d was actually able to bring together a dedicated bunch that said to the big toolmakers, “not only do us bunch of enthusiastic users want a common data transport for vrml scenegraph, but so do you!” or something like that and out of that, with the deep technical knowledge and inspiration of that group, we got the basic vrml stuff, first built from the basic ogl stuff. So it may be likely that if somebody tells you that you can’t get the data that you just authored out of their box in x3d compatible form, well, it might range all the way from might have even been in on the birth of vrml or not yet born then. For now, we have seen ogl, vrml/x3d, collada, webgl, and now gltf, and all with the goal of providing reliable creation and transport of data. While all these provide certain features, none is directly aimed at simulation quality realtime interactive interactions for the WWW, that I have seen. .
>> 
>> 
>> 
>> Heck this is just me from the peanut gallery but if “perhaps completely unfeasible in some cases” then your engine has wrong features. Be sure you start with something that is aimed at readable user code and realtime event-driven runtime, that it knows what a schema snd x3d object model is, and capable of doing skins and physics and medical graphics for skeletons. As far as “unfeasible” x3d (my opinion) is first looking for features and structures proven to be widely “feasible” and in fact industry standard and actually demonstrated in a couple of open applications. However, I see your point, before it was shown to be possible, it was infeasible to expect even a 3d spinning tennis shoe but then something happened and computers got a lot better at doing some complicated 3d.
>> 
>> 
>> 
>> Thanks,
>> 
>> Joe

-- 
Andreas Plesch
Waltham, MA 02453

On Tue, Dec 11, 2018, 9:30 AM Joseph D Williams <joedwil at earthlink.net wrote:
From: Andreas Plesch
Sent: Monday, December 10, 2018 4:54 PM
To: Joe D Williams
Cc: X3D Graphics public mailing list; Michalis Kamburelis; Humanoid Animation (H-Anim) Working Group
Subject: Re: [x3d-public] glTF - HAnim conversion

Great feedback.

On Mon, Dec 10, 2018, 12:10 PM Joseph D Williams <joedwil at earthlink.net wrote:

➢ X3D xml
Some nice work and analysis and glad to see some success and understanding, however please label your “X3D xml” example as experimental or proposed, because the code is not current x3d hanim and will not work in any that I have found.

• Which browser would you recommend for best conformance ? I know it is working with x3dom  but HAnim was implemented there based on the somewhat limited number of available models.

The best one is BS Contact or Instant, I think. 

• Can you be more specific which construct is most problematic in the experimental xml ?

Well, using translation instead of center will not work that I have seen. 
See, it is different when you try to translate each joint instead of using center where all joints are really in the same space, the humanoid space.
When you use translation for each joint, then where are you translating one joint with resect to another? 
I will try and get this running from the data you have given. 
I think, as you add more joints to produce a realistic skeleton and skin, you can see the need to add some non-hanim nodes in basic x3d to produce simple joint-driven animations like you are showing. It should not be necessary to create the humanoid container in order to use freestyle joint structures, add some skin, and have a nice smooth animation of something. And, of course, Displacer also.
Yes, joint driven animation is universally supported in 3d engines. I think a boundary to HAnim would be somewhat arbitrary, the LOA levels then becoming the main feature.
 I think there is no boundary, either you want to build a humanoid out of joints or build a general purpose free-style animation. For the second case, It would be nice to build it out of objects that do not require the humanoid container. 
• One of the bigger differences is that glTF associates a joint or
multiple joints to each vertex of a mesh while HAnim associates
vertices to joints. If there are many vertices (here 96) it becomes
hard to invert that mapping manually but it is actually pretty
straightforward to do with a few lines of code which are outlined at
the end of the page.

• glTF associates a joint or
multiple joints to each vertex of a mesh 

yes, gltf apparently associates ‘all’ or always at least 16(?) joints (and weights) with each vertex. That means there is feature called a vertex that has a list of joints and weights associated with it. There is one of these for each vertex, all listing all joints and weights.

Yes. For glTF, the minimum a compliant viewer needs to support is 4 joints (out of many) per vertex. A viewer is free to ignore additional joints, with the lowest weights.

Hanim does not have any max number of joints to control a vertex. The code is run as authored with no freedom to drop stuff. I think we have already seen that in the gltf each vertex gets filled with with 16 or so by default, right?  

>while HAnim associates vertices to joints. 