[x3d-public] FW: FW: [x3d] Spec Comment by dougsanden on 19774-2: HAnimMotionCapture -V1.0

Sat Jun 27 14:14:00 PDT 2020

➢ . Q, what are the gltf inverseBindingMatrices
Notice the Joint node which lists the inverse binding matrices by listing the vertex(s) controlled by that Joint and the weights. This tells the system how to act when a joint is rotated. Move the vertex (linear along radius) with Joint rotation according to the weight. For gltf, the binding is shown for each vertex and each Joint that produces an effect on the vertex. I think in general, this data represents a vertex and the joint weights for each frame. So the data for that frame is telling the hardware how to compute the vertex displacement for the frame. 
This is basically how, if someone wants to mostly hide the actual animation routine, like a timer/interpolator/route system that wants to remain fairly hidden, and to mostly hide the binding data, to stash the binding and precomputed animation data for each frame in a multitude of small files the represent what to compute for each vertex at each frame. 

Again, to me this is fine for collecting data to make the movie, but not for realtime. For realtime, this data is basically computed the same way, but in general, the gltf data is extracted from the runtime and saved. The realtime system has to actually compute the data according to which joints have moved for each frame.

This is why some of those gltf file sets are so big, right. It is because it does not represent actual user code (except for maybe the person who lives in there) but precomputed stuff. With a file for each vertex for each frame. From the gltf it is not that easy to reconstruct the actual animation routine or even the bindings. All tyou get is basically precomputed stuffs from whatever runtime generates the files. At least this data is exposed for us to ponder over. In the old days, with tools generally oriented towards making video or film, then when you asked for the export, all you could get was a giant array, one set of actual rendered data for each finished frame. If that is what you have, then Please notice x3d includes an interpolator that allows you to increment through a stack of completed frames. 

Thanks,
Joe

From: GPU Group
Sent: Saturday, June 27, 2020 5:39 AM
To: Joseph D Williams
Cc: X3D Graphics public mailing list
Subject: Re: FW: FW: [x3d] Spec Comment by dougsanden on 19774-2: HAnimMotionCapture -V1.0

Thanks for insight. Q, what are the gltf inverseBindingMatrices mentioned in p.5, 6 https://www.khronos.org/files/gltf20-reference-guide.pdf in HAnimn terminology? 

On Sat, Jun 27, 2020 at 12:33 AM Joseph D Williams <joedwil at earthlink.net> wrote:

From: Joseph D Williams
Sent: Friday, June 26, 2020 9:53 PM
To: GPU Group
Cc: X3D Graphics public mailing list
Subject: RE: FW: [x3d] Spec Comment by dougsanden on 19774-2: HAnim MotionCapture -V1.0

• Getting past blockages in the HAnim project and on to node improvements might help later when harmonizing with glTF skinning.

Any hanim blockages are just getting some really rather straightforward simple updates (names and hierarchies) done. 

We are already harmonized with gltf, because it is the same data in json form as we author in x3d human-readable form. Nothing magic or not harmonious, just the gltf data is built for hardware instead of humans. If you have figured out what the gltf stuff is and how to use it, then you will have learned about x3d hanim. 
Likewise, the reverse is true but it is more difficult because the gltf  data is so scattered around. For example, the skin deformation is indexed by vertex usually with about 16 (for convenience) joints and weights, and one of these for each vertex.  If you look at hanim Joint, the vertices and weights associated with each joint are listed in the Joint node. So, what would you do with hanim Joint data to create some gltf vertex objects?
Joe

From: GPU Group
Sent: Friday, June 26, 2020 3:46 PM
To: Joseph D Williams
Cc: X3D Graphics public mailing list
Subject: Re: FW: [x3d] Spec Comment by dougsanden on 19774-2: HAnim MotionCapture -V1.0

Q. HAnim Examples > What does the to-do list look like? 
I ask because I got a sense that no new / modified HANim nodes would be considered beyond official HAnim2 until examples cleaned up.
Is there anything blocking that? I ask in case someone has something that can help.
One thing that popped to mind: is there a problem getting authors permission? If so and an example was demonstrating a particular thing, perhaps that scene can be dropped and a new one authored or volunteered by others reading this who may be sitting on some assets or capabilities that can volunteer equivalent scenery.
-Doug
more..
On the horizon: glTF skinning which may have the effect of bypassing HAnim, perhaps obsoleting HAnim in practice.
https://www.khronos.org/files/gltf20-reference-guide.pdf 
- p.5,6,7
Getting past blockages in the HAnim project and on to node improvements might help later when harmonizing with glTF skinning.

On Fri, Jun 26, 2020 at 3:52 PM Joseph D Williams <joedwil at earthlink.net> wrote:

• My conceptual model is a bit different. 

That is OK by me as long as it works. Whatever, Joint hierarchies will operate just like practical hierarchies of standard x3d Transform nodes, using center of rotation concept. 

For the Humanoid, if there are more joints in the capture than the playback, then you “ignore” some sets of values. If there are more joints in the playback than the capture, then, if the hierarchy is still ok, you don’t send events to the extra joints, or just send the default. 
The x3d tool should be able to help an author by first giving a list of what joints, segments, sites are available in the playback skeleton defined In the Humanoid skeleton. All the author needs to do to use existing x3d timer/interpolator/route animations is connect up the animation events to the skeleton Joint DEFs. Multiple, selectable animation routines can reside inside or outside the Humanoid, and prototype behaviors are very usable and very sharable between similar characters. In general, these animations are designed for realtime rather than frametime, but of course can be used to produce a frame at any time. However, these animations by themselves may not directly define the skeleton they apply to, or the initial pose. 

For x3d HAnim Humanoid Motion style, I think we are trying to access a perhaps more public range of animations, by the easiest way possible. From what I have seen these mostly use capture intervals related to video or film and are typically oversampled for typical realtime needs, and also, since mainly using xyz data, are subject to unpredictable animations, particularly fast wide motions. However, we want to make it convenient for an author, so we want to use the data in the bvh file to help connect the skeletons with the data. 
Turns out this may be easy because the bvh file contains configuration data for the capture skeleton as well as the animation data. Now all we must do is connect the skeletons and then the data and events are all handled under the covers. In this case, it is possible that a tool could for example, display a model of the capture skeleton and the playback skeleton to help the author decide how to proceed, or even, if the names matched, do it automagically. 
A Humanoid Motion animation resides in the Humanoid object and for best automation, should include enough information to reconstruct the capture skeleton hierarchy. In order to establish this baseline, the existing Motion node wants the author to transcribe important bvh data into x3d data forms and place it within the Humanoid Motion node. This practice also tends to help transportability between various typical capture skeletons and x3d hanim ‘standard’ skeletons. 

• Your loa4 idea makes sense if you don't want to ignore…. 

I still think the idea behind ‘ignore’ is that if you have more joints and data in for the capture skeleton than the playback skeleton can use. The author has determined that some of the capture data has to be ignored and not sent to playback skeleton, so those joints are labeled with ‘ignore’ keyword in order to declare that the related data is not to be used. Please tell me if this is not what is intended. 
If that is the case then the playback skeleton will never know because no events are sent to joints that are not there. 
If you have more joints in the playback skeleton, then it is ok and no capture data needs to be ignored and playback joints that don’t get data don’t care and work as if not there. 

Thanks for thinking about this, 
Joe

From: GPU Group
Sent: Wednesday, June 24, 2020 4:33 PM
To: Joseph D Williams
Cc: Spec Feedback; x3d at web3d.org
Subject: Re: [x3d] Spec Comment by dougsanden on 19774-2: HAnim Motion Capture -V1.0

My conceptual model is a bit different. 
- skeletal joint asks for joint information from MotionData/MotionClip, joint by joint when traversing the skeletal joint hierarchy.
Your loa4 idea makes sense if you don't want to ignore.And -depending on your implementation, like you say you'd get identity transforms -no joint motion- for joints MotionClip doesn't have.
-Doug

On Wed, Jun 24, 2020 at 5:20 PM Joseph D Williams <joedwil at earthlink.net> wrote:
• and they have a brief statement somewhere saying its ignored ie the values field has it, but the humanoid doesn;t.
• But they didn't say mathematically what to do about it.

OK, so there are Motion values for a joint in from the capture skeleton but either they do not apply to a Joint in the playback skeleton, or there is no corresponding joint in the playback skeleton. Thus, as an author, you just want to ignore that set of values in the Motion values field. 

For a typical animation routine, if you send events to a node that is not there, you get an error.
If there is a Joint in the capture skeleton that does not appear in the playback skeleton, then you don’t try to send events to that joint, because it is not there. So, if the Joint is actually present, then that Joint will remain in its default orientation. 
This is OK, and the skeleton does not mind at all, it is like the Joint is not there except it is and may have child Joint hierarchy. Of course in that case the parent of the ignored joint controls the child hierarchy. If whatever animation device is under the covers thinks it must send data to that joint, then don’t use the Motion values, just send  0 0 1 0 each cycle.

Ignore, means Hey, my capture skeleton has a Joint and I have data for it but my playback skeleton does not have that joint. I am trying to import, for example, some LOA4 animations into my LOA3 playback skeleton. As an author you are choosing to refuse to give your loa3 an update and you must confess that fact by using IGNORE in your Motion node list.  

At least that has been my experience, when using animations aimed at a higher or lower loa than the playback skeleton. For lower loa playback skeleton, just comment out the interpolators and routes for the unused Joint nodes. If higher loa playback skeleton, then problems because the hierarchy does not care about ignoring intermediate joints. Just don’t send data or if you must, just send 0 0 1 0 to the ignored joint.

This is again why you want to just go ahead and start work with the loa4 playback skeleton. Because it is ok to ignore it if you don’t have data for it. It is always not ok to not send data to something that is not there and any browser should tell you and probably should fail if you try to do that.
Thanks,.
Joe

From: GPU Group
Sent: Wednesday, June 24, 2020 3:33 PM
To: Joseph D Williams
Cc: Spec Feedback; x3d at web3d.org
Subject: Re: [x3d] Spec Comment by dougsanden on 19774-2: HAnim Motion Capture -V1.0

The IGNORED is in the Motion AnnexD example scene:
 <HAnimMotion frameCount="392" enabled='true' loop='true' frameTime = "0.033333" frameDuration=".033333"
             joints="humanoid_root, l_hip, l_knee, l_talocrural, r_hip,
                   r_knee, r_talocrural, vl5, IGNORED, l_shoulder,
                   l_elbow, l_radiocarpal, IGNORED, r_shoulder, r_elbow,
                   r_radiocarpal, IGNORED, skullbase"
and they have a brief statement somewhere saying its ignored ie the values field has it, but the humanoid doesn;t.
But they didn't say mathematically what to do about it. My interpretation is that you would accumulate the ignored values as you go down the bvh limb tree to get the next joint. If you're working call-by-call, to make sure the transform fetcher understands what's been skipped, you woult tell it the previous joint you asked for when asking for the next (and it would look at the bvh and see you are ignoring some joints)
getNextJointTransform(lastJoint,currentJoint,time,&transform)
something like that. I'm not doing that yet, and haven't proven the theory of joint transform accumulation.
The likely reason the AnnexD model didn't seem to mind / looked like its bvh animation in blender: the IGNORED values were zeros anyway. I was lucky,
The nature of matching up free downloaded bvh with any loa hanim character I have means we;ll be skipping and some and some joints never captured by the mocap system / not in the bvh list. For freewrl users, that's OK - doesn't need to be perfect, just needs to get us into the game and having fun experimenting. .
-Doug

On Wed, Jun 24, 2020 at 4:18 PM Joseph D Williams <joedwil at earthlink.net> wrote:
• - IGNORE - the transform fetcher needs to accumulate the ignored transforms to apply to the next non-ignored joint in the

Not sure what the transform fetcher does but this is no different than having an intermediate joint that is not animated.? In operation, not animating a Joint does not cause any problems. That is why it is always good to pick the loa4 skeleton for most fun. If a joint is ignored, it is just not animated and remains in its default position. Its children may still be animated. 
Anyway, am I missing something? Ignored just means there is no animation driving that Joint. The bigger problem is when you have animation trying to drive a joint that is not there. Mostly the browser will tell you that.

• No need for the software to complain if bvh doesn't match.

I think if Motion node cannot find a matching Joint somewhere, then failure until you get the list right and all named joints matched up[. 
If you look at some sample typical timer/interpolator/route setup and you try to drive a joint that is not there or has the different name, then it will usually tell you about missing sources or targets. Maybe I don’t understand what ignore refers to? What does ignore refer to? A Joint in the Humanoid skeleton, a joint in the capture skeleton,  or a set of data for a joint in the bvh file?
Thanks, 
Joe

From: GPU Group
Sent: Wednesday, June 24, 2020 5:56 AM
To: Joseph D Williams
Cc: Spec Feedback; x3d at web3d.org
Subject: Re: [x3d] Spec Comment by dougsanden on 19774-2: HAnim Motion Capture -V1.0

No need for the software to complain if bvh doesn't match.
Already in the HAnim2 specification for Motion they use the IGNORE keyword when when HH is skipping a joint the bvh has. Not shown in the spec, but likely to happen is the bvh not having a joint that the HH has, what I call NOT_IMPLEMENTED, to keep it separate from IGNORE
- IGNORE - the transform fetcher needs to accumulate the ignored transforms to apply to the next non-ignored joint in the limb tree I didn't do this yet, but should have it somewhere, so it applies also in in the original Motion node if there are IGNOREs.
- NOT_IMPLEMENTED (bvh doesn't have the joint) - the HH would get an identity transform back for that joint.
-Doug

On Tue, Jun 23, 2020 at 9:36 PM Joseph D Williams <joedwil at earthlink.net> wrote:
• Then one mapping node … 

Currently the Humanoid skeleton declares its active Joint nodes by naming each Joint node and by including list of used Joints in the Humanoid joints field.
I think the Motion node also declares the joints it wants to use by listing in the Motion node. 
To my knowledge there is no requirement that the Humanoid joints list be in any specific order. 
I think the Humanoid Motion joints field needs the list in some order relating to the data. 
The point is that for Motion to work, it must find the appropriate Joint name in the Humanoid joints field. 

My only big point is that both of those lists should look about the same (MF strings). If you want a skeleton that is matched to the bvh data, then we change the names in the imported bvh to match the names used in the skeleton field and enumerated in the Humanoid joints field. Some help could be given to an author helping to match the bvh skeleton nomenclature with the Humanoid joints. Or if you are really serious about using bvh data, then gather enough examples to suggest a ‘typical standard’ bvh skeleton and data, then compose a skeleton in the Humanoid skeleton field to match and then you might use a ‘typical standard’ Motion fields. 

The main idea of the Humanoid joints field is to list the actual joints that are available in the skeleton model. The purpose of the joints list in the Motion node is to declare which of those joints will be active in the current simulation. In this case, if the capture skeleton is the same as the playback skeleton, no problem to just change the data. 

So, the only things the browser needs to figure out is: Do the Motion joints match the skeleton joints and complain if not.

Best Regards and Thanks for thoughts on this. 
Joe

From: Spec Feedback
Sent: Tuesday, June 23, 2020 6:02 AM
To: x3d at web3d.org
Subject: [x3d] Spec Comment by dougsanden on 19774-2: HAnim Motion Capture -V1.0

-- Submitter indicates that this comment may be public: *Yes* --

Comment on 19774-2: HAnim Motion Capture - V1.0

-----------------
Proposed Explicit Name Mapping node
NameMapping
MFString A []
MFString B []

Esample:

Discussion:
not necessary to have IGNORED or NOT_IMPLEMENTED - that would be the default
if not found during lookup.
Then one mapping node can be used for multiple .bvh from various sources.
Putting a mapping node into both HAnimHumanoid HH and HAnimMotionClip/Data
HMC would allow a 2-step lookup:
- HH to loa, loa to HMC
- A previous comment shows the math advantage of 2 step
If no mapping nodes present, browser would assume names are same in HH and
HMC
- if one mapping node in either HH or HMC, then a 1-step lookup is done

-----------------

Submitted on Tuesday, 2020,  June 23 - 7:02am
by dougsanden (dougsanden )
IP: 75.159.18.239

See: https://www.web3d.org/node/1694/submission/4051

_______________________________________________
x3d mailing list
x3d at web3d.org
http://web3d.org/mailman/listinfo/x3d_web3d.org

_______________________________________________
x3d mailing list
x3d at web3d.org
http://web3d.org/mailman/listinfo/x3d_web3d.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20200627/9b10f9a5/attachment-0001.html>