[x3d-public] X3D Facial Animation examples and Facial Action Coding System (FACS)

[Brutzman, Donald (Don) (CIV)]

At long last we have published and refined an important new set of examples, using 30 excellent original X3D models created by Dr. Myeong Won Lee and her team.

  *
X3D Example Archives: Humanoid Animation, Facial Animation
  *
Https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation
  *
Background. Computer facial animation<https://en.wikipedia.org/wiki/Computer_facial_animation> is an area of current work for HAnim. Example models in this directory use the Facial Action Coding System (FACS)<https://en.wikipedia.org/wiki/Facial_Action_Coding_System> to provide Facial Animation examples. A variety of Action Unit (AU)<https://web.cs.wpi.edu/~matt/courses/cs563/talks/face_anim/ekman.html> enumeration values are defined for each facial expression in FACS. The FACS Visual Guidebook<https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system> showcases the expressive power of this approach.
  *
Initial examples. The original example models<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals> show basic patterns for modifying coordinates of a custom mesh. The Suwon VRLAB X3D Converter<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/SuwonHAnimFacialMotionConverter.mp4> was used to compute CoordinateInterpolator animation values.
The README: Original FACS Models<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html> page describes the original contributions and subsequent X3D model refinements. The HAnim Facial Animation Design Patterns<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/HAnimFacialAnimationDesignPatterns.png> diagram illustrates these modifications.
  *
Next steps. The Future Work: FACS for X3D HAnim<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FutureWork.html> page describes plans for continuing progress. Continuing refinement of animation models may lead to formally defining additional features and functionality in the Humanoid Animation (HAnim) International Standard<https://www.web3d.org/documents/specifications/19774/V2.0/index.html>.

Details regarding model changes follow, along with plans for future work.

Special thanks to Joe Williams for insights and assistance working through these issues.  It feels like a solid basis for X3D HAnim Facial Animation is emerging.

These deserve review by the HAnim working group, perhaps in a special meeting.  Thanks in advance for all insights, improvements, and feedback.

=======================================

  *
README: Original FACS Models
  *
https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html

Facial Action Coding System (FACS) original X3D models
Dr. Myeong Won Lee and team created these original models. She provided the following documentation 8 July 2024.
The HAnim FACS animations are generated as follows:

  1.  A simple motion of facial expression is represented by a single action unit of FACS, such as inner brow raise or wink. More complex facial animation that is made up of a combination of motions, such as a smile made up of both eye and mouth motions, is defined by combining several action units.
  2.  To represent an action unit in FACS, a minimum of three sets of an LOE1 HAnim facial model (three X3D files) can be used. An LOE1 HAnim facial model is defined by 43 facial regions. Currently, up to five sets of HAnim faces can be used to generate a FACS action unit for a facial expression.
  3.  The sets of HAnim facial models for an action unit of FACS are animated using X3D keyframe interpolators. This animation can be generated by the HAnim facial converter.

Example X3D animation files for FACS action units for facial animation are located at the first shared drive<https://drive.google.com/drive/folders/1Ja95KSOmoY54iqDdO7fZ3KR2Jo80WbFT?usp=sharing> link.
The HAnim Facial Motion Converter used for the HAnim FACS animation can be downloaded at the second shared drive<https://drive.google.com/drive/folders/1v5CPn_EFig9jxLpfjAW34WKDQU6KdmCk?usp=drive_link> link.
All the files submitted or notified to Web3D can be used freely without any restriction. It would be good if the facial examples can be uploaded to the Web3D resource pages. In addition, it is ok if all the submitted files that we developed are assigned Web3D software licenses.
X3D Model Changes
Continuing changes have been applied to content in this originals<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/> subdirectory to produced modified models in the parent FacialAnimation<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/> directory. List of changes follows.

  1.  File names adjusted to meet X3D Scene Authoring Hints: Naming Conventions<https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#NamingConventions>
  2.  Added DOCTYPE DTD statements for improved XML validation
  3.  X3D canonicalization (c14n) for consistent whitespace and attribute ordering
  4.  Model metadata of various kinds, following best practices described in X3D Scene Authoring Hints: meta Statements<https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#metaStatements>
  5.  Added Web3D Consortium open-source license<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/license.html>
  6.  Fixed DEF/USE of ImageTexture files to avoid repetitive, time-consuming reloading of the renamed file from Jin.jpg<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/Jin.jpg> to JinImageAtlas.jpg<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/JinImageAtlas.jpg>
  7.  Fixed DEF of Coordinate nodes to end in _COORD vice -COORD for best portability and to match naming conventions
  8.  Fixed DEF/USE of Material files to avoid repetitive reloading and then Appearance files
  9.  Renamed all files to put "Jin" first, which will help alphabetization when we add more FACS examples in the future
  10. Applied correct name to meta title, meta identifier, and WorldInfo title attributes
  11. Added meta reference links to original version of each file
  12. Combined duplicative clocks and ScalarInterpolator adapters, omit unnecessary Adapter ROUTEs
  13. Added directory package descriptions
  14. Under evaluation: changing animation loop period from cycleInteval="4" to cycleInteval="0.99" in order to make model-animation visualizations more obvious.
  15. TODO adjust select models to restore prior more-complex ScalarInterpolator clock reversals
  16. TODO rescale units to meters

Many model refinements have been applied. For example, see the subversion diff log at revision 36730<https://sourceforge.net/p/x3d/code/36730>. These changes all preserve and enhance the orginal features of this model set.
=======================================

  *
Future Work: Facial Action Coding System (FACS) for X3D HAnim models
  *
https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FutureWork.html

Future Work: Facial Action Coding System (FACS) for X3D HAnim models
These plans are based on work adapting models by Dr. Myeong Won Lee, as described in the README: Original FACS Models<https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html>.
Lessons Learned, So Far

  1.  Every head will likely have different geometry, textures, colors, etc. There is no need for every head to be the same.
  2.  A single head model file needs to align with HAnim joints and segments, if it is going to be interoperable with other HAnim skeleton/skin models.
  3.  Each head model will connect to the skeleton as shown in

    Part 1:  Humanoid animation (HAnim) architecture, Section 4 Concepts, 4.9.6.3 LOA-3 hierarchy<https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#Hierarchy3>

    humanoid_root : sacrum
        [...]
            vc7 : c7
            | vc6 : c6
            |   vc5 : c5
            |     vc4 : c4
            |       vc3 : c3
            |         vc2 : c2
            |           vc1 : c1
            |             skullbase : skull
            |               l_eyelid_joint : l_eyelid
            |               r_eyelid_joint : r_eyelid
            |               l_eyeball_joint : l_eyeball
            |               r_eyeball_joint : r_eyeball
            |               l_eyebrow_joint : l_eyebrow
            |               r_eyebrow_joint : r_eyebrow
            |               temporomandibular : jaw


  4.  Typically a new head will connect at HAnimJoint skullbase and HAnimSegment skull, possibly including neck joints as well.
  5.  Typically an interoperable HAnimHumanoid head will include HAnimJoint and HAnimSegment definitions for eyelid, eyeball, eyebrow, and jaw.
  6.  The HAnim specification includes guidance on connecting partial body models together in 4.8.6 Partial HAnim figures<https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#PartialHAnimFigures>.
  7.  No HAnimSite feature points are required in the HAnim specification.
  8.  We can add HAnimSite feature points to the above HAnimSegment notes, without interfering with any of the native HAnim motion animations. However adding HAnimSite locations to the other facial geometry many need further attention. Adding a useful set of HAnimSite feature points for facial animation might be very useful.

Next Steps

  1.  We want head models that include the ability to use an FACS Action Unit (AU) enumeration that shows a new expression.
  2.  We currently have single expressions demonstrated in separate model files. Showing one expression at a time is the initial goal for facial expression animations.
  3.
Integrating the animations together efficiently, within a single X3D model file, is the path towards having interoperably animatable heads that can move eyes/jaw and also show expressions.
  4.  We first want single-AU responses working, and then multiple-AU responses working, in order to achieve the full range of FACS expressions that are possible.
  5.
For full animation compatibility, head models need to include the standard HAnim joints and segments for eyelids, eyeballs, eyebrows, and jaw. This requirement deserves careful consideration when designing mesh geometry.
  6.
See the FACS Visual Guidebook<https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system/> for video examples with corresponding descriptions that illustrate the expressions conveyed by each Action Unit (AU), either individually or in combination.

Building example models is the path towards showing repeatable capabilities. Conceivably these capabilities might become a new HAnim node for HAnimHumanoid. Successful, repeatable design patterns can then be formalized in the future HAnim Specification.
=======================================

Summary: interoperable facial animation behaviors are beginning to look feasible and repeatable.   More demonstration work awaits.

Have fun with X3D Humanoid Animation (HAnim)!   🙂


all the best, Don

--

Don Brutzman  Naval Postgraduate School, Code USW/Br        brutzman at nps.edu

Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA    +1.831.656.2149

X3D graphics, virtual worlds, navy robotics https://faculty.nps.edu/brutzman


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20241029/03ba86d8/attachment-0001.html>