[x3d-public] X3D Facial Animation examples and Facial Action Coding System (FACS)
John Carlson
yottzumm at gmail.com
Wed Jan 8 09:17:38 PST 2025
Understood that not all systems animate HAnimDisplacers or even
HAnimSegments. I understand that adding Jaw, eyeball, eyelid and eyebrow
joints would make animations easier, especially for skinning. I know some
skeletons in Blender represent facial features as bones (aka joints). I
delivered a Blender export to Joe recently of the most complex one I could
find. I will continue, as best I can, to provide at least Joint exports.
Skinning will take a bit more work, which I hope to complete by the
weekend, assuming I don’t have any surprises. This does not mean I will
have multiple mesh skin or geometry working, just a single mesh for now.
Multiple mesh still requires more work. There’s no estimate for that.
I’m also working on non-bone based animation export working Blender, but
single skin is the priority.
On Tue, Jan 7, 2025 at 7:26 PM Brutzman, Donald (Don) (CIV) <
brutzman at nps.edu> wrote:
> Joe and I looked at this a bit more today. Some observations:
>
> - There are many HAnimSegment nodes that are not needed. They can
> probably be removed without harm, since they are not receiving any events.
>
> - There are no regular facial HAnimJoint nodes present so it is not
> possible to consistently animate the jaw or eyes/eyelids/eyebrow. This
> might be a more interesting variation to have. It would entail having
> Shape nodes for corresponding skin geometry in those locations.
>
> - HAnim 2.0, 4.9.5 The face
> -
> https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#TheFace
>
>
> *Table 4.6 — Face Joint object names*
> l_eyeball_joint
> r_eyeball_joint
> l_eyebrow_joint
> r_eyebrow_joint
> l_eyelid_joint
> r_eyelid_joint
> temporomandibular
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, navy robotics
> https://faculty.nps.edu/brutzman
>
>
>
> ------------------------------
> *From:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
> *Sent:* Tuesday, January 7, 2025 2:37 PM
> *To:* Myeong Won Lee <myeongwonlee at gmail.com>; puk at igraphics.com <
> puk at igraphics.com>; Joseph D Williams <joedwil at earthlink.net>; William
> Glascoe <eosocxo at comcast.net>; Carol McDonald <cemd2 at comcast.net>; Katy
> Schildmeyer KS APPAREL DESIGN <katy at ksappareldesign.com>; John Carlson <
> yottzumm at gmail.com>; Anita Havele <anita.havele at web3d.org>; Nicholas
> Polys <npolys at vt.edu>
> *Cc:* hanim at web3d.org <hanim at web3d.org>; X3D Public Mailing List (
> x3d-public at web3d.org) <x3d-public at web3d.org>
> *Subject:* Re: X3D Facial Animation examples and Facial Action Coding
> System (FACS)
>
> Reposting this report that John Carlson's excellent combination of models
> is available. Full screen version:
>
> - X3D Example Archives: Humanoid Animation, Facial Animation, Facial
> Animation Comparison Scripts
> - User selection of one or more X3D models with alternate Facial
> Animation Control System (FACS) behaviors with animation performed by
> interpolators.
> -
> https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FacialAnimationComparisonScriptsX_ITE.html
>
> Am noticing at least one error, the meta tag indicates a TODO for adding
> HAnimDisplacer nodes, but many are there already.
>
> Suggestion: there are so many possible selections to combine that it is
> hard to see the face. How about:
>
> - Split bright-blue menus into 2 or 3 adjacent columns on the
> right-hand side
> - Bring Jin's animated face closer on left-hand side
>
> Feedback and updates welcome. Have fun with X3D HAnim! 🙂
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, navy robotics
> https://faculty.nps.edu/brutzman
>
>
>
> ------------------------------
> *From:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
> *Sent:* Tuesday, October 29, 2024 3:17 AM
> *To:* Myeong Won Lee <myeongwonlee at gmail.com>; puk at igraphics.com <
> puk at igraphics.com>; Joseph D Williams <joedwil at earthlink.net>; William
> Glascoe <eosocxo at comcast.net>; Carol McDonald <cemd2 at comcast.net>; Katy
> Schildmeyer KS APPAREL DESIGN <katy at ksappareldesign.com>; John Carlson <
> yottzumm at gmail.com>; Anita Havele <anita.havele at web3d.org>; Nicholas
> Polys <npolys at vt.edu>
> *Cc:* hanim at web3d.org <hanim at web3d.org>; X3D Public Mailing List (
> x3d-public at web3d.org) <x3d-public at web3d.org>
> *Subject:* X3D Facial Animation examples and Facial Action Coding System
> (FACS)
>
> At long last we have published and refined an important new set of
> examples, using 30 excellent original X3D models created by Dr. Myeong Won
> Lee and her team.
>
> - X3D Example Archives: Humanoid Animation, Facial Animation
> -
> Https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation
>
> - *Background.* Computer facial animation
> <https://en.wikipedia.org/wiki/Computer_facial_animation> is an area
> of current work for HAnim. Example models in this directory use the Facial
> Action Coding System (FACS)
> <https://en.wikipedia.org/wiki/Facial_Action_Coding_System> to provide
> Facial Animation examples. A variety of Action Unit (AU)
> <https://web.cs.wpi.edu/~matt/courses/cs563/talks/face_anim/ekman.html> enumeration
> values are defined for each facial expression in FACS. The FACS Visual
> Guidebook
> <https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system> showcases
> the expressive power of this approach.
> - *Initial examples.* The original example models
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals> show
> basic patterns for modifying coordinates of a custom mesh. The Suwon
> VRLAB X3D Converter
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/SuwonHAnimFacialMotionConverter.mp4> was
> used to compute CoordinateInterpolator animation values.
> The README: Original FACS Models
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html> page
> describes the original contributions and subsequent X3D model refinements.
> The HAnim Facial Animation Design Patterns
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/HAnimFacialAnimationDesignPatterns.png> diagram
> illustrates these modifications.
> - *Next steps.* The Future Work: FACS for X3D HAnim
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FutureWork.html> page
> describes plans for continuing progress. Continuing refinement of animation
> models may lead to formally defining additional features and functionality
> in the Humanoid Animation (HAnim) International Standard
> <https://www.web3d.org/documents/specifications/19774/V2.0/index.html>.
>
> Details regarding model changes follow, along with plans for future work.
>
> Special thanks to Joe Williams for insights and assistance working through
> these issues. It feels like a solid basis for X3D HAnim Facial Animation
> is emerging.
>
> These deserve review by the HAnim working group, perhaps in a special
> meeting. Thanks in advance for all insights, improvements, and feedback.
>
> =======================================
>
> - *README: Original FACS Models*
> -
> https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html
>
> *Facial Action Coding System (FACS) original X3D models*
> Dr. Myeong Won Lee and team created these original models. She provided
> the following documentation 8 July 2024.
>
> The HAnim FACS animations are generated as follows:
>
> 1. A simple motion of facial expression is represented by a single
> action unit of FACS, such as inner brow raise or wink. More complex facial
> animation that is made up of a combination of motions, such as a smile made
> up of both eye and mouth motions, is defined by combining several action
> units.
> 2. To represent an action unit in FACS, a minimum of three sets of an
> LOE1 HAnim facial model (three X3D files) can be used. An LOE1 HAnim facial
> model is defined by 43 facial regions. Currently, up to five sets of HAnim
> faces can be used to generate a FACS action unit for a facial expression.
> 3. The sets of HAnim facial models for an action unit of FACS are
> animated using X3D keyframe interpolators. This animation can be generated
> by the HAnim facial converter.
>
> Example X3D animation files for FACS action units for facial animation are
> located at the first shared drive
> <https://drive.google.com/drive/folders/1Ja95KSOmoY54iqDdO7fZ3KR2Jo80WbFT?usp=sharing>
> link.
> The HAnim Facial Motion Converter used for the HAnim FACS animation can be
> downloaded at the second shared drive
> <https://drive.google.com/drive/folders/1v5CPn_EFig9jxLpfjAW34WKDQU6KdmCk?usp=drive_link>
> link.
> All the files submitted or notified to Web3D can be used freely without
> any restriction. It would be good if the facial examples can be uploaded to
> the Web3D resource pages. In addition, it is ok if all the submitted files
> that we developed are assigned Web3D software licenses.
>
> *X3D Model Changes*
> Continuing changes have been applied to content in this originals
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/> subdirectory
> to produced modified models in the parent FacialAnimation
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/> directory.
> List of changes follows.
>
> 1. File names adjusted to meet X3D Scene Authoring Hints: Naming
> Conventions
> <https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#NamingConventions>
> 2. Added DOCTYPE DTD statements for improved XML validation
> 3. X3D canonicalization (c14n) for consistent whitespace and attribute
> ordering
> 4. Model metadata of various kinds, following best practices described
> in X3D Scene Authoring Hints: meta Statements
> <https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#metaStatements>
> 5. Added Web3D Consortium open-source license
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/license.html>
> 6. Fixed DEF/USE of ImageTexture files to avoid repetitive,
> time-consuming reloading of the renamed file from Jin.jpg
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/Jin.jpg> to
> JinImageAtlas.jpg
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/JinImageAtlas.jpg>
> 7. Fixed DEF of Coordinate nodes to end in _COORD vice -COORD for best
> portability and to match naming conventions
> 8. Fixed DEF/USE of Material files to avoid repetitive reloading and
> then Appearance files
> 9. Renamed all files to put "Jin" first, which will help
> alphabetization when we add more FACS examples in the future
> 10. Applied correct name to meta title, meta identifier, and WorldInfo
> title attributes
> 11. Added meta reference links to original version of each file
> 12. Combined duplicative clocks and ScalarInterpolator adapters, omit
> unnecessary Adapter ROUTEs
> 13. Added directory package descriptions
> 14. Under evaluation: changing animation loop period from
> cycleInteval="4" to cycleInteval="0.99" in order to make model-animation
> visualizations more obvious.
> 15. TODO adjust select models to restore prior more-complex
> ScalarInterpolator clock reversals
> 16. TODO rescale units to meters
>
> Many model refinements have been applied. For example, see the subversion
> diff log at revision 36730 <https://sourceforge.net/p/x3d/code/36730>.
> These changes all preserve and enhance the orginal features of this model
> set.
> =======================================
>
> - *Future Work: Facial Action Coding System (FACS) for X3D HAnim
> models*
> -
> https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FutureWork.html
>
> *Future Work: Facial Action Coding System (FACS) for X3D HAnim models*
> These plans are based on work adapting models by Dr. Myeong Won Lee, as
> described in the README: Original FACS Models
> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html>
> .
> *Lessons Learned, So Far*
>
> 1. Every head will likely have different geometry, textures, colors,
> etc. There is no need for every head to be the same.
> 2. A single head model file needs to align with HAnim joints and
> segments, if it is going to be interoperable with other HAnim skeleton/skin
> models.
> 3. Each head model will connect to the skeleton as shown in
>
> Part 1: Humanoid animation (HAnim) architecture, Section 4 Concepts, 4.9.6.3 LOA-3 hierarchy <https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#Hierarchy3>
> humanoid_root : sacrum
> [...]
> vc7 : c7
> | vc6 : c6
> | vc5 : c5
> | vc4 : c4
> | vc3 : c3
> | vc2 : c2
> | vc1 : c1
> | *skullbase : skull*
> | l_eyelid_joint : l_eyelid
> | r_eyelid_joint : r_eyelid
> | l_eyeball_joint : l_eyeball
> | r_eyeball_joint : r_eyeball
> | l_eyebrow_joint : l_eyebrow
> | r_eyebrow_joint : r_eyebrow
> | temporomandibular : jaw
>
> 4. Typically a new head will connect at *HAnimJoint skullbase* and *HAnimSegment
> skull*, possibly including neck joints as well.
> 5. Typically an interoperable HAnimHumanoid head will include
> HAnimJoint and HAnimSegment definitions for eyelid, eyeball, eyebrow, and
> jaw.
> 6. The HAnim specification includes guidance on connecting partial
> body models together in 4.8.6 Partial HAnim figures
> <https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#PartialHAnimFigures>
> .
> 7. No HAnimSite feature points are required in the HAnim specification.
> 8. We can add HAnimSite feature points to the above HAnimSegment
> notes, without interfering with any of the native HAnim motion animations.
> However adding HAnimSite locations to the other facial geometry many need
> further attention. Adding a useful set of HAnimSite feature points for
> facial animation might be very useful.
>
> *Next Steps*
>
> 1. We want head models that include the ability to use an FACS Action
> Unit (AU) enumeration that shows a new expression.
> 2. We currently have single expressions demonstrated in separate model
> files. Showing one expression at a time is the initial goal for facial
> expression animations.
> 3. Integrating the animations together efficiently, within a single
> X3D model file, is the path towards having interoperably animatable heads
> that can move eyes/jaw and also show expressions.
> 4. We first want single-AU responses working, and then multiple-AU
> responses working, in order to achieve the full range of FACS expressions
> that are possible.
> 5. For full animation compatibility, head models need to include the
> standard HAnim joints and segments for eyelids, eyeballs, eyebrows, and
> jaw. This requirement deserves careful consideration when designing mesh
> geometry.
> 6. See the FACS Visual Guidebook
> <https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system/> for
> video examples with corresponding descriptions that illustrate the
> expressions conveyed by each Action Unit (AU), either individually or in
> combination.
>
> Building example models is the path towards showing repeatable
> capabilities. Conceivably these capabilities might become a new HAnim node
> for HAnimHumanoid. Successful, repeatable design patterns can then be
> formalized in the future HAnim Specification.
> =======================================
>
> Summary: interoperable facial animation behaviors are beginning to look
> feasible and repeatable. More demonstration work awaits.
>
> Have fun with X3D Humanoid Animation (HAnim)! 🙂
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, navy robotics
> https://faculty.nps.edu/brutzman
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20250108/bdf221d0/attachment-0001.html>
More information about the x3d-public
mailing list