[x3d-public] X3D Facial Animation examples and Facial Action Coding System (FACS)
John Carlson
yottzumm at gmail.com
Tue Jan 7 18:25:40 PST 2025
Here are the “Segments” from FACS:
# Segment prefixes for related nodes def_prefixes = ["Lower_teeth", "Hair",
"__0", "__2", "__4", "Center_lower_vermillion_lip", "Chin", "Glabella",
"Left_bulbar_conjunctiva", "Left_cheek", "Left_dorsum", "Left_ear",
"Left_eyebrow", "Left_forehead", "Left_lower_eyelid",
"Left_lower_vermillion_lip", "Left_nasolabial_cheek", "Left_nostril",
"Left_pupil", "Left_temple", "Left_upper_cutaneous_lip",
"Left_upper_eyelid", "Left_upper_vermillion_lip",
"Left_upper_vermillion_lip001", "Mid_forehead", "Mid_nasal_dorsum",
"Mid_upper_vermillion_lip", "Nasal_tip", "Neck", "Occipital_scalp",
"Philtrum", "Right_bulbar_conjunctiva", "Right_cheek", "Right_dorsum",
"Right_ear", "Right_eyebrow", "Right_forehead", "Right_lower_eyelid",
"Right_lower_vermillion_lip", "Right_nasolabial_cheek", "Right_nostril",
"Right_pupil", "Right_temple", "Right_upper_cutaneous_lip",
"Right_upper_eyelid", "Right_upper_vermillion_lip", "Tongue",
"Upper_teeth"]
Note that there’s an upper eyelid and lower eyelid.
Note that I have tongue, upper teeth and lower teeth as segments, and
there’s a jaw segment on the standard.
On Tue, Jan 7, 2025 at 8:13 PM John Carlson <yottzumm at gmail.com> wrote:
> Agreed on unneeded HAnimSegments. Realize that segments some come from the
> 30 original documents, and we mostly selected the geometry from a single
> document. The list of segments is exhaustive on purpose, and stored in the
> code. That’s how I query the XML.
>
> I am confused because HAnimJoints do not provide for Shapes (?). Please
> provide mapping from Joints to Segments (non-standard, perhaps—map FACS
> segments to HAnim), as I am only aware that a joint maps to one segment.
> If the geometry is a child of a HAnimSite, those should be added.
>
> On Tue, Jan 7, 2025 at 7:26 PM Brutzman, Donald (Don) (CIV) <
> brutzman at nps.edu> wrote:
>
>> Joe and I looked at this a bit more today. Some observations:
>>
>> - There are many HAnimSegment nodes that are not needed. They can
>> probably be removed without harm, since they are not receiving any events.
>>
>> - There are no regular facial HAnimJoint nodes present so it is not
>> possible to consistently animate the jaw or eyes/eyelids/eyebrow. This
>> might be a more interesting variation to have. It would entail having
>> Shape nodes for corresponding skin geometry in those locations.
>>
>> - HAnim 2.0, 4.9.5 The face
>> -
>> https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#TheFace
>>
>>
>> *Table 4.6 — Face Joint object names*
>> l_eyeball_joint
>> r_eyeball_joint
>> l_eyebrow_joint
>> r_eyebrow_joint
>> l_eyelid_joint
>> r_eyelid_joint
>> temporomandibular
>>
>>
>> all the best, Don
>>
>> --
>>
>> Don Brutzman Naval Postgraduate School, Code USW/Br
>> brutzman at nps.edu
>>
>> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
>> +1.831.656.2149
>>
>> X3D graphics, virtual worlds, navy robotics
>> https://faculty.nps.edu/brutzman
>>
>>
>>
>> ------------------------------
>> *From:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
>> *Sent:* Tuesday, January 7, 2025 2:37 PM
>> *To:* Myeong Won Lee <myeongwonlee at gmail.com>; puk at igraphics.com <
>> puk at igraphics.com>; Joseph D Williams <joedwil at earthlink.net>; William
>> Glascoe <eosocxo at comcast.net>; Carol McDonald <cemd2 at comcast.net>; Katy
>> Schildmeyer KS APPAREL DESIGN <katy at ksappareldesign.com>; John Carlson <
>> yottzumm at gmail.com>; Anita Havele <anita.havele at web3d.org>; Nicholas
>> Polys <npolys at vt.edu>
>> *Cc:* hanim at web3d.org <hanim at web3d.org>; X3D Public Mailing List (
>> x3d-public at web3d.org) <x3d-public at web3d.org>
>> *Subject:* Re: X3D Facial Animation examples and Facial Action Coding
>> System (FACS)
>>
>> Reposting this report that John Carlson's excellent combination of models
>> is available. Full screen version:
>>
>> - X3D Example Archives: Humanoid Animation, Facial Animation, Facial
>> Animation Comparison Scripts
>> - User selection of one or more X3D models with alternate Facial
>> Animation Control System (FACS) behaviors with animation performed by
>> interpolators.
>> -
>> https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FacialAnimationComparisonScriptsX_ITE.html
>>
>> Am noticing at least one error, the meta tag indicates a TODO for adding
>> HAnimDisplacer nodes, but many are there already.
>>
>> Suggestion: there are so many possible selections to combine that it is
>> hard to see the face. How about:
>>
>> - Split bright-blue menus into 2 or 3 adjacent columns on the
>> right-hand side
>> - Bring Jin's animated face closer on left-hand side
>>
>> Feedback and updates welcome. Have fun with X3D HAnim! 🙂
>>
>>
>> all the best, Don
>>
>> --
>>
>> Don Brutzman Naval Postgraduate School, Code USW/Br
>> brutzman at nps.edu
>>
>> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
>> +1.831.656.2149
>>
>> X3D graphics, virtual worlds, navy robotics
>> https://faculty.nps.edu/brutzman
>>
>>
>>
>> ------------------------------
>> *From:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
>> *Sent:* Tuesday, October 29, 2024 3:17 AM
>> *To:* Myeong Won Lee <myeongwonlee at gmail.com>; puk at igraphics.com <
>> puk at igraphics.com>; Joseph D Williams <joedwil at earthlink.net>; William
>> Glascoe <eosocxo at comcast.net>; Carol McDonald <cemd2 at comcast.net>; Katy
>> Schildmeyer KS APPAREL DESIGN <katy at ksappareldesign.com>; John Carlson <
>> yottzumm at gmail.com>; Anita Havele <anita.havele at web3d.org>; Nicholas
>> Polys <npolys at vt.edu>
>> *Cc:* hanim at web3d.org <hanim at web3d.org>; X3D Public Mailing List (
>> x3d-public at web3d.org) <x3d-public at web3d.org>
>> *Subject:* X3D Facial Animation examples and Facial Action Coding System
>> (FACS)
>>
>> At long last we have published and refined an important new set of
>> examples, using 30 excellent original X3D models created by Dr. Myeong Won
>> Lee and her team.
>>
>> - X3D Example Archives: Humanoid Animation, Facial Animation
>> -
>> Https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation
>>
>> - *Background.* Computer facial animation
>> <https://en.wikipedia.org/wiki/Computer_facial_animation> is an area
>> of current work for HAnim. Example models in this directory use the Facial
>> Action Coding System (FACS)
>> <https://en.wikipedia.org/wiki/Facial_Action_Coding_System> to
>> provide Facial Animation examples. A variety of Action Unit (AU)
>> <https://web.cs.wpi.edu/~matt/courses/cs563/talks/face_anim/ekman.html> enumeration
>> values are defined for each facial expression in FACS. The FACS
>> Visual Guidebook
>> <https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system> showcases
>> the expressive power of this approach.
>> - *Initial examples.* The original example models
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals> show
>> basic patterns for modifying coordinates of a custom mesh. The Suwon
>> VRLAB X3D Converter
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/SuwonHAnimFacialMotionConverter.mp4> was
>> used to compute CoordinateInterpolator animation values.
>> The README: Original FACS Models
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html> page
>> describes the original contributions and subsequent X3D model refinements.
>> The HAnim Facial Animation Design Patterns
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/HAnimFacialAnimationDesignPatterns.png> diagram
>> illustrates these modifications.
>> - *Next steps.* The Future Work: FACS for X3D HAnim
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FutureWork.html> page
>> describes plans for continuing progress. Continuing refinement of animation
>> models may lead to formally defining additional features and functionality
>> in the Humanoid Animation (HAnim) International Standard
>> <https://www.web3d.org/documents/specifications/19774/V2.0/index.html>
>> .
>>
>> Details regarding model changes follow, along with plans for future work.
>>
>> Special thanks to Joe Williams for insights and assistance working
>> through these issues. It feels like a solid basis for X3D HAnim Facial
>> Animation is emerging.
>>
>> These deserve review by the HAnim working group, perhaps in a special
>> meeting. Thanks in advance for all insights, improvements, and feedback.
>>
>> =======================================
>>
>> - *README: Original FACS Models*
>> -
>> https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html
>>
>> *Facial Action Coding System (FACS) original X3D models*
>> Dr. Myeong Won Lee and team created these original models. She provided
>> the following documentation 8 July 2024.
>>
>> The HAnim FACS animations are generated as follows:
>>
>> 1. A simple motion of facial expression is represented by a single
>> action unit of FACS, such as inner brow raise or wink. More complex facial
>> animation that is made up of a combination of motions, such as a smile made
>> up of both eye and mouth motions, is defined by combining several action
>> units.
>> 2. To represent an action unit in FACS, a minimum of three sets of an
>> LOE1 HAnim facial model (three X3D files) can be used. An LOE1 HAnim facial
>> model is defined by 43 facial regions. Currently, up to five sets of HAnim
>> faces can be used to generate a FACS action unit for a facial expression.
>> 3. The sets of HAnim facial models for an action unit of FACS are
>> animated using X3D keyframe interpolators. This animation can be generated
>> by the HAnim facial converter.
>>
>> Example X3D animation files for FACS action units for facial animation
>> are located at the first shared drive
>> <https://drive.google.com/drive/folders/1Ja95KSOmoY54iqDdO7fZ3KR2Jo80WbFT?usp=sharing>
>> link.
>> The HAnim Facial Motion Converter used for the HAnim FACS animation can
>> be downloaded at the second shared drive
>> <https://drive.google.com/drive/folders/1v5CPn_EFig9jxLpfjAW34WKDQU6KdmCk?usp=drive_link>
>> link.
>> All the files submitted or notified to Web3D can be used freely without
>> any restriction. It would be good if the facial examples can be uploaded to
>> the Web3D resource pages. In addition, it is ok if all the submitted files
>> that we developed are assigned Web3D software licenses.
>>
>> *X3D Model Changes*
>> Continuing changes have been applied to content in this originals
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/> subdirectory
>> to produced modified models in the parent FacialAnimation
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/> directory.
>> List of changes follows.
>>
>> 1. File names adjusted to meet X3D Scene Authoring Hints: Naming
>> Conventions
>> <https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#NamingConventions>
>> 2. Added DOCTYPE DTD statements for improved XML validation
>> 3. X3D canonicalization (c14n) for consistent whitespace and
>> attribute ordering
>> 4. Model metadata of various kinds, following best practices
>> described in X3D Scene Authoring Hints: meta Statements
>> <https://www.web3d.org/x3d/content/examples/X3dSceneAuthoringHints.html#metaStatements>
>> 5. Added Web3D Consortium open-source license
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/license.html>
>> 6. Fixed DEF/USE of ImageTexture files to avoid repetitive,
>> time-consuming reloading of the renamed file from Jin.jpg
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/Jin.jpg> to
>> JinImageAtlas.jpg
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/JinImageAtlas.jpg>
>> 7. Fixed DEF of Coordinate nodes to end in _COORD vice -COORD for
>> best portability and to match naming conventions
>> 8. Fixed DEF/USE of Material files to avoid repetitive reloading and
>> then Appearance files
>> 9. Renamed all files to put "Jin" first, which will help
>> alphabetization when we add more FACS examples in the future
>> 10. Applied correct name to meta title, meta identifier, and
>> WorldInfo title attributes
>> 11. Added meta reference links to original version of each file
>> 12. Combined duplicative clocks and ScalarInterpolator adapters, omit
>> unnecessary Adapter ROUTEs
>> 13. Added directory package descriptions
>> 14. Under evaluation: changing animation loop period from
>> cycleInteval="4" to cycleInteval="0.99" in order to make model-animation
>> visualizations more obvious.
>> 15. TODO adjust select models to restore prior more-complex
>> ScalarInterpolator clock reversals
>> 16. TODO rescale units to meters
>>
>> Many model refinements have been applied. For example, see the subversion
>> diff log at revision 36730 <https://sourceforge.net/p/x3d/code/36730>.
>> These changes all preserve and enhance the orginal features of this model
>> set.
>> =======================================
>>
>> - *Future Work: Facial Action Coding System (FACS) for X3D HAnim
>> models*
>> -
>> https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/FutureWork.html
>>
>> *Future Work: Facial Action Coding System (FACS) for X3D HAnim models*
>> These plans are based on work adapting models by Dr. Myeong Won Lee, as
>> described in the README: Original FACS Models
>> <https://www.web3d.org/x3d/content/examples/HumanoidAnimation/FacialAnimation/originals/README.html>
>> .
>> *Lessons Learned, So Far*
>>
>> 1. Every head will likely have different geometry, textures, colors,
>> etc. There is no need for every head to be the same.
>> 2. A single head model file needs to align with HAnim joints and
>> segments, if it is going to be interoperable with other HAnim skeleton/skin
>> models.
>> 3. Each head model will connect to the skeleton as shown in
>>
>> Part 1: Humanoid animation (HAnim) architecture, Section 4 Concepts, 4.9.6.3 LOA-3 hierarchy <https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#Hierarchy3>
>> humanoid_root : sacrum
>> [...]
>> vc7 : c7
>> | vc6 : c6
>> | vc5 : c5
>> | vc4 : c4
>> | vc3 : c3
>> | vc2 : c2
>> | vc1 : c1
>> | *skullbase : skull*
>> | l_eyelid_joint : l_eyelid
>> | r_eyelid_joint : r_eyelid
>> | l_eyeball_joint : l_eyeball
>> | r_eyeball_joint : r_eyeball
>> | l_eyebrow_joint : l_eyebrow
>> | r_eyebrow_joint : r_eyebrow
>> | temporomandibular : jaw
>>
>> 4. Typically a new head will connect at *HAnimJoint skullbase* and *HAnimSegment
>> skull*, possibly including neck joints as well.
>> 5. Typically an interoperable HAnimHumanoid head will include
>> HAnimJoint and HAnimSegment definitions for eyelid, eyeball, eyebrow, and
>> jaw.
>> 6. The HAnim specification includes guidance on connecting partial
>> body models together in 4.8.6 Partial HAnim figures
>> <https://www.web3d.org/documents/specifications/19774/V2.0/Architecture/concepts.html#PartialHAnimFigures>
>> .
>> 7. No HAnimSite feature points are required in the HAnim
>> specification.
>> 8. We can add HAnimSite feature points to the above HAnimSegment
>> notes, without interfering with any of the native HAnim motion animations.
>> However adding HAnimSite locations to the other facial geometry many need
>> further attention. Adding a useful set of HAnimSite feature points for
>> facial animation might be very useful.
>>
>> *Next Steps*
>>
>> 1. We want head models that include the ability to use an FACS Action
>> Unit (AU) enumeration that shows a new expression.
>> 2. We currently have single expressions demonstrated in separate
>> model files. Showing one expression at a time is the initial goal for
>> facial expression animations.
>> 3. Integrating the animations together efficiently, within a single
>> X3D model file, is the path towards having interoperably animatable heads
>> that can move eyes/jaw and also show expressions.
>> 4. We first want single-AU responses working, and then multiple-AU
>> responses working, in order to achieve the full range of FACS expressions
>> that are possible.
>> 5. For full animation compatibility, head models need to include the
>> standard HAnim joints and segments for eyelids, eyeballs, eyebrows, and
>> jaw. This requirement deserves careful consideration when designing mesh
>> geometry.
>> 6. See the FACS Visual Guidebook
>> <https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system/> for
>> video examples with corresponding descriptions that illustrate the
>> expressions conveyed by each Action Unit (AU), either individually or in
>> combination.
>>
>> Building example models is the path towards showing repeatable
>> capabilities. Conceivably these capabilities might become a new HAnim node
>> for HAnimHumanoid. Successful, repeatable design patterns can then be
>> formalized in the future HAnim Specification.
>> =======================================
>>
>> Summary: interoperable facial animation behaviors are beginning to look
>> feasible and repeatable. More demonstration work awaits.
>>
>> Have fun with X3D Humanoid Animation (HAnim)! 🙂
>>
>> all the best, Don
>>
>> --
>>
>> Don Brutzman Naval Postgraduate School, Code USW/Br
>> brutzman at nps.edu
>>
>> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
>> +1.831.656.2149
>>
>> X3D graphics, virtual worlds, navy robotics
>> https://faculty.nps.edu/brutzman
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20250107/7ee6b68e/attachment-0001.html>
More information about the x3d-public
mailing list