PeCap W3 '24: Mouth in Faceware Analyzer

In our lecture this week we were taught about FACS aka the Face Action Coding System. FACS is a taxonomy that is used to break down the parts of the face when analysing motion capture footage. The system was created by Paul Ekman. 

Examples of FACS:

Then I learnt that we can use blend shapes and joint controls to manipulate areas of the digital character that correlate to FACS. It is also best to have a blend shape that aligns with each action code. This way we can work in isolated zones (for example, the left corner of the lip) in order to preserve as much of the original actors performance as possible.

In addition to FACS, we looked at which areas of the face are the most significant in terms of strength and deformation. This information will be useful for when I need to animate movements in areas left untracked, like the cheeks and the forehead.

Week 3 has been devoted to the mouth. The most frustrating part has been dealing with the pesky lip balm. The lip balm covers the markers so the software loses track and gets confused. I did my best to key the markers around the area where the lip balm made contact with the lip, leaving out the markers that were covered.

When training and tracking, the lip balm caused a lot of jitter, in some cases falling off the lips altogether. To combat this, in addition to the key movements of the lips I also keyed the moments where the lip balm contacts the lips and breaks contact.

I got some helpful feedback from Dr Kennedy on my progress so far.

He mentioned

• track a strong eyelash instead of the eyelid marker (because the proportion of the eye closing is more precise by tracking the eyelash.

• try the best I can to match the markers around the mouth when the lipbalm is on the lips

• make sure the upper inner lips don't fall off the lip and onto the upper teeth.

All in all, the lips were the most challenging part to analyze and I have a feeling that they will continue to challenge me throughout this project. I ended up with 33 key frames where I can now say I am happy with the tracking for the most part. The mid section is pretty perfect where she's speaking which leads me to believe I have done an acceptable job. However once that pesky lip balm reenters the mouth area again, it's very janky. Hopefully I can get it sorted with lots of manual keying in Maya!