Creating Custom FaceFX using FaceFX Editor

Mass Effect 3's FaceFX are exactly what they sound like they are - facial animation effects. These control lip-sync and expression information for the character named in a given Dialogue Node. FaceFX work in conjunction with gestures and other types of animation, but as you might expect, control only the character's face.

This tutorial covers:

• The basic principles of animation necessary to understand the FaceFX window
• Adding a blink animation to a line
• How to use FaceFX to prevent the clipping of dialogue that has been overwritten or redirected
• Best practises for using FaceFX to create entirely bespoke facial effects
• Importing FaceFX sets from one node to another, in whole or in part

This tutorial assumes basic familiarity with Dialogue Editor. If it is unfamiliar to you, you may wish to visit the Dialogue Editor overview tutorial first.

1. What am I Looking at, Here?

Use the Dialogue Editor to open the package file containing the conversation you are modding. Open the conversation containing the line you wish to edit the FaceFX for. Click the node, and on the right-hand side under its String Reference ID, navigate to the Assets panel.

Figure 1

This particular node is a little bit different. Those familiar with the Dialogue Editor will notice straight away that although this is a Reply Node, and thus spoken by the player character, there is no associated Wwisestream asset for a male character. That is because this node can only ever be accessed by a female Shepard.

Both the Male and Female sections have their own sets of FaceFX.

Mark Meer and Jennifer Hale's line readings can differ in terms of timing, even on the same set of words, and so one set of animations may not work for both. They may need to be edited independently.
For non-player characters, both sets may also need to be edited independently.
In all cases regardless of an NPC's gender, NPC audio is stored in the Male section of the dialogue node, and it is usually the male set of FaceFX used. There are exceptions to this.

Be Aware: Even when no conditionals are checked, when the Listener of a conversation node is the player, and the player is female, NPCs may use audio assets from the male section of the dialogue node, yet use the female FaceFX when speaking. Be sure to check what the Listener is.

Fortunately, there is a process for exporting and importing FaceFX animation. More on that later.

In the Female section, click the button with the tragedy and comedy masks on it to bring up our friend, the FaceFX Editor.

Figure 2

This is the anatomy of the FaceFX Editor.

Under File, it shows us the conversation currently loaded, and what character it belongs to.
Under Lines, it shows us the FaceFX from within this conversation. Directly beneath that panel is a pane that shows us details about this line's audio. Because we opened FaceFX Editor through a dialogue node, the line we want to edit is already selected for us.
The Animations column lists a lot of different things, but more on that in a moment.
The large grey grid is our Timeline. The X axis shows us a selected animation's weight value. The Y axis shows us the progression of time. Underneath the Timeline, there is a subtitle to remind us of what's being said, although if you are working on a node where you have overwritten the audio, this is probably inaccurate.

2. Phonemes and Controls

The Animations column looks very similar across all FaceFX sets. The first 15 items in the list are called phonemes in animation terms. These correspond to the shapes human mouths make when they speak. Not every letter appears on the list, and this is because many sounds share the same shape. Humans and asari share phoneme sets. Turians, krogan and salarians all have their own. For the purposes of this tutorial, we will focus only on human phonemes, which all begin with m_. Most of these are fairly self-evident as to what they control, however note that plosives, that is p and b sounds, are controlled by combinations of m_M and m_Flap, and sounds like 'uh' are controlled by m_Open. m_Jaw+ and m_Jaw- are special cases - these control opening and shutting the jaw, respectively, and do nothing else to the mouth. Because of the way animation works, most jaw operations can be done solely with m_Jaw+; its opposite is rarely used. Facial animations are created by combining these effects.

Figure 3

Clicking through the list will show patterns of dots and lines. These dots are called keys, and the lines between them are called tangents. Animations have minimum and maximum weight values. A weight value of '0' describes the shape not being present on the face at all, and 1 plays it at its maximum value. So, with something seen here like m_Jaw+, '0' means the jaw is completely closed. '1' means the jaw is wrenched all the way open, and is usually not a great value to input if you want naturalistic looking animation.

Some animations, such as Orientation_Head_Pitch, Orientation_Head_Roll, and Orientation_Head_Yaw, can go through 0 and into negative numbers. This means that the animation is playing in the "opposite direction." So, Orientation_Head_Yaw at 0 will be looking straight on. At 1, it will be to the right. At -1, it will be to the left.

Not all animations can or should be used with negative values, and an animation's first key must always be at 0.

Each key describes how strongly this shape should be present on the face at any given point in time. The tangent shows how the shape will "move" to meet where it should be at the next key. This known as in-betweening, or tweening in animation terms. Smoother curves are slower, gentler movements and sharper ones are quicker and more dramatic. Here we see m_Jaw+'s movements, which are usually fairly sharp, but subtle. In fact, throughout this line, the strongest weight it ever has is 0.40, which corresponds roughly with the start of the word "one" in this line.

Tips on Effective Use

• Different phonemes have different relative strengths. For example, m_Jaw+ is a very strong shape, and in most cases, values do not exceed 0.40 in regular speech. Once a character is speaking, they usually keep their mouth open a little bit until they're totally done. m_Jaw+ is usually at a value of around 0.15 to 0.20 in between words. It is relatively rare that their jaw returns to 0 until the sentence or sequence is complete.
• By contrast, Eyebrow_Raise is very weak, and to produce noticeable effects, values of whole numbers are often required. To make a human visibly raise their eyebrows as if in surprise, typical values are between 4 and 7.
• Shapes like m_M, m_FV, and m_N generally require values close to 1 to be effective. Because these shapes take time to form on the mouth, try to give them "space to breathe." Animating lip-sync for Mass Effect is less about strict accuracy, and more about the effective "landmarking" of shapes in concert with sounds.
• Use m_OW at a high value, usually around 0.70 to 0.80, for "sh" sounds, and m_ZZ at around 0.40 to 0.50 for soft "s" sounds.
• Blink is another strong shape. There isn't a need to go over a value of 1. At 0, the eyes will be fully open, and at 1, fully closed. For characters with unusually widely open eyes (such as Joker) try setting the Blink track to a value of 0.12 to 0.15 or so throughout the line for a more natural look.

Timeline Display Behaviour

Continuing to click through this list will show that some things only have one or two movements, and their timeline seems to stop much sooner than the end of the audio. Things may be keyed to remain at a value of 0, or something very subtle. This is because the effects of all of these work in conjunction with one another, and they stack. You will note that the timeline resizes as you click through. This is because in its default mode, the timeline shows you only the selected animation's keys up until the last one keyed in the sequence.

Figure 4

In the top right of the grey graph area, you will note a checkbox labelled "Fixed Time Span." When ticked, this locks the display to view a defined span of time for all phoneme tracks, regardless of keyed information. This allows you to isolate focus on a word or sentence, or view the entire line at a glance. Clicking through different tracks shows things in their relevant and proportional place.

Figure 5

Right-clicking anywhere in the Timeline gives these options.

• Add Key will place a key at exactly where your cursor is, both time and weight-wise. This can be adjusted after the fact.
• Add Key with 0 Weight places a key at the point in time where your cursor is, but at a weight value of exactly 0. Excellent for quickly establishing beginning and end points for peaks.
• Offset All Keys After This Point is used to push and pull segments of keys back and forth in time by defined amounts. Entering negative numbers will pull things back, positive pushes forward. It affects all keys past your cursor in a given track.

Key Slide

• To slide a key up or down in weight value (vertical) left-click it and drag up or down with the mouse.
• To slide a key up or down in time (horizontal) hold SHIFT, left-click and drag.
• To slide a key in both directions at once, hold CTRL, left-click and drag.

Reference Curve

Right-clicking on any phoneme or gesture track in the animations column gives these options, one of which is Set as Reference Curve. This makes any track appear as a grey curve behind whatever track is currently being edited. This allows easy synchronisation of phonemes and gestures.

3. Putting it Into Practise: Adding in a Very Simple Animation

We are going to make a character blink during a line.

Right-click, Add Key with 0 Weight. Right-click the key itself.

• Set Time moves the key to a point in time.
• Set Value sets it weight value.
• Break Tangents is usually used to fix weird curves after deleting keys.
• Delete Key does what's on the tin.
• Flatten Tangents is a contextually available option that creates a straight line from one key across to another. This option usually appears when a tangent does something it shouldn't and takes on a value it should not have. If your tangents look weird, see if this option is available.

Note: Keys depend on what is ahead of them. You cannot move a key ahead in time if it overtakes another in front of it. You must delete the key, and then place another ahead, in the desired position.

Set your key's time to 0. If you have not selected a Fixed Time Span range, the screen will update and the grid will change. At about 0.50 in the timeline, Add Key with 0 Weight again. This defines the beginning and end of the animation. Now, add a third key as close to in the middle of them as you can, and set its value to 1 - this means that over the course of a half second, the character will close their eyes, then open them again. Your completed keys should look something like this:

Figure 6

But, the timeline shows this animation starts at 0 and completes at 0.50 - blink and you'll miss it. Right-click before the first key, and choose Offset All Keys After This Point. Bump it up to somewhere else in the line. Upon confirming, FaceFX Editor will update. It's as simple as that for a blink! Setting keys at different values will make the character hold their eyelids lower or higher, useful for different expressions. Change the length of the blink to make it something slow and owlish, or something quick and fluttery.

4. Using FaceFX Editor to Fix Clipping Audio

FaceFX has a close relationship with audio. As something of a failsafe, audio will stop playing if the animation sequence is complete and there are no more facial animations. This is meant to stop the game from giving dead air if there is a second or two of silence accidentally left on the end of an audio asset. However this can present problems for modders who have overwritten dialogue nodes with files significantly longer than their original settings. Correctly setting a dialogue node's InterpLength and its associated WwiseEvent's DurationMilliseconds values are very important to do first and are beyond the scope of this guide. However, once those steps are done, if audio still skips, adding FaceFX will solve the issue. Simply add keys to any animation in the FaceFX set and set its time to fit the DurationMilliseconds. Make sure its value is above 0, even if small. This works as a temporary measure to ensure the audio will play, but more work in FaceFX will probably be needed.

Deleting keys down to a certain point from all animations in a FaceFX set will cause an audio line to terminate early if needed.

5. Keying a Completely Custom Line of Dialogue

It's important to understand that whilst FaceFX is a powerful tool, it is limited, and its use comes with several drawbacks that are not present in conventional animation environments. Unlike the original animators, you will not be able to see your changes reflected live as you do them. You will not have access to a preview of the animation. You will be animating this line through a combination of guesswork and memory. There are, fortunately, a few tips tricks and tools to help make this process a little easier, although it can still be accomplished without them.

Suggested Programs:

• Audacity - If you are dealing with replacing and editing audio, you probably have this already. But, it's a free program for editing audio.
• OBS Broadcaster - For recording what things look like in-game. Also free. Pretty easy to set up.
• GIFCam - A fantastic little app for making GIFs quickly. It was used for making the GIFs in this tutorial, which were used as iterative reference.

Good Practise:

• Have a save file as close as possible to the conversation your line is editing.
• Have a backup of your mod before you start messing with FaceFX.
• Look around in dialogue nodes at the vanilla's FaceFX. Compare what keys look like, what values are usually used.

So, you have a modded Dialogue Node with fresh audio injected into it. It's a combination of lines from all over the place and doesn't have animation data you can steal easily from other places. You have checked that the line plays properly, but the FaceFX look weird, because of course they're set up for a different line. You are looking at the line's FaceFX not knowing where to start.
If I'm working from scratch, I tend to delete almost all the existing FaceFX information. Sometimes, I leave in things like blinks, eyebrow movements, or head movements if I think they work.
Because these need to be done one phoneme at a time, I like to start working with m_Jaw+. The reason for this is it usually has movements that go all the way through the line from beginning to end. It's something that makes an immediate and obvious difference. So I go ahead and delete what's there. There is an option to easily facilitate this, when working on human or asari lines. Right-clicking on your line in the "Lines" column gives this:

This does as described and clears away all keys in the m_* tracks only. Head movements, emotion sets and everything else will be left intact. Alternatively, by right clicking on any phoneme or gesture track in the animations column, Delete All Keys will clear just that track's data.

Open your line's audio in Audacity.

"Hehe. I already figured that out. It's not a big deal. Heh. I'll look forward to our next talk."

Note the peaks and valleys in the waveform. Usually, big peaks correspond to larger jaw movements. To get a good sense of what you need to do, try saying your line aloud along with your recording. Note the way your face moves. What's most useful about Audacity here is that it's easy to correlate the shape of a waveform with its timeline, and sync up what you're doing on yours in FaceFX.

(This line was animated in a previous version of FaceFX Editor where the colours were different.) These are all my m_Jaw+ keys, from beginning to end. By comparing the waveform to the keys, you will see a basic correlation. Ingame, this won't look like much yet, however:

First Iteration

This has a head gesture left in, and a few other phonemes at the very beginning that I want to work with. The rest is all the jaw. You can see there's a lot to be done.

Second Iteration

This is the same line, with just a few more phonemes added later on in the sequence, including the blink from earlier. It's beginning to look a lot more lively, like she's talking!

Final Iteration

The final line sequence. It can be seen in context with sound here.

Iteration is key when animating blind. When I make a few significant changes, I reapply my mod, load it up and record what's happening using OBS. Then, I look at my recording and use GIFCam to make a GIF of the line in its current state. I use this for visual reference in adding more changes or focussing on problem areas, because it's nice to have a looping example of what my changes have done right in front of me. ME3's phonemes are quite forgiving, so long as an effort is made to basically match sounds, it will look believable enough, or at least on par with the rest of ME3's lip-sync.

Exporting and Importing FaceFX Animation

You may wish to take entire portions of a different FaceFX set and import them into yours. Perhaps it has a word or expression you like. Or, once your animation is complete, you may need to duplicate it into another set. This is usually a relatively painless process.

Warning, these options deal with all information in a given set, not just the selected phoneme or gesture. Everything.

• Delete Section of line deletes all information from all phonemes and gestures in a set between defined points of time.
• Import Section of line takes .json file information and adds all keys from it to everything in the timeline between defined points.
• Export Section of line exports all information from all phonemes and gestures in a set between points of time. Its output is a .json file.
• Offset keys after time is like the function inside the editor, except it applies it to everything in the set at once.
• Clear all 'm_' Lip-Sync Keys as previously discussed, gets rid of human or asari lip-sync data in one go and leaves everything else as it is.
• Clone does what's on the tin, clones the FXA.
• Delete Line does what's on the tin, the nuclear option. Caution.

Best Practises

If your intent is to take information from one node to another to work on as a template, it is best to delete all unneeded information first, as importing a .json file in this manner is additive. If the timeline from your import overlaps with information you already have, the imported file will add its own keys to the ones you already have. Be aware of what keys you already have at what points in time, or you will end up needing to do extensive repair work.

If you just want one phoneme or gesture, open two instances of FaceFX. Find the set with the desired phoneme or gesture, and drag and drop it into your target set's animations list. It should populate at the bottom.

A Note on Expressions

By clicking around in dialogue nodes, you will notice that different FaceFX sets have different emotional values attached to them.
Things like E_S_Amusement, E_B_Anxiety, E_Y_Concern and so on.
What these do is change the way phonemes interact with each other to produce an overall expression, the general effect of which also has its own weight value that can be turned up or down for emphasis. You can even transition through emotions in a sentence in this way, by blending and fading weights across emotion sets. Try them out!