FaceCue - Real Articulation Lip Sync & Facial Performance Studio
Firecrest Roost
$77.00
$154.00
50%OFF
(no ratings)
Jump AssetStore
Drives the whole face from your dialogue: mouth, eyes, emotion, head, breath. Bake offline or run live. Ten languages tuned, fifty more supported, text to speech built in.FaceCue is a complete facial performance system for Unity. It drives the whole face from your dialogue: the mouth, the eyes, the emotion, the head, the breath. Every part is optional, so you can take the lip sync on its own or the whole performance.It bakes cues, not animation. A bake writes down what was said and how it was said, like sheet music, and each character performs it live, forming every shape by the same linguistic rules real speech obeys. Retune a character, even during play, and every line you have already baked changes delivery style, with no re-bake.It drives live when the audio cannot be baked.FaceCue was built for standalone VR narrative work, where a character's face sits an arm's length from the player and everything has to run on a mobile chip. That combination is unforgiving. It wants detail you can inspect from inches away, and it wants it cheap.FaceCue is built for high fidelity and low cost at the same time, with intense optimizations, and what is left runs mostly in the background. That leaves as much of your CPU time as possible for other game tasks.Baking runs in the Editor, only on Windows. Baked Performances and Live Amplitude play wherever Unity does while Live Viseme runs on most platforms. Unity 2021.3 and above are supported, Unity 6.6 included.See Technical Details Below.DOCUMENTATION | ROADMAP | FORUM THREAD | DISCORDVIDEO | INTERACTIVE WEB DEMO | INTERACTIVE PC DEMO | INTERACTIVE MOBILE DEMOHow Does Lip Sync Work?Baked Speech.Bake a clip once in the editor and it plays with no runtime analysis cost.What was said is settled once, from the whole recording, and every character performs it in its own way, down to the small variations that keep a repeated line from looking copied. The baker also picks out laughs, hums, and other sounds, so they play as what they are rather than read as words.Live Speech.Drive straight from audio with nothing prepared: a clip playing now, a microphone, player voice chat, speech generated at runtime.Two live modes ship.Live Viseme reads visemes from the audio and puts them through the same shaping the baked paths use. The recognizer is less accurate than the baked one, but fast enough to run live, and it runs in the background.Live Amplitude follows loudness alone, the lightest option and the classic approach to live lip sync. It is tuned so out of the box it reads as close to a baked mouth as loudness can manage.Live is a safety net for anything unbaked, for characters in the distance, and the answer for audio that can never be baked.Real Articulation.Sounds are shaped by the sounds around them, like real speech, instead of stepping between poses. That is what separates FaceCue from amplitude lip sync, systems that go straight to visemes, and the reason the language tuning matters.Blending alone would dilute the sounds that carry the words, so a layer of linguistic rules sits over the top, protecting what has to land and easing back what should not show as much. Those rules are tuned per language, because languages do not behave the same way.Those rules apply mostly to baked lines, and to the live modes where possible.Between the Words.A real mouth is almost never shut, and never slack. FaceCue keeps it alive through the pauses, after the last word, and before the first. The pose it rests in between lines is yours to author, and one of the most personal things about a face.What Languages Are Supported?Accurate on 10 Languages, Custom-Tuned.English, Greek, Spanish, German, Mandarin Chinese, French, Japanese, Russian, Korean and Brazilian Portuguese. Each has its own recognizer, its own refinements, and extensive testing behind it, and where it would make a difference, you can pick the dialect.Multilingual Support Over 50 More.A multilingual decoder that works with or without a transcript, and no per-language setup. It relies on the same technology and tuning as the 10, derived from them rather than tuned language by language, so results should be close.Any Language At All.The universal path reads the audio itself and produces visemes directly rather than by a more linguistically accurate route. Results vary widely by language, but any language should give a usable mouth. It is the same technology behind one of the live paths, with added advantages.What Makes FaceCue A Performance Studio?Emphasis.FaceCue measures the qualities of the voice and calculates which words and phrases are being emphasised, then carries that continuously through the head and the brows, so the performance moves with the delivery instead of sitting still underneath it.Every drive path includes it, most precisely on the paths that know where the words are, and still from the voice alone on the universal one.Emotion.FaceCue ships 28 named emotions, each one already custom tuned with their own weights and timing.On a clip, author it two ways. Shape the ramps in the Composer, previewing live, or write it into the transcript and let the baker place it. "This is a great place to live!" gives the same result, which you can then tune.Emotion can also come from a Timeline track, from your own code, or as a lingering mood.Gaze.Two models ship.Biomechanical is built from the research on how people look. It separates what a character believes about a target from what its body can do about it, and the belief holds when the target cannot be seen, so looking away does not erase it. A target that slips behind a wall is followed along its path and picked up again when it comes back into view. Hold a look at the edge of comfort and the head tires and eases back.Simple aims the chain straight at the target, with limits on each joint and smoothing over the result. It is a conventional approach. It's not as realistic, but it is familiar and predictable, and sometimes that is what you want.You choose how much of the body either one owns, from the eyes alone to the full chain through the torso.Eye Fidelity and Head Motion.A resting face is never still. The eyes drift, and make two kinds of small movement: the ones that hold a target and the ones that look elsewhere. Lids lag behind fast vertical moves, blinks keep their own schedule, the head carries a slow idle sway and shifts posture while the eyes hold their target, and rights itself against a leaning body.None of it runs in isolation. These behaviours affect each other, rather than being independent noise layered over a still face. The world reaches the face too: the pupils answer the light, a character thinking hard blinks less and looks away briefly, and one approached too closely leans back to make room.Breath.An autonomous breath cycle shaped by speech and exertion. Speech lands on exhale, so a character runs low while talking and refills after.IK Compensation.A look that turns the body would drag a reaching hand off its mark, so FaceCue can correct the hand or any other bone, as the body moves.How Do I Start With FaceCue?Start Simple.Controls are grouped into Basic, Advanced and Expert, and the last two stay hidden until you ask. One selector in the Hub moves the whole tool at once, so the surface matches how deep you want to go. Expert values are tuned already and rarely need touching. They are exposed anyway, because a system you cannot open is a system you cannot fix.Preview with the Composer.Traditional lip sync is a sealed box between audio and blendshapes. The Composer opens it. Every stage of a clip is laid out, from the baked data through to the final shapes on the rig, worked out by the same code that plays it. Scrub it both ways, edit cues in place, undo anything. It saves as you go. Assign a character and preview it all in edit mode.Face Rig Designer.Every character drives its shapes through a map, one per channel: Speech, Prosody, Emotion and Eyes. A map is a set of recipes, and a recipe names which shapes make up one target, and how much of each. The “ah” mouth shape is a recipe. So is a blink.In the designer you edit and tune them directly, previewing live on the assigned character. Matching shapes are found across every renderer, so a beard, the brows and the eyelashes move with the face. Maps are assets, and a tuned one travels across a family.OneClick Wizard.Drop a character in and step through it. FaceCue detects which family the rig belongs to, then builds every map for you. Each family carries hand-authored, tuned recipes rather than a generic mapping onto whatever shapes the rig ships with. For a supported rig you can usually accept each step and be done.Families Supported: MetaHuman (DNA-263 & ARKit-52), Character Creator and iClone (CC3, CC4 and CC5: Traditional, Standard, Extended and HD), Daz Genesis (9, 8.1, 8 and 3), Ready Player Me, Microsoft Rocketbox, Synty Sidekicks and VRoid, plus Generic ARKit. Anything else maps by hand. Feel free to request more via support.What Else Can FaceCue Do?Text to Speech & Voice Changer.23 languages, create custom voices, all inside the editor with local models. No audio? Synthesize a clip from your transcript. No transcript? Generate one from the audio.Timeline.Phoneme, emotion cue, gaze and cognitive tracks, plus a watcher that drives the face from a Timeline audio track, nested Timelines included.Performance Dials, Per Character.FaceCue does the work that would change the face, and little more. The rate dials reduce the cost further. Set how often each character is calculated and written, independently, and Display Smoothing covers the gaps, so you're not paying for every frame. You can run a hero at full quality and a crowd at a fraction in the same scene.What is left of the cost is calculated in the background: on worker threads, in parallel with the main thread, and written to the rig just before it is drawn, so the main thread is free for other work. It holds up on standalone VR, often better than other platforms.Unity & ProjectUnity 2021.3 through Unity 6.6, for both the runtime and the baker. The floor is a compile requirement, not a recommendation: FaceCue's editor code is written against C# 9, which Unity supports from 2021.2 onward, and 2021.3 is the Long Term Support release in that line. On anything older the package fails to build.All three render pipelines. FaceCue drives blendshapes and bones, which are pipeline-agnostic, so Built-in, URP and HDRP are all fine, and so is a project that switches between them.Both scripting backends, Mono and IL2CPP, including the targets that offer nothing but IL2CPP.FaceCue adds no package dependencies. Timeline and Addressables are used if you already have them, and ignored if you do not.Tested on 2021 LTS, 2022 LTS, 2023, Unity 6.3, Unity 6.6, on Windows, and macOSMulti-Threading & OptimizationsEvery character is calculated on worker threads by default. Where a platform has no worker threads, WebGL among them, the character runs on the main thread instead. Nothing is lost but the saving. You can always turn workers off per character.A few rare conditions keep a character on the main thread on purpose: a shared random seed or a diagnostic listening to it. The brain’s inspector names the reason for each character.Running on workers doesn't mean that scripts of your own cannot read or write the same bones and blendshapes. FaceCue has documented best practices and alternatives on the documentation page "Script Coexistence".Baked PerformancesBaked Performances carry NO PLATFORM LIMITATION: Windows, macOS, Linux, iOS, Android, Quest and WebGL and anywhere else Unity works.Cue files load through the built-in registry, or through Addressables if you use it.Stress tested on Windows, Android, Quest 2 and 3, WebGL, iOS, and macOS on Apple silicon, both natively and as an Intel build under Rosetta.Live PerformancesLive Amplitude carries NO PLATFORM LIMITATION: Windows, macOS, Linux, iOS, Android, Quest and WebGL and anywhere else Unity works.Live Viseme runs on 64-bit Windows, Intel Macs, Android (Quest included) and iOS.It uses Meta's OVRLipSync, which is free and ships inside the package.It is not available on Linux or WebGL. The recognizer is a native plugin and there is no binary for either.On a Mac with Apple silicon, it needs two things: Unity running under Rosetta (the editor or your build), and Rosetta set to report the instructions the recognizer uses, which it does not do by default. Without both, it falls back. See Requirements.A Windows build that uses Live Viseme requires the Microsoft Visual C++ Redistributable (2015 to 2022, x64) on the player's PC. A Unity build does not include it, so ship it with your installer or through your store's redistributable step.Both Live paths need readable audio: a clip imported Decompress On Load, or Compressed In Memory with PCM compression. A Streaming clip can never be read.Microphone input is not available on WebGL, because the platform provides no microphone API. Bring a feed of your own into a clip and the loudness path should run on it, since that path reads the audio it is given.Wherever a live mode cannot run, FaceCue moves the character down its Fallback Chain rather than stopping, and the Console says what is missing.Stress tested on Windows, Android, Quest 2 and 3, WebGL, iOS, and macOS on Apple silicon, both natively and as an Intel build under Rosetta.BakingBaking runs in the Unity EDITOR ONLY, on WINDOWS ONLY. It is never in a player build and never runs at play time. Other desktop platforms are on the roadmap.The English recognizer ships inside the package, so English works out of the box with no downloads, with or without a transcript.Every other model, including the speech recognizers, the text to speech, the voice changer and the transcript generator, installs on demand at no cost with a single click in the editor, so you only download what you use. They run locally on your machine. They are hosted on GitHub, and if you prefer you can download them by hand and point FaceCue at the folder through the setup menu.The baker supports CPU and GPU baking. Both GPU backends ship with the package: DirectML uses any modern graphics card, and CUDA uses an NVIDIA one and is considerably faster.One GPU backend is active at a time. FaceCue ships with both switched off, so an install bakes on CPU until you choose otherwise, and you pick one in the baker's setup window. Switching restarts the editor, because the backend binds once as the editor starts. If a backend is ever left in a bad state, both are disabled on the next load and baking falls back to CPU.Third-Party ComponentsFaceCue bundles various third-party tools and libraries. The third-party components included in this package are used under these licenses: MIT, BSD-3-Clause, Microsoft Public License (MS-PL), SIL Open Font License 1.1, Oculus Audio SDK License 3.3, and the Microsoft DirectML and NVIDIA CUDA and cuDNN redistribution terms. Each is listed with its license in Third-Party Notices.txt in the package.The creation of icons and other UI elements in the asset's editor windows and demos was assisted by ChatGPT (OpenAI), and the results were edited afterwards.The Timeline Demo's narration audio was generated with FaceCue's own text-to-speech.


