So let’s talk DIY face mocap. I was looking to record videos showing digital avatar that would use face mocap. I’ve explored a few options in terms of avatar and face mocap. My first tool to try was Unity as it’s most familiar to me. I found a workflow that used Unity SDK, an iPhone’s true depth camera and a Ready Player Me avatar. Using Ready Player Me allowed for a certain degree of customisation, you can choose from an already existing repository of bodies, clothes and features.
The other option I eventually decided to go with was using Replica AI, the solution has been presented to me by my collaborator Miguelangelo Rosario. He’s been experimenting with creating a different character for the project but as the results were quite good I decided to test this solution. First, we worked on getting the avatar polished, we used Reallusion software. Reallusion gives options for full customisation, options are endless, you can change body shape, materials, and literally everything about the avatar. Once the avatar was nailed we started to experiment with lip sync using Replica AI. In this solution, we didn't need to rig the avatar as Replica AI is a lip sync solution meaning it has a repository of lips shapes for words and creates facial movement and expression based on that. The only things left to do is decide on: voice, expression and checking that the text is understandable to AI. The first version of AI interpretation didn’t come out well at all, it was jumbling words together and some words were not clearly pronounced. Based on the results I’ve re-written the text to make sure that the words used can be understood and that there are no log sentences and that there is plenty of punctuation. It took a day of listening and revising the text but once completed the results were good. One thing to bear in mind is that as with any lip sync solution the expression can be a bit robotic, however for my project that was an added benefit. I keep hearing good things about metahumans in Unreal, however, these have a smaller degree of customisation and it’s pretty much impossible to make an alien character as they are all humanoid. Despite that I’m definitely up for trying face mocap with metahumans as they look amazing, will surely report on the results here ;)
0 Comments
|
AuthorI'm an actress, writer and producer. My mantra: write, perform, repeat. Archives
May 2023
Categories |