Request to Enhance Unreal Metahuman with AI
Imagine being able to generate facial animation from any video recorded on any device, whether it’s an Android, an iPhone, a camera, a webcam, or extracted from any movie, TV show, or conference from any era, and bring it to your Unreal Metahuman! It would be huge, right?
I made a video to showcase this concept, but the model I used didn’t give me the expected result, Oh my god it looks awful. I’m really sorry !! But I know for sure that the model works. As you can see, with AI, it’s already possible to track facial movements and transfer them onto a 2D image. What would be even more amazing is being able to transfer this data onto the bones of your Unreal Metahuman. This would allow thousands and thousands of artists around the world to create awesome games and films without the need for sophisticated camera setups, helmets, or complicated workflows.
Body motion to metahuman isalready possible with Rococo Studio, but skeleton dosn’t match Unreal’s one and need o lot of actions to make it work. Also ,facial motion capture from a video to a 3D model has been done in the past by Avatary, but the solution isn’t updated with Unreal Engine 5 and only works with Maya. I think it is interesting to bring these technology directly to the Metahuman interface, in addition to the new Metahuman Animator solution. So, I invite the developers of Unreal to try to train a model for video driven motion to Metahuman. If not, we can still organize ourselves and focus our efforts to create a plugin that can be integrated with Metahuman, just like the “Voice to Facial Rig” plugin, the Metahuman SDK, which I’m currently using to animate my mouth.
I believe we can succeed in doing it. We can meet in the Facebook group ‘Unreal Engine Developers Community’ and discuss together solutions to make it happen. Personally, I don’t have any knowledge in machine learning or deep learning programming, but I am willing to help with simple tasks like preparing dataset of reference videos or facial rig poses. We can also try to contact the developers of this model to see if we can apply it to the Metahuman rig.
Come on, guys, we can really do it .. because if we don’t, someone else will. I mean, an engine that starts with U and ends with Y. Okay, guys, see you later. What a time to be alive!