Using Artificial Intelligence in the various stages of making a VR Animated Film
This project is an exploration in using tools that use Artificial Intelligence in the process of VR 360 filmmaking using animation as the medium. The aim was to see how many AI tools could be used across the various steps in the animation pipeline so as to minimise human efforts as much as possible.
PRE-PRODUCTION
CONCEPT ART - MIDJOURNEY AI
Midjourney is an AI based algorithm much like Stable Diffusion or OpenAI Dall.E that uses word prompts and image references to generate high quality images in any required style be it photorealistic or artistic. This application was used to imagine character concept paintings of what popular fast food mascots would look like as real people.
CHARACTER DESIGN - METAHUMAN CREATOR
The character designs were based on the actual mascots of the different fast food brands. The models were made on MetaHuman Creator with the character concepts as reference.
PRODUCTION
MOTION CAPTURE - ROKOKO VIDEO
Developed by ROKOKO, a motion capture studio that specializes in mocap equipment, ROKOKO Video uses AI image processing to capture motion data from any video file and translate that into a .fbx or .bvh format which can further be used in 3D softwares for animating characters. all of the body animation in the film was entirely done using this method.
FACE CAPTURE - OMNIVERSE AUDIO2FACE
Omniverse Audio2Face is an AI powered animation tool being developed by Nvidia that uses deep neural net and machine learning to convert speech audio input to corresponding facial animation. It also allows for the control of the emotions to be assigned to the animatable character.
EDITING - AFTER EFFECTS VR COMP EDITOR
The rendered 360 frames from Unreal Engine were edited and composited in the VR Comp Editor inside of Adobe After Effects. This workflow enables us to work with 360-degree footage as regular footage. It also allows for view based editing that simulates the view of a VR headset giving more control of how we want our footage edited.
FINAL OUTPUT COMING SOON
