Real time performance in VR

Recently I find more and more people are trying to crack on with live performance in VR. As my knowledge of mocap developed through months, I can share some insight on what might be possible at the current date. At the beginning of this project, I was very excited to explore whether it was possible to stream live performance into VR, and to some degree it is possible. However what I’ve learned is that the issue of streaming live performance into VR is related to few things: the first is difficulty of joining face, body, hands motion capture in real-time, second is having a perfect mocap - often there will be some sort of jittering happening which requires clean up afterwards, the third processing power required to capture mocap and stream it in real-time.

Achieving high-level fidelity motion-capture performance is not easy and in order to get to perfect motion capture clean up is required. However, having said that I found that there are ways to work around it. A perfect example of it is VR performance of Tempest by Tender Claws. I was very intrigued by it as it’s been advertised as an immersive theatre with live interaction with an actor. The way that the show got around the motion capture issue is that it didn’t intend to use high fidelity movement. Instead, actors used tracking that is within the headset, so the headset can detect movement in space and hands movement and the voice can be directly streamed through the microphone in the headset. The avatars were simple but worked together with the rest of the environment. The audience had even simpler shadow like avatars with hands which allowed for interaction with the environment. The actor navigated the audience through the story and asked for participation in key parts where the audience either collected objects or enacted scenes by simple physical gestures. Bottom line is that this sort of interaction worked, it was real-time performance that didn’t have full motion capture but sometimes the simplest of solutions work best which the show proved. I’ve heard that similarly simplified avatars are used to design performances in Mozilla Hubs and VR chat. I’m currently exploring these so more to come on live events on VR social platforms.