Virtual worlds already seem very realistic. But the people moving in them not so much. Ingo Feldmann, Dr. Oliver Schreer and Peter Kauff from the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute HHI, have now developed a technology with which lifelike people can be integrated into virtual worlds. For this new technology, they have been awarded the Joseph von Fraunhofer Prize.
In movies, the movement of the camera is predetermined: While watching, you are following the camera. This, however, is different in a film that you can step into. Here the viewer is completely immersed in the film scene: When wearing VR glasses, viewers can let their gaze wander as they wish, dive into the action and circle around the actors. Nevertheless, no matter how realistic the surroundings appear in the virtual world, the people still appear artificial and their movements seem unnatural. Also texture – for example the way the fabric of the sweater is woven, beard stubbles or birthmarks on the skin – still is far away from reality. Until now.
Ingo Feldmann, Dr. Oliver Schreer and Peter Kauff from Fraunhofer HHI have developed a novel technology that solves these issues. "We are able to integrate real people into the VR environment who then appear lifelike and move in the artificial world just as smoothly and naturally as they would in the real world," says Ingo Feldmann, group leader at Fraunhofer HHI. "The texture also appears lifelike to the viewer. Ingo Feldmann, Dr. Oliver Schreer and Peter Kauff have been awarded this year's Joseph von Fraunhofer Prize for the development of the "3D Human Body Reconstruction" technology used here.
The trick: The researchers film the actors with 32 cameras, each camera with ten times HD, which are combined into pairs and optimally distributed in the room. The cameras record up to 30 images per second and from this data a software calculates the corresponding depth information. In other words, it calculates how far away each part of the body is from the camera on the respective shot. It then fuses the data of the individual camera pairs with each other. The result is a lifelike three-dimensional image of the person and their movements. This three-dimensional model is then directly integrated into the virtual world, with all the movements the person performs. Until now, people were scanned statically and their movements were then animated on the computer.
In October 2017, researchers from Fraunhofer HHI built a first prototype to optimally "capture" people and their movements. In the summer of 2018, they also set up a studio at the Babelsberg films studios. Here, external clients can have images digitized and can also book additional services. The production company Volucap GmbH was founded in conjuction with Studio Babelsberg, ARRI, Interlake and UFA.
The researchers at Fraunhofer HHI also want to use the technology in other areas in the future. Possible scenario: In the case of difficult operations, an expert could call in VR, see the patient in real time and provide assistance. Also, a popular soccer player could visit a fan at home before a Champions League game. "Essentially, our technology can be used wherever authenticity is needed – for example, people from society and media, but also for training and further education," explains Peter Kauff, who has an eye on the commercial exploitation of the technology. His colleague Oliver Schreer, who developed the algorithms together with Ingo Feldmann, adds: "Our technology opens a door to a huge new field of application – it remains to be seen exactly where it will go."
Fraunhofer-Gesellschaft