Volumetric video is regarded as the next important development step in the field of media production, especially in the context of the virtual reality (VR) and augmented reality (AR) industries. The researchers from Fraunhofer HHI have developed a new variant of this technology known as 3D Human Body Reconstruction (3DHBR). 3DHBR transmits the realistic image of a person into a virtual world.
3DHBR technology captures real persons with multiple cameras at the same time and creates naturally moving dynamic 3D models, which can be observed from within the virtual scene. The volumetric video capture system consists of 16 stereo cameras that create 3D information from all different viewpoints around the person, similar as humans perceive with their two eyes. This 3D information is fused and transformed to a consistent, natural and dynamic 3D representation of the person.
Post-processing modules adapt the data structures and formats and allow for direct integration in standardised post-production workflows and VR software. In contrast to classic animation of virtual characters, facial expressions as well as moving clothes are captured from video information and reconstructed at high geometrical detail and texture quality, thereby a naturalness of the visual impression is achieved.
The latest version of the technology features a complete workflow and is fully automatic. Due to the standardised output format, further post-processing of the dynamic 3D model is possible where previously the image that is produced had been far too large and complex for most software to handle.
Beyond the processing of dynamic 3D models, an integrated multi-camera and lighting system for full 360-degree acquisition of persons has been developed. The system supports diffuse lighting from any direction, automatic keying and flexible multi-camera arrangement. 3DHBR does not require a green screen, and thus provision of diffuse lighting from all directions can offer the best possible conditions for re-lighting of the dynamic 3D models at the design stage of the VR experience.
The 3DHBR technology was used for the first time in cooperation with UFA GmbH in the course of a joint test production for the immersive film Gateway to Infinity as a volumetric VR experience. In this test production, set in a science fiction world on a foreign planet, the 3D models were created based on real actors so that gestures, facial expressions and textures (skin, hair and fabric) could be recorded in detail. The 3D models of people were subsequently integrated into a virtual scene, and viewers with VR head-mounted displays (HMDs) were immersed directly in the scene, experiencing the story close up and personally.
The next steps for Fraunhofer’s 3DHBR technology are not yet known, however the company is showing interest in expanding its portfolio of VR experiences. When the technology will become available to be licensed to development studios working within the VR medium is not currently known, but VRFocus will keep you updated with all the latest details.