A Guide To Mixed Reality Capture on Oculus Rift
How to capture footage from a VR experience while also including footage of the real world.
With Oculus recently introducing the ability to natively capture Mixed Reality (MR) footage, VRFocus thought that we would be helpful and provide a guide in how to set yourself up to capture such footage. Mixed reality footage means that viewers can see the person playing alongside the virtual reality (VR) environment, giving viewers a better idea of what is happening in the VR experience.
Before You Start
Its important to verify first of all that you have a PC and camera that meet the minimum requirements for MR capture. MR capture creates a greater demand on a system than simply running a VR application on a PC, so users who find that they are struggling with standard VR applications may wish to consider an upgrade before attempting MR capture. A list of recommended PC specs from Oculus can be found here.
The first thing you will need to do is decide if you are using a fixed or dynamic camera. Static or fixed cameras are easier to set up and calibrate, but only give you a single view of a scene. A dynamic camera is one that has been associated with a particular VR object and allows you to move the camera around. This of course means that if you are planning to use a dynamic camera, you will need to create the VR Object, usually a Touch controller, to associate the camera to.
If you are using a static camera, this step can be skipped.
Then the Camera needs to be calibrated to capture images in the real world. You can view the diagram below for positioning information for your sensors, camera and Green Screen. You will then need to download 1) The calibration image and 2) The Oculus CameraTool from the Mixed Reality Tools section of the Oculus Downloads page.
Calibrating Your Camera
Once you have everything in the optimum position, launch the Cameratool. Select ‘Calibrate Camera’. If you are using a Dynamic camera, make sure the VR object is active and in view of the sensors. Select your camera from the drop-down menu and enter an identifier for this configuration. You will then need to enter your camera’s resolution, this information can usually be found on the box your camera came in, or from the manufacturer’s website.
Select ‘Start Intrinsic Calibration and align the calibration print-out with the box placed on screen, once this is aligned press the X or A button on the Touch controller. The tool will then use this to remove any distortion. You are aiming for a result value of less than 1. Anything higher and its a good idea to re-run the calibration.
If you are using a Dynamic camera, you will need to select what object will be tracked as the camera at this point.
You will then need to position your Rift in in the boxes that will appear on-screen. You’ll need to adjust the distance from the headset to the camera in order to align this correctly. When you’ve lined it up, again press X or A on the Touch controller.
Finally, virtual controllers will be overlaid over your Touch controllers to show the calibration results, you can use sliders to adjust this. It’s best to keep the controllers totally still for this step.
Once all this is completed, you can save this configuration to a file or the system.
There are two ways that apps can render MR, either the app handles all the compositing, green screen clipping and other rendering internally, or it can use a 3rd-party application such as OBS Studio or Xsplit to combine the various elements. For this guide, we will be using OBS Studio as an example.
First, launch the app in Mixed Reality Mode. Then launch OBS studio and create your first source. This will be your background, so you will need to select ‘Window Capture’ to grab the Window that contains the scene in VR. Some apps also capture a foreground, you will need to add a filter to select the foreground. Fill the background and foreground to the whole window by selecting Transformer – Stretch.
Create another source. This will be your external camera. Apply the same transform that you did to the foreground and background layers.
Add a chroma key effect to remove your Green Screen and adjust the scenes as required. The order should go foreground, external camera images, background.
Once all that is all done, you will be ready to capture your Mixed Reality Experience.