With the Oculus Quest 2, Facebook's dual ambitions in virtual and augmented reality may meet in the middle. A new Quest developer toolkit enables the mixing of pass-through cameras with VR graphics, allowing for mixed reality.
On the Oculus Quest, mixed reality has been brewing for a while: Facebook has added a few modes where web browsing and typing on a Logitech keyboard work with pass-through camera data, besides the headset's way of scanning the world and painting play boundaries for VR using the black-and-white cameras on the headset. There's also hand-tracking.
The Pass-through API Experimental developer mode isn't yet intended for actual apps, but Facebook hopes to add mixed reality to Oculus Quest apps later this year. Work, social, and gaming are among the suggested applications. The ability to project avatars into the real world is like what Microsoft Mesh promises with HoloLens 2. With the Oculus Quest 2, however, it would be possible to do so on a $300 consumer headset. Another area where this could be very useful is in fitness games, where motion-intensive games like Beat Saber could be played while being layered over a pass-through view of the actual room — without the risk of accidentally punching the wall.
Facebook's post emphasizes that apps using pass-through camera data cannot access, view, or store images or videos captured by the Quest 2 cameras, a point of data privacy that the company also emphasized when announcing its VR ad push.
However, the Quest 2's cameras are low-resolution and black-and-white, making them unsuitable for capturing much real-world data or detail. This points to a future Quest upgrade with better cameras: Mark Zuckerberg has confirmed that a Quest Pro is in the works, which could push mixed reality even further. With Facebook's AR glasses plans still a long way off, the Quest appears to be a step toward figuring out how to use AR with existing apps.
You should also check out the following articles: