In an exclusive interview, Zuckerberg explains the path from VR to AR, and why Facebook’s doubling down on its popular Quest headset.
I am led down a staircase by a small team of Facebook communications people, into a high-ceilinged, wide-open office in Facebook’s Building MPK 21 in the heart of Silicon Valley. Open desks cut across the space. The air is calm. The room is silent. At one end of a long open-office desk, back to me, sits someone who looks like Mark Zuckerberg.
In fact, it is Mark Zuckerberg.
We head into a glass-walled meeting room, and I take my seat on a sofa in Zuckerberg’s private conference room. The CEO of Facebook walks in, shakes my hand and sits down on a chair a few feet away from me. Zuckerberg looks more relaxed in person than I expect. He reminds me of friends I’ve known who’ve led startups. I expected something unusual. Instead, our talk feels disarmingly normal. I don’t remember what shirt he wore. It seemed nondescript and comfortable.
“The [privacy] threats evolve & you need to work on them. But I would hope by the time these ecosystems are mature, our approaches to those issues will also be quite mature.” – Mark Zuckerberg said.
You’ll have to take my word for all of this, because Facebook didn’t let me take any photos or videos of our conversation – even after I offered to post it all on Facebook.
I’m here to listen to Zuckerberg’s pitch for Oculus, the virtual reality company he bought in 2014 for a then eye-popping $2 billion as part of his plan to expand Facebook’s social media reach into the interactive virtual world and help fulfill his mission of “helping people come together.” Oculus is promising “a new chapter in AR and VR” this year, months after the VR company’s original founders exited along with a string of other high-profile execs whose shops had been bought by Facebook. Zuckerberg gets philosophical within the first minute of our 30-minute conversation: “There’s a lot of questions that people have about where’s this all going, why is Facebook doing it.”
I expected news of a futuristic augmented reality headset, similar to the Microsoft’s HoloLens 2 and Magic Leap gear I’ve tried out. Those visits had a familiar routine: get invited to headquarters, tour the premises and experience something new. My trip to Facebook on Monday, two days ahead of Oculus Connect, followed a similar pattern. But without the new-product part.
Instead of wowing us with a new AR headset, Zuckerberg used his keynote address at the developer conference Wednesday morning to talk about VR, and in particular, the far more affordable VR hardware Oculus already sells. Because, while Facebook doesn’t have an AR headset ready yet, most people don’t even know what AR is, let alone want to buy a $2,000-plus headset. It’s hard enough getting people to try VR. Which is why the standalone $399 Oculus Quest might still be Facebook’s most accessible immersive product for at least the next year.
The Oculus Quest is “the best thing I’ve tried this year,” I wrote in my review. CNET awarded it our prestigious Editors’ Choice award because it’s an affordable, wire-free VR headset that’s accessible to everyone.
Zuckerberg’s team told me the Oculus Quest is getting key updates in the next year: It’ll work with PCs to play content on Oculus’ PC-connected Rift platform using a USB-C cable, using new technology called Oculus Link. I got a firsthand try at hand tracking, which is coming “early next year” and will enable people to try VR without learning how to use the standard controllers. And a new social platform in beta next year, called Facebook Horizon, is aiming to be the next attempt at bringing multiplayer experiences into Oculus, with games and creative tools.
Zuckerberg is pretty clear that Facebook’s VR vision is still coming into focus, even after all these years. He gives me the standard caveats about the company not being where it needs to be, and how he knows the headsets have to get smaller, more comfortable and easier to use. But he also tells me VR is just the opening salvo in Facebook’s longer-term ambitions, and that it’s the bridge to a future where we’ll communicate using augmented reality, which blends computer-generated images with the real world.
He’s also clear that Facebook is the company – the only company – that should be leading the move into a future in which we work, play and interact inside these virtual and augmented worlds. And then he takes a thinly veiled dig at rivals Apple, Google, Microsoft and maybe even Magic Leap.
“It’ll probably turn out pretty different if we’re helping to shape this versus any of the other companies that might work on it, who I think are more inclined to push a model that’s more just around ‘here’s your app, here’s your content, I’m gonna pull it from a store,'” he says.” We want to help shape the next computing platform to be more about interacting with people and not just apps and tasks.”
As for when that’s all going to happen – the move from VR to AR, from niche market to mass consumer usage – Zuckerberg, who’s already five years in with Oculus, tells me it’s going to take more time to get to a place where “people will kind of expect excellence from the work that we do.”
“I don’t think it’s a 2020 thing. But hopefully it’s not a 2030 thing.” And even then, Zuckerberg sees VR and AR still living together. “If you think about how we use screens, phones are the ones we bring with us, but half of our time with screens is TVs. I think VR is TV and AR is phones.”
Aside from talking about larger-scale visions of the future, my time at Facebook was spent demoing the Oculus Quest’s hand tracking, a straightforward and practical software addition that allows my hands to directly plunge into virtual worlds without needing a controller. While it may not sound exciting, it’s probably the most helpful step forward toward putting Oculus’ impressive but still off-putting VR goggles into more hands.
The Oculus Quest can see my fingers now
In a well-lit room in Facebook’s Building MPK 21.8, a team of engineers hands me a familiar-looking Oculus Quest. Controller-free hand tracking is coming in early 2020 to Facebook’s standalone VR headset, powered by the four outward-facing black-and-white cameras on the front that also track full room motion and controller movement. I’m giving it a test drive.
Once the first demo is launched – called “Elixir” and made by developer Magnopus and Oculus Studios – I see my hands as two disembodied glove shapes, floating in front of me. I can move each finger. I can give a thumbs-up, point or make a peace sign. It’s almost like holding the Oculus Quest’s versatile and button-filled Oculus Touch controllers, which also simulate hand motion but can be difficult for newcomers to figure out. This time, it’s just my hands. I stare at my virtual fingers. It’s mesmerizing.
“VR is going to be bigger than people realize.” – Mark Zuckerberg.
I’m in a cartoon-like magical laboratory. I dip my hands into a vat of yellow dragon vomit and my hands become yellow and goopy. I walk over to a fiery panel and place my hands there, and they burst into flames. I can pinch my fiery fingers, popping floating bugs. Blue toxic ooze gives me giant octopus hands, and I stare at my floppy, sucker-covered tentacle fingers.
It all really works, but I do notice a lag between my movements and the action that doesn’t happen when you’re using Oculus Quest controllers. Facebook’s engineers say the Quest controllers are still the best way to play games, but hand controls will be for those who find the controllers confusing or want a more casual way to interact. Or, more importantly, for employee training apps for businesses.
Zuckerberg told me earlier that the company’s aim with hand tracking is part of its pursuit of “presence.” What this means may not even be a headset as we know it in the future. Zuckerberg sees presence in AR as different from what’s available in AR today. “I think people ship different products that they will call AR, that don’t have the ability to actually deliver a person, here, that you feel is with you. You might get notifications, you might get some glasses products that could be pretty useful and could do different things. But I think there’s still some basic technology things that need to get worked on, that we’re focused on.”
My second demo is more pedestrian. It’s a water damage insurance claim inspection simulation for Farmers Insurance. I walk around a virtual kitchen looking for wet spots: a moldy area near the fridge, a drip under the sink. At each spot, I extend my hand and find a pop-up button, which I can tap to indicate whether I thought an area was damaged or not.
Hand tracking has been around in both AR and VR
I’ve seen my hands mapped into virtual worlds at immersive Tribeca art installations and in location-based VR attractions like The Void. Expensive AR headsets, such as Microsoft’s $3,500 HoloLens 2, rely on hand tracking instead of controllers. Facebook has acquired VR hand-tracking technology companies, including Nimble VR in 2014 and Pebbles Interfaces in 2015, and has promised breakthroughs in hand tracking on its headsets for a while. But this is the first time I’ve seen hand tracking working on an Oculus headset. The hand-tracking computer vision tech, developed within Facebook’s Reality Labs, can recognize 20 landmarks or points per hand.
Facebook plans to offer the hand-tracking update along with a software development kit for app makers next year, and hand tracking will work in Oculus’ Home software and in the VR OS, which power basic browsing. When controlling farther-off things, like a pointer or a remote control, you’ll pinch your fingers together and navigate like an air mouse. Your hands could be a remote for browsing the web, playing videos or exploring 3D maps. I may prefer the excellent Oculus Touch controllers when I’m playing games like Beat Saber, but if I were casually connecting to watch a video, or even using an app like a virtual art museum, I’d probably prefer using my hands.
Zuckerberg agrees, telling me he doesn’t expect everyone to ditch their controllers. “There are things people want a controller for, because there are things that you can only do when you get feedback. But I think the main way people are going to want to interact with it is hands.”
The hand-tracking demo isn’t perfect, though. Aside from the lag, there’s a lack of precision when trying to manipulate smaller objects, and the demo required a better lit room than the Oculus Quest’s cameras rely on for room tracking. Certain features like recognizing hand-on-hand occlusion (recognizing when a hand is overlapping with another hand, like Apple is starting to do with objects in AR) aren’t ready yet either, but are being worked on, according to Facebook’s Jenny Spurlock and Robert Wang, who led hand-tracking development on the Oculus Quest.
Oculus has already made it clear the camera-based Oculus Insight tracking tech in Facebook’s latest VR headsets is the beginning of what will be needed in a future Facebook AR headset. Maybe recognizing other objects in a room using those cameras – and bringing more real objects into VR – is next.
“One of the advantages of using the Quest’s regular 2D monochrome cameras is that we have these advantages of size, weight, power and cost,” Wang says. “It’s a harder computer vision problem, but it keeps us in a pretty good cost boundary.”
“VR is such a new technology, we’d rather reduce friction than add weight to the headset,” Spurlock adds.