Raphael Thys
Ever since Star Wars first showed space wizards talking to holograms, calling someone and chatting with their 3D simulacrum has become a staple of science fiction. But based on a patent application published in January, it sure looks like FacebookMeta is trying to make 3D conversations a reality right now — even if not quite the way George Lucas envisioned it.
With the disclaimer that many patents never end up becoming an actual product, the patent does hint at how Meta envisions the future of communication.
What’s the patent about?
Meta, then Facebook, filed a Patent Application in 2020 titled 3D Conversations in an Artificial Reality Environment. Specifically, the application outlines a pipeline for capturing and transmitting 3D representations of callers, “allowing conversation participants to appear as if face-to-face.”
To be clear: this is a patent application that’s recently been made public; the patent hasn’t been granted yet.
What exactly does Meta mean by ‘3D’ conversations?
The patent is written in broad language covering a multitude of devices and use cases, but the gist is that Meta is trying to call dibs on a process for capturing and transmitting your body in 3D — whether to be viewed on an AR headset, a phone, or some newfangled future technology.
In other words, the goal is to make it feel like the other person is in the same room as you.
But video calls are pretty good. What does 3D bring to the table?
Meta’s patent suggests it is interested in using realistic 3D depictions to communicate body language for more natural conversations. This is contrasted with typical video calls, which tend to only focus on the face.
“Understanding body language and context can be difficult with only a two-dimensional (“2D”) representation of a sender. Further, communication often relies on interpersonal interactions, such as spatial movements between participants. Yet communication over video calling does not provide the ability for participants to move relative to each other, as the point of view is fixed to the sender’s camera. In addition, the limitation of video calling on a flat panel display introduces an intrusive layer of technology that can distract from communication and diminishes the perception of in-person communication.”
But this patent doesn’t deal with cartoony leg-less avatars. Instead, the goal is to capture real depth data from participants and transmit their likeness as realistically as possible: imitating real-life conversation is the goal.
So is Meta cooking up a holographic device?
No. Well, it might be, but the patent doesn’t specify that. Instead, Meta seems more concerned with how that 3D data is captured, transmitted, and rendered.
Notably, the patent says these 3D conversations can potentially happen on pretty much any display imaginable, from mixed reality glasses to phones, to light-field displays, to mist screens to… yes, actual holograms.
Basically, Meta seems to be interested in creating the backend and platform that allows 3D calls to happen in the first place.
That said, while there’s no mention of specific products that might leverage this 3D conversation technology, we know Meta is working on everything from AR glasses to a metaverse-friendly smartwatch. And of course, the company does already make the most popular VR headsets in the world.
How can these conversations look realistic if I’m wearing a headset?
It’s hard to have a conversation that feels ‘face-to-face’ if one or more of the participants is wearing an AR or VR headset. To that, Meta has a solution: AI.
The company describes how AI filters could be applied for the “removal of artificial reality devices the sending user may be wearing,” letting your beautiful mug shine through in your 3D calls. How well that would work when wearing a bulky VR headset is anyone’s guess. Meta also notes that filters can remove callers’ backgrounds to make them look like they are occupying the same space.
How can 3D conversations be transmitted on 2D devices?
On 2D displays, you obviously wouldn’t get a true 3D representation, but instead might have an effect more akin to Facebook’s 3D photos or existing AR filters. Your eyes wouldn’t be able to see true stereoscopic depth, but you should be able to move around the person you are talking to like an AR object.
All that 3D information would use a ton of data, wouldn’t it?
Fancy compression algorithms aside, Meta notes that the 3D representations don’t always have to be pixel-perfect versions of you. In many cases, AI or pre-stored 3D models of you could fill in the blanks.
“In some cases, part or all of the 3D representation may not be direct translations of captured data, such as where none of the captured images of the sending user depict a portion of the 3D representation or where bandwidth or processing limitations did not allow sufficient time to provide a portion of the images depicting the sending user. In such cases, some or all of the 3D representation can be approximated with an avatar representation of the sending user, a machine learning estimation of the missing portion(s), or a previously captured version of the missing portion(s) of the sending user.”
Meta also describes how if certain parts of the body are occluded by objects or are outside the camera’s field of view, they might be digitally reconstructed using AI as well. And similar to how we have compression for existing video calls, Meta’s technology could change the polygon count and overall quality of your 3D model depending on which devices are involved in the call.
What does this say about Meta’s future plans?
A lot. Or nothing. But probably a lot.
A couple of years ago, I might’ve written this patent off as just another tech company trying to call dibs on some esoteric technology. But in 2022, now that Zuckerberg has gone all-in on the metaverse — and now that people seem to actually care about VR and AR again — it seems a given that Meta is trying to make conversations in XR a thing.
Again, I need to caution that this is just a patent, and tech companies file patents that never come to fruition all the time. The patent is light on specifics, but instead broadly posits numerous ways in which 3D conversations could be adapted to work on all sorts of devices.
But it’s that broadness that makes it interesting too. Right now, one of the biggest hurdles for Zuckerberg’s vision of the metaverse is the fragmentation between virtual and real spaces.
Virtual spaces allow for infinite flexibility but require bulky, expensive headsets that isolate you from those around you. And there’s always going to be a subset of the population that simply doesn’t care for XR experiences – 2D devices are good enough for them.
But if the metaverse is going to become a thing, conversations can’t only happen on VR or AR headsets — at least not any time soon. There needs to be a way to allow conversations and interactions to happen across device types.
Granted, Meta isn’t the only company working on 3D calls. Google has Starline, Cisco has Webex Hologram, and Microsoft has its Mesh platform. But given Meta’s existing dominance across social platforms and VR experiences, it’s better positioned than anyone else to create the connective tissue between 3D and 2D spaces.