with Mark Zuckerberg inside the Metaverse.

Mark and I are hundreds of miles apart

from each other in physical space,

but it feels like we're in the same room

because we appear to each other

as photorealistic codec avatars in 3D

with spatial audio.

This technology is incredible,

and I think it's the future of how human beings

connect to each other

in a deeply meaningful way on the internet.

These avatars can capture many

of the nuances of facial expressions that we use,

we humans use, to communicate emotion to each other.

Now I just need to work on upgrading my emotion,

expressing capabilities of the underlying human.

This is "The Lex Fridman Podcast."

And now, dear friends, here's Mark Zuckerberg.

(both laughing)

This is so great.

Lighting change.

Wow.

- [Mark] Oh yeah, you can put the light anywhere.

- And it doesn't feel awkward to be really close to you.

- No, it does.

I actually moved you,

I moved you back a few feet before you got into headset.

You were like right here.

- I don't know if people can see this,

but this is incredible.

The realism here is just incredible.

Where am I?

Where are you, Mark?

Where are we?

- [Mark] You're in Austin, right?

- No, I mean this place.

(both laughing)

We're shrouded by darkness with ultra-realistic face,

and it just feels like we're in the same room.

This is really the most incredible thing I've ever seen.

And sorry to be in your personal space.

We have done jiujitsu before.

- Yeah, no, I was commenting to the team before

that even that I feel like we've choked each other

from further distances

than it feels like we are right now.

(Lex laughing)

- I mean, this is just really incredible.

I don't know how to describe with words.

It really feels like we're in the same room.

- [Mark] Yeah.

- It feels like the future.

This is truly, truly incredible.

I just wanted to take it in.

I'm still getting used to it.

It's like, it's you, it's really you,

but you're not here with me, right?

You're there wearing a headset

and I'm wearing a headset.

It's really, really incredible.

So, can you describe what it takes currently

for us to appear so photorealistic to each other?

- Yeah, so I mean, for background,

we both did these scans for this research project

that we have at meta called Kodak Avatars.

And the idea is that instead of our avatars being cartoony,

and instead of actually transmitting a video,

what it does is we've sort of scanned ourselves

and a lot of different expressions.

And we've built a computer model

of sort of each of our faces and bodies

and the different expressions that we make

and collapse that into a Kodak

that then when you have the headset on your head,

it sees your face, it sees your expression,

and it can basically send an encoded version

of what you're supposed to look like over the wire.

So, in addition to being photorealistic,

it's also actually much more bandwidth efficient

than transmitting a full video

or especially a 3D immersive video

of a whole scene like this.

- And it captures everything, like the flaws.

Like to me,

the subtleties of the human face,

like even the flaws,

that's all amazing.

It makes it so much more immersive.

It makes you realize that like perfection

isn't the thing that leads to immersion.

It's like the little subtle flaws like freckles

and like variations in color and just.

- Yeah, wrinkles

- Asymmetry.

- Yeah, asymmetry.

- Yeah.

And just the different,

like the corners of the eyes,

like what your eyes do when you smile,

all that kind of stuff.

- Yeah, eyes are a huge part of it.

- It's just incredible.

- Yeah, I mean, there's all the studies

that most of communication,

even when people are speaking

is not actually the words that they're saying, right?

It's kind of the expression and all that.

So, and we try to capture that with the kind of classical

expressive avatar system that we have.

That's the kind of more cartoon designed one.

You can kind of put those kind of expressions

on those faces as well.

But there's obviously a certain realism

that comes with delivering kinda

this photo realistic experience

that I don't know.

I just think it's really magical.

I mean, this gets to kinda the core

of what the vision around virtual and augmented reality is,

of like delivering a sense of presence,

as if you're there together

no matter where you actually are in the world.

And I mean, this experience,

I think is a good embodiment of that,

where it's like, I mean,

we're in two completely different states

halfway across the country,

and it just like, you know,

looks like you're just sitting right in front of me.

It's pretty wild.

- Yeah, I mean, I can't, I'm almost getting emotional.

It's like, it feels like a totally,

it's fundamentally new experience.

Like for me to have this kind of conversation

with loved ones,

it would just change everything.

Maybe just to elaborate,

so I went to Pittsburgh

and went through the whole scanning procedure,

which has so much incredible technology,

so software and hardware going on,

but it is a lengthy process.

So what's your vision for the future of this

in terms of making this more accessible to people?

- You know, it starts off with a small number

of people doing these very detailed scans, right?

Which is, that's the version that you did and that I did.

And you know, before there were a lot of people

who we've done this kind of a scan for,

we probably need to kind of over collect expressions

when we're doing the scanning

because we haven't figured out

how much we can reduce that down

to a really streamlined process

and extrapolate from the scans

that have already been done.

But, you know, the goal,

and we have a project that's working on this already,

is just to do a very quick scan with your cell phone

where you just take your phone,

kind of wave it in front of your face

for a couple of minutes, you know, say a few sentences,

make a bunch of expressions,

but overall, have the whole process

just be two to three minutes

and then produce something that's of the quality

of what we have right now.

So I think that that's one

of the big challenges that remains.

And right now we have the ability

to do the scans if you, you know, have hours to sit for one.

And with today's technology,

I mean, you're using

a meta headset that exists.

It's a product that's kind of for sale, now.

You can drive these with that,

but the production of these scans

in a very efficient way is one of the last pieces

that we still need to really nail.

And then obviously there's all the experiences around it.

I mean, right now we're kind of sitting

in a dark room, which, you know,

is, you know, familiar for your podcast.

But I think part of the vision for this over time

is, you know, not just having this be like a video call.

I mean, that's fine.

It's cool or it feels like it's immersive,

but you know, you can do a video call on your phone.

The thing that you can do in the metaverse

that is different

from what you can do on a phone is like doing stuff

where you're physically there together

and participating in things together.

And we could play games like this.

We could have meetings like this in the future.

Once you get mixed reality and augmented reality,

we could have Kodak avatars like this

and go into a meeting

and have some people physically there

and have some people show up

in this photorealistic form,

superimposed on the physical en environment.

I think that stuff like that is gonna be super powerful.

So we gotta still build out all those kinda applications

and the use cases around it.

But I don't know,

I think it's gonna be a pretty wild

next few years around this.

- I mean, I just, I'm actually almost at a loss of words.

This is just so incredible.

This is truly incredible.

I hope that people like watching this

can get a glimpse of like,

how incredible it is.

It really feels like we're in the same room.

Like there is that,

I guess there's an uncanny valley

that seems to have been crossed here.

Like it looks like you.

- Yeah.

- Like really you.

I mean, I think there's still a bunch of tuning

that I think will want to do

where different people emote

to different extents, right?

So I think one of the big questions is, you know,

like when you smile, how wide is your smile?

And how wide do you want your smile to be?

And I think getting that to be tuned on a per person basis

is gonna be one of the things

that we're gonna need to figure out.

You know, it's like to what extent

do you wanna give people control over that?

You know, some people might try

to, you know, might prefer a version

of themselves that's more emotive

in their avatar than their actual faces.

You know, so for example, you know,

I always get a lot of critique

and shit for having like a relatively stiff expression.

But, you know, I mean, I might feel pretty happy,

but just make a pretty small smile.

So, I mean, maybe, you know, for me,

I would, it's actually, you know,

it's like I'd wanna have my avatar really

be able to better express like,

how I'm feeling than how I can do physically.

So I think that there's a question

about how you want to tune that.

But, overall, yeah, I mean you,

we wanna start from the baseline

of capturing how people

actually emote and express themselves.

And I mean, I think the initial version

of this has been pretty impressive.

And like you said,

I do think we're kind of beyond

the uncanny valley here.

And it does feel like you.

It doesn't feel weird or anything like that.