字幕列表 影片播放
MICHIEL BACCHIANI: My name is Michiel Bacchiani,
I am part of the Google speech team.
So we make that microphone button on your Android phone
do something.
So more specifically, I am a senior staff research scientist
here at Google, and I lead the acoustic modeling group.
That's part of the speech team.
And so I've been doing speech recognition
research for about 20 years.
I've been at other places before here.
I worked for about three years in Japan in the ATR--
the Advanced Telecommunications Research labs,
which is sort of like a Bell Labs of Japan.
And I worked at AT&T research for about five and 1/2 years,
and I had a bit of a stint at IBM research
in upstate New York for about a year and a half.
And I've been at Google for almost exactly eight years.
And speech recognition is sort of a typical hard
applied science.
So there's a lot of math a lot of statistical modeling.
And as a result of it, we like to process
large amounts of data, and so that
brings about a large amount of computer science.
And so Google looked like a very interesting place to work.
So I was intrigued by this place when
I was offered the opportunity to do something there.
About eight years ago, I looked at it,
and it certainly is a very different culture
from what I was used to.
So when I came from is the typical research
lab where there are somewhat of the ivory tower syndrome
where you sit together with all the other PhDs
and, you know, you're doing exhaustive research,
and then there's the development group that's somewhere else,
and we don't really talk to each other.
And there's a bit of finger pointing
going back and forth where the researchers would say,
they never take our new idea.
And the development group would say
that whatever code we get from research is such a mess,
we have to all re-implement this.
And both of those is true, but it doesn't really work well.
And as a result of it, you don't really
have an outlet that is built for your product.
So you would live off publications, largely,
and that's one way to get your stuff out
and that would be exciting.
But, so now I'm looking at this Google place
that is clearly very different.
Actually, one of the concerns I had for real
is that, are these guys all sitting on colorful skippy
balls high fiving each other and saying awesome all day long?
Which is not really something I enjoy,
because I did enjoy the hard science of what I was doing.
Also in retrospect, what I think is very different from Google
to do other places, I think, is sort of the three aspects,
I think.
There's a people component to it,
which is we are extremely collaborative
between engineering and research.
In fact, we make hardly any distinction,
where it's hard to distinguish whether somebody is a software
engineer or a researcher.
And it's just because, if you want to get your stuff out,
it has to be engineered.
And if you're an engineer and you engineer something,
you might run into a problem and you
have to research on how to solve that correctly.
And so that's a nice environment where people are collaborative
and there's no hang ups about what department
you're in or what your title is exactly.
and you really jointly get something out.
I think the second aspect of it is a product.
Your stuff really gets used by a lot of people.
There's a real path to get whatever
you code to get in the hands of people,
and actually, people use it.
And not only is it sort of a satisfaction
that somebody will use your code.
There's some interesting stuff that it brings up for research.
I mean, it gives you signals to look at on how people are using
things, whether it's working for them or not.
And you can really focus on a user
and sort of make their experience
better, which is an interesting research setting in itself.
Sort of, how do you make use of those signals
to improve your product?
And the third thing is sort of impact, because you know,
you really can improve people's lives.
We did some, you know, I work in the speech teams.
Working with some deaf people and giving them
opportunities they never had before is exciting.
You really, I think, have a more profound impact
on somebody's life.
And I think a lot of Googlers have
this mean altruistic streak where they really
like the fact that their stuff can
do something good for somebody.
And that brings a lot of excitement about.
So that might sound a little naive,
and naive can be bad, because I have colleagues that
look like they're 13-year olds.
And I know they're not, but they look like that to me.
And they think that the normal world is whatever our Google
environment is, and even if you tell them that's not true,
you can't really convince them.
And we have this rich food culture,
as you might have noticed.
I literally heard a complaint to somebody that said,
do they have to really put truffle oil and everything?
There's somewhat of a bubble here, that's true.
But in general, I think it's a pretty happy place to work.
So more specifically to speech, I
think that it is really this kind of company in this mindset
that it's not surprising that it took speech, I think,
from when I joined eight years ago from sort of user torture.
You know, where it's like, say or press one for sales,
or say or press two for, you know.
To now, we're sitting at home and looking at an ad,
and people are like, I like this cellphone
because it has voice recognition,
as a feature that they actually want.
So again, sort of this user focus, I think,
is a positive thing.
And I think, in more general, it's not just the speech thing
where it's been lucky that they had some success.
Information retrieval, Google search has been applied
and really is useful to people, I think machine translation,
I think there are many, many more areas to come.
So don't just take it from me.
I think we have a very interesting panel
with various other people in various other areas
that you can quiz and we can discuss things.
Some.
Of the things I think that we will talk about a little
before we go in our Q&A session is
to discuss this whole collaboration, which I think
is contrasting some of our relationships internally,
as well as with the research community outside.
So let's start with introductions, I guess,
of the panel.
So how about we go from left to right.
So I'll let you guys do it.
UMESH SHANKAR: My name's Umesh Shankar.
I've been at Google for about seven years, previous to which
I did my Ph.D. at UC Berkeley.
I entered Berkeley as a systems student
and fairly quickly ended up focusing
on security and privacy, which is
what I still do here at Google.
I'm a software engineer.
I lead the data protection engineering group,
so we spend pretty much all our days
figuring out how to make sure that all
the private information that Google stores stays that way.
And as far as I'm concerned, Google
is really the most interesting place
to work on security and privacy, because we don't really
have that many theoretical attacks.
Most of the stuff that most people think of as theoretical
is real.
And there's really no playbook for how
to do data protection at the scale
that we want to do it at with the performance and usability
requirements that we have.
I mean, it's just been an incredible learning experience
for me here.
And to a degree that I never really thought I would,
I've used almost everything I learned in graduate school.
EUGENE WEINSTEIN: Hi, I'm Eugene Weinstein,
and I think I'm one of the 13-year-old looking people that
Michiel was--
MICHIEL BACCHIANI: You wish.
EUGENE WEINSTEIN: Well, I don't know, you might disagree,
but in the sense that I've never really worked-- well,
some small internships aside, I've
never really worked at another company other than Google.
So I was lucky enough to be able to do
kind of the applied part of my Ph.D. Here as an intern
while I was a Ph.D. Student at NYU,
and I joined here full time about 3 and 1/2 years ago,
and I work on speech with Michiel.
So I work a lot on the internationalization side
of speech.
So at Google, internationalization
is a big thing.
We can't just get away with making our products
work in English anymore.
But for speech, this is like a whole new ball game.
Because it's not just, like, translating some strings,
right?
We have to model new languages, right?
So this is a difficult thing.
So I kind of enjoyed being able to scale things up
in that respect.
Now, voice search supports 40 languages,
we just launched four new ones a couple of weeks ago.
So that's been so that's been really fun to work on.
And the other scale issue that I've
gotten to work on that I think has been really
meaningful to me is the work that I've
done with keeping speech models fresh.
The things that people are talking about
and the ways in which they say them are changing, right?
As demographics change, things come
into and out of popularity, and people
change the way they search for things with their voice.
And we have to take the petabytes of data that's
coming in from users of our products
and we have to make sure that we're
servicing the user in a way that's
still current and relevant.
And so to me, it's been really fun to take the, let's
say, theoretical things that I learned about in grad school
and apply them at this ridiculous scale
that I could never even conceive before I came here.
So it's been really fun.
OKSANA YAKHNENKO: Can you hear me?
I can't hear myself.
So, my name is Oksana Yakhnenko.
I've been at Google for a little bit over two years.
I joined in like December, 2010.
I have my Ph.D. In computer science
and I was mostly working on machine learning
and applications to various things
like text classification, computational biology,
and computer vision.
And then I've been [INAUDIBLE] in computer vision
for about a year and a half.
And one of the reasons, when I got on the job market,
I was a little bit disillusioned with academia, in terms of,
I publish a paper and you spend so much time writing this paper
and you so much work in the paper.
And then you have like four citations and that's kind
of the end of it.
I mean, sometimes you have more than that,
but I actually felt like my work was affecting
a very small niche of people.
So basically, I started looking for a place where
I could actually do some more impact.
And now I have joined a group in Knowledge,
so we work in search.
And specifically, my group, we don't really
do anything that you guys see directly,
but we work sort of behind the scenes,
and we work on information extraction.
So there's this effort called Knowledge Graph,
and this is basically the graph of related entities
that Knowledge Panels uses to answer
the questions and the queries that you
guys type in the search engine.
And basically, our project is to automatically populate
this graph with a lot of unknown entities
and unknown relationships.
So things that we do, we do a lot of experimentations
with like, hey, here I have this model.
How good is this model?
Not only on paper, that like, this model
is going to give me 95% accuracy, which is like, OK,
cool.
But like, how many users is it going to impact, right?
And it's like, even if my model says that it's 95% accurate,
do I know that it's 95% accurate?
Because not everything that's on the web is true, right?
So that was basically some of the more challenging things
that I've been working on, things
that I enjoyed doing here.
GIDEON MANN: Thank you.
And thank you, Michiel.
I don't think I've heard such an eloquent description of what
it's like to work at Google.
And I think I would echo a lot of the same things.
Especially, Michiel talked about a mean altruistic streak.
So I think that, if I have an affliction, or one
of the afflictions I might have would be that,
and so I've always been looking for opportunities at Google
to use the work that I've done to have some kind of a broader
impact.
And one of the nice things about this place
is that you have a platform to do that,
and the opportunity to do that, and the support to do that.
So in particular, I have worked on machine learning.
My background's in natural language processing,
but I'm moving a little bit more into machine learning.
And in particular, I've been looking
at how to make machine learning something that everybody can
use, and something that is easy to use,
and something that you can drop into your web service,
and something that you can training easily.
And so, it's been really exciting
to pursue that here, because there's so many opportunities
to get feedback on whether you're doing the right thing
and going in the right direction.
So I'll cut myself off there and turn it over,
but I'd love to talk about that more with you guys.
OMAR BENJELLOUN: So my name is Omar Benjelloun.
Can you hear me?
So yes, I've been at Google for about seven years.
Before that, I did a Ph.D. In databases
and I spent two years doing up [INAUDIBLE]
also, working on databases and uncertain databases
and reference reconciliation problems.
And I also joined Google for similar reasons
to what was discussed earlier, like having an impact
and doing things that are really useful to more
than the community of researchers that would maybe
read my papers.
So at Google, I've been also working
on projects related to structured data.
And so, I just want to mention two projects that I worked on.
One is about, so I lead a project
about public data and statistics.
And the goal of that project was to
and continues to be to get official data
about, important statistics about, the world, countries.
Your city gets that official data
directly from the sources of the data, which
are international or national organizations,
or local organizations.
There are lots of reconciliation data integration problems
to solve in order to do that, and provide also means
for any provider to give the data themselves
and using an easy to understand format.
And then reconsolidate all that data in a place
where we can make it useful to users through visual tools
to explore the data.
Visualizations, ways to drill down,
compare, and so on in a tool called Public Data Explore.
And more recently, we've been surfacing that data directly
in Google search.
There's a statistics feature that we just
launched about three months ago where you can,
if you search for a population of India,
you'll see graph that's interactive,
and you can move your mouse and see the value over time.
But we also try to give you context by figuring out
what are the most relevant other countries to compare
this data to in order to help you understand the content
and help you explore.
And currently, I'm working on another project in parallel,
which is about, again, structured data,
but figuring out what is important,
how to rank the information that we show you.
So if you search for, say, Brad Pitt,
we show you a knowledge panel about this actor,
and we'll select a few facts that are important about him.
So how to select the right set of information
and rank that information is this different problem
than search ranking, because it's the structured data world.
It's a very sort of both research-y and challenging
problem, but it has direct applications
and direct impact on our users.
And so, that's what I find exciting about our work.
MICHIEL BACCHIANI: All right, so you have some idea, I think,
who is on our panel.
So I would like to bring up some of the topics
that I brought up, and you can chime in
whether you think this is appropriate or not.
One of the things is always the perpetual question
about researcher versus engineering, and how it's
organized, and why they can never get a straight answer out
of any of us, whether they're a software
engineer or a researcher.
So anybody wants to chime in on that?
GIDEON MANN: I'll take the bait.
So if you can't get a straight answer,
I think that's almost by design.
I think that at the company, there are really
a tremendous number PhDs, and the PhDs
are in every part of the organization.
I took a training today from someone
who had a psychology PhD.
And she was in the people [INAUDIBLE].
And I was surprised, though I probably shouldn't have been.
Because that's really the nature of this organization.
It's an amazing place in terms of the level of education
and intelligent [INAUDIBLE] and everybody here.
So as far as the distinction between these different roles,
I don't think there is much, except it's more determined
by your local contacts, who your local manager is,
and your local group is.
Alfred is one of the people that's
been thinking about research a lot at Google.
Likes to talk about that research
being done opportunistically.
[INAUDIBLE]
I think that that's probably about right.
MICHIEL BACCHIANI: I think you want
to use the microphone so that we can hear you.
when
OKSANA YAKHNENKO: So we have to do annual reviews for yourself
and each other in the company.
So do the publications come into play for research scientists
when you guys do your work?
MICHIEL BACCHIANI: It's an important issue, I guess.
OKSANA YAKHNENKO: Take the blue pill.
OK.
So when you do your performance evaluation,
do the publications-- how much publications actually
weight for research scientist?
GIDEON MANN: So this is, again, my biased assessment,
but I think Google as a company, really judges you on impact.
And impact is a lot of different things.
Publications, launches, standards, talks,
contacts that you develop.
So I think that it really, again,
is you want to self define your impact
and how much, according to the way that you've defined
your impact, you've achieved it.
Eugene, you feel like you want to add something?
EUGENE WEINSTEIN: I definitely see it
as, let's say you come to Google and you just
do engineering projects, right?
And don't write any papers and don't
work on what people outside would
view as quote unquote "research," right?
That's OK.
It's an engineering organization, right?
But let's say you come here and you
expect that you'll be able to just, like, work on papers
and do theoretical stuff and not take part
in contributing to any products, like some of these more
traditional research organizations
that Michiel alluded to, that's not really OK.
So I think that it can definitely
be a boost for researchers when it comes to performance, when
it comes to evaluating your impact.
But that can't be the totality of the contribution
that you bring to the company.
MICHIEL BACCHIANI: So can any of you sort of comment on projects
that you're doing which you think
are essential to have this integration between engineering
and research?
That you don't think could ever get
done in the sort of traditional model, where
it's completely distinct silos doing either research
or engineering?
Or where it would be a much longer process
or much more involved to actually get that
off the ground?
You all work in different areas, so I
guess there should be a wealth of examples of things
you get out there that you think are
good examples of mixing the two.
UMESH SHANKAR: This those kind of
works if you hold it really, really close to your face.
OK.
Cool.
So maybe my group isn't the best example
of this sort of inextricable link,
but we're definitely forced to confront
kind of the edges of standard practice very, very often.
So in cryptography, one of sort of the first things that you
learn is, don't make up our own, right?
It's like very, very hard to get right and wherever possible,
you should use a standard thing.
But once in awhile, we come across situations
where the standard thing doesn't quite
have the properties that we need, and it's
times like that I'm really, really glad that we have
absolutely world class cryptographers who
work at Google, that we talked to
and say, OK, what are our options here?
And sometimes, they'll actually come up
with something different.
In other cases they'll say, well, there's
this new encryption mode that's sort of experimental.
And we believe it has these very specific weaknesses
and it either is or isn't appropriate
to the particular thing that you're doing.
Absent that expertise, I just don't
see how we could possibly make a good decision of what to do.
Conversely, because you're confronted
with such big challenges, and because the culture
of the company is one that really encourages people
to take big bets, that already tends
to blur the line between research and engineering.
Because I think a lot of people's mind,
engineering is something where, it's sort of really clear
what you have to do and you just kind of do it
on a quarter by quarter basis.
And research is this thing where you really
don't know what you are supposed to do,
and you get a really long time to do it.
In reality, I don't think either of those is really true.
I certainly know as a graduate student,
like, I had to publish stuff every year
to kind of keep my name out there and do a good job, right?
and I think for new faculty, it's probably much, much worse
before they have tenure.
Conversely, here, as I was saying before,
there really isn't a playbook.
It's very rare that we know in advance
what is the right thing to do.
And a big part of the challenge is figuring out that thing,
trying really bold and innovative things,
and finding out what the right thing is.
Whether it's published at the end or not,
I mean, that sounds like research to me, right?
The difference as I was saying to some folks
earlier is that it has to work.
You don't get to make assumptions that aren't true.
That's a really good forcing function, right?
And then I certainly know that in my previous time
in academia, like, yeah.
A lot of people would publish stuff and at a conference,
someone would ask a question like, but is that really true?
And really, the person brushes it off,
and that's kind of the end of it, right?
And the difference is that here, you
can't do that, because it has to work.
And I think that constraint of it having to work
is incredibly positive for innovation.
It forces you to kind of stop doing things
that don't work and try new avenues,
and I think the end result is much,
much better as a result of it.
And I think you're saying also about how
you want your impact to be real.
And the net result is, that's what everyone wants here.
We want to make a real difference,
and that's part of that same function.
I personally find it extremely satisfied to do that.
OMAR BENJELLOUN: Just one last comment.
I think one difference that I see between doing research
or just innovative work at Google versus in academia
is that the constraints that we have here are the real ones.
They're the real, most challenging constraints that,
if we solve them, then we have the potential
to have an impact on-- to solve the problem for real.
And I find that in academia, sometimes,
at least in my experience, we tend
to solve, sometimes, problems within constraints that
sometimes can be artificial, or we
try to define a set of constraints that
make the problem tractable, without knowing if these are
the real, most realistic constraints.
While here, we have users and they use our services.
And if we don't come up with something that works,
then it doesn't work.
So it has to work
GIDEON MANN: Here, I'll give a concrete example of an area.
So, distributed optimization.
When you have a particular optimization problem
with very many variables, larger than it's
going to fit on one machine, you want to solve it very quickly.
It's important to the business.
How do you do that efficiently?
How do you do that with low loss?
People have been looking at this problem for decades.
In the early '80s, there was work out of MIT
for tickets to [INAUDIBLE] looked at this problem.
But the networks have changed and the problems have changed.
And so, there are a lot of people
looking at this problem or variants.
One of the key parts of the Google Brain project,
if you read the paper carefully, is
they had a very clever implementation
of a large scale distributed optimizer.
I mean, as Omar and Umesh have talked about,
you're not forced to look at this problem,
you're not forced to solve this problem in every context.
Here, you're really forced to.
You really need both a theoretical perspective
and applied perspective to solve it.
And then the end result is something pretty spectacular.
EUGENE WEINSTEIN: So, we talked a bit
about how the constraints are real, right,
of what you have to-- oh sure we talked a lot about that.
That's an area where we're constrained,
but I think it's also worthwhile to bring up
the areas where the constraints, the-- how shall I say it,
the leeway that we have seems almost so real,
which is in the computation infrastructure
that we have access to hear Google.
Sometimes, you have to solve a research problem just
to get the thing to parallelize at the scale
that you're allowed to run it at.
The fact that we have to answer to Google's users
introduces the constraint in that sense
that we can't do stuff that's wrong, right?
But the situation with the infrastructure
that we have here at Google, it makes things that much more,
I think, appealing to a computer scientist from just
coolness standpoint and from the standpoint of how much you
can get done.
Sort of like the amount of leverage
you can sort of bring to the effort
that you are able to make as an individual
by using this infrastructure.
OKSANA YAKHNENKO: So I think this
is to Omar's comment on constraints.
So in my perspective, it's almost
like your definition of research changes a little bit.
So in academia, you have, like, either a set of known problems
that you're working on, or you try to make up a problem,
and you want to convince everybody else
that this is important.
Whereas research here is finding-- sometimes actually
the problems jump you immediately.
Because you have an algorithm that works amazingly well
on paper and it works well on, like, a million examples,
but then if you try to round it over the entire web,
it just doesn't work.
So how do you scale it up?
So this is something that Eugene brought up.
And then there's another way of looking at it is, like, how
do I find a problem that I want to solve
that's worth pursuing, right?
And even answering a question like that
is a research in itself.
So it's like, what are users want, right?
And if I want to get from A to B,
what are the resources that I need?
What are the kinds of people that I need to talk to?
How do I evaluate the results that I have?
How do I scale it up?
How do I put it into production?
Things like that.
I think that's a lot cooler and has
a lot more-- you learn a lot more, I think,
by actually doing research that way than just, like, hey,
here's a cool problem.
Let me try to come up with another kernel for a support
[INAUDIBLE] machine.
MICHIEL BACCHIANI: So there's been a fair amount,
I think, of academia bashing.
But I think all of us have a warm and fuzzy feeling when
it comes to academia, in the sense
that we engage with the community quite a bit.
And I think in various ways, we try to be supportive.
So in contrast to all the telling them
how they do everything wrong, maybe we
should tell them a bit sort of, how we are in favor of them,
and try to play nice with them.
UMESH SHANKAR: We love you guys.
No, seriously.
OKSANA YAKHNENKO: Are there people who are in academia?
Can you raise your hands please?
OK, so we're not offending anybody.
MICHIEL BACCHIANI: So I would like
to bring up some other things we do, in terms of, you know,
we do publish quite a bit.
I think it's encouraged, which I think is a bit of a misnomer
that we don't care.
I think we do give a lot of monetary funds
to people that are in academia.
We have a giant program, in terms of visiting researchers.
I think we try to be collaborative outside as well
as inside, and maybe you could share
some of you experience with that.
GIDEON MANN: So I think, in addition to everything
Michiel said, one of the interactions that I personally
have enjoyed most is the intern program.
Because it's a way not only of getting
kind of fresh blood and fresh ideas,
but it's also a way of strengthening connections
to the academic community.
So friends of mine that have gone into academia
will send interns my way.
And this is a way of kind of keeping
those connections strong and also transmitting knowledge
back and forth.
So I'm thinking of a friend of mine, Noah Smith, who
works at CMU/LTI, and he is like super,
super into these advanced latent variable models
and graphical models, and it's great, actually,
to have his-- he has a student here, now.
It's great to have his student here and get
that learning here.
And then I think also, it changes
his student's perspective about how to run experiments
and what experiments might be like.
To say that, you know, can simply
run 1,000 iterations or 1,000 different models at once,
and what does that mean about the search space?
What does that mean about the techniques that you might try
and you might want to investigate?
And so, I'm hoping that it really--
and I'm expecting that it's not going to be a one way street.
It really is kind of like an exchange.
I had a student from another friend of mine
who was working on systems related problems.
And I think when he came here, he
saw that the depth of kinds of things
was very different than the problems he had looked at.
And so, it was also a lot of insight
that he could bring back.
And so that kind of relationship I find especially valuable.
And that intern program is vast So
among our panel, who has current interns in their team?
So I think that gives you a sense.
I don't think that's uncommon.
UMESH SHANKAR: Just to kind of add
to that, as I was saying before, and it sounds kind of funny
but its actually also true.
You know, as a proof that academic research really
is valuable is that I really did end up
using so much of what I learned in operating
systems, databases, networking, of course security and privacy,
all these areas.
The concepts that were developed in academia, things
that kind of maybe seem far fetched
at the time, that only made sense
to use an algorithm at an extremely large scale.
Well, that time arrive.
We're very grateful that kind of work had been done.
And I think I wholeheartedly agree
about the kind of interchange between a place like Google
and academia.
I mean, we go to conferences.
That's a great example of that, where, not just Google, right,
but many other companies in the industry
have people who actively work on research topics, who publish.
You go to the conferences, you learn from each other, right?
And in some sense, the conferences, like,
shows you how many similarities there
are in the kinds of problems that people work.
There is also something that, the flip side of my comment
earlier that the things you do have to work,
is that it takes time to keep them working.
And there's a certain tax that you pay,
often substantial, for that.
And in academia, it is a little bit freer
of those sorts of constraints, and there's
something very nice about that too.
It lets you go in perhaps more different directions
than you would otherwise.
If I were to draw another sort of,
like, compare and contrast between academia and a place
like Google is that I think Google actually really rewards
taking long term big bets in maybe a smaller number of areas
as an individual, which can have a really big payoff.
Of course, the flip side is, maybe it doesn't work out
and now you're like, man, I just a couple of years on this thing
and it didn't end up really working out.
Academia, I think, in general, it's quite common for people
to work on more different projects at once,
and kind of, OK, the idea captures your fancy,
you can kind of start working on it, even
more so than you could at a place like Google.
And I think having both those models out
there probably is a really good thing, in total, for when
you look at what we, together, can produce.
MICHIEL BACCHIANI: Gideon disagrees
GIDEON MANN: You know, I'm not sure I disagree,
but I don't think I've ever thought about that way.
It's really interesting.
I think that you're right.
I mean, there's like a historical baggage
you start to accumulate of, this is
what the world, meaning Google, ecosystem looks like,
and that's what you have to operate in.
I don't think I've ever thought about it that way.
Interesting
MICHIEL BACCHIANI: Sometimes with publishing,
this is something that I think I frequently hear,
is that we are not allowed to published.
There's, I think a rumor that I've-- not universally,
but that's something I've heard frequently that--
or it's discouraged, or whatever.
And I think it's very unfair, but again, this
is one person saying it.
But you guys want to comment on whether you think
this is a fair statement, or where does this perception
come from, if you had to guess?
OKSANA YAKHNENKO: I think the perception comes
from that a lot of people who actually go to Google sort of,
like, disappear for a couple of years, and they don't publish.
The reason they don't publish, I think, is actually twofold.
Not enough time and not enough time to write a paper.
Because you get so interested in what it is that you're doing
and you become so involved in actually getting the thing
to work and seeing the impact of this thing.
Sometimes, you just don't really care to write it up,
or sometimes you just don't really
have enough time to write it up.
With that said, when I joined I still
had a couple of leftover projects for my grad-work,
for my graduate work.
And I took like two weeks or three weeks
that I was allowed to use Google's resources,
and like even like Google [INAUDIBLE]
actually wrap it up and submit a publication that got accepted.
They sent me to the conference.
And this was, like, for the work that I've
done that was still the ideas that where
the academic ideas when I was in school.
I'm an engineer, but I've done a couple of collaborations
with people in research where it's sort of like a two way
street, like where I tell them the types of problems
that we're really interested in, and I simply just don't really
have the time to look into it and where somebody
from research actually was like, oh yeah, cool, yeah.
This is like my area of expertise.
I have an algorithm that could be well suitable for this.
I can run some experiments and see
what will come will come out of this.
And then basically from something like this,
a paper evolved, and we actually submitted a paper
to conference.
In terms of submitting a paper it's like super easy.
This person worked for Corina Cortez,
and it was basically like telling Corina about the paper,
and she was like, yep, cool, you can submit.
Like, this is the level of approval
that it had to go through, pretty much.
EUGENE WEINSTEIN: I think with respect to publishing,
the things that we're saying here,
you guys don't have to take our word for it.
Just go to research.google.com, right,
and you'll see that there are hundreds
of papers coming out of Google.
These aren't only people that are
in the research organization.
So I think, for most people here,
I think it's pretty much a personal choice
about whether you care, like Oksana said,
whether you're motivated by getting your work out there
in front of people outside of Google,
or whether you want to focus more internally
about presenting your work to your colleagues
and getting buy-in from them and getting collaboration
opportunities there.
It's very much a personal choice,
but if that's really your passion,
I would say that the vast majority of the people
are able to publish, and publishing
is encouraged by managers, by VPs.
MICHIEL BACCHIANI: We have a question.
AUDIENCE: So I wonder, talk a little bit about the origin
of problems.
Obviously Google has lots of products that do things,
and clearly problems that are directly related
to those things are important and valuable to the company.
Is it possible to have an idea that maybe doesn't
have such an obvious connection to something for the company,
and to what extent would you be able to actually run with that?
A month?
A year?
Two years?
Whatever.
Does that make sense?
EUGENE WEINSTEIN: So I don't know if you're familiar
with the Chauffeur project, the self-driving car out
in Mountain View, so that's one example of something that's
totally not connected to the mission of the company,
as most people would see it.
So there are things like that.
Like, would I be able to, right now, go to my manager
and say, can I go and spend 50% of my time
building a self driving car?
Maybe not.
That's not always the case, but there
are these things that happen in that kind of organic way
and that aren't always directly related
to what your job role is.
MICHIEL BACCHIANI: So it seems to be the perfect time
to bring up the 20% project, which isn't quite 50%,
but, you know.
In contrast to what you said about the freedom
to do off-the-wall things, I think
there is some provision that provides a lot of people
to do so.
I think a lot of us, because we're so ingrained what we do
and like what we do, we don't really make use of it.
But it's a reality that exists, I think.
Maybe can you guys comment on whether you
have people doing 20% things?
OMAR BENJELLOUN: So I considered as a manager,
I've had many of the people on my team
who are interested in doing 20% projects,
and I've always encouraged them to do that,
especially if it's something that's
going to help them learn a new skill
or advance in their career or establish new connections
or find new, potentially interesting areas to develop.
And it's generally been a very positive experience.
Some people do it for a long time,
some people do it short period of time,
and then they stop doing it and then they come back to it.
It varies.
It varies a lot.
One other comment I wanted to make is, also, in the projects
that we work on, there's sort of a mix of projects.
So I think that many product areas at Google
have some established sort of short term projects
that they know they want to get done and launch quickly,
but they also have some sort of fraction of projects that
are more sort of exploratory and are trying to do explore sort
of longer term opportunities, and those
can be more researchy.
And it's always possible to go and propose an idea
and say, hey, I think there's something really interesting
here, and that's an interesting challenge,
and if we solve this problem, this
is going to have a big impact on what we're doing.
And if you convince your director,
your VP, then you might even get some headcount and engineers
that can work on this project.
GIDEON MANN: So Tom, to address your question,
I think one thing that I didn't realize about this place,
before I started is the autonomy and self
direction that each individual engineer has.
They have it in the choice of project,
but they also have it in their interest of time.
And so, you are so valuable as an engineer to the company
that you are actually a prized resource,
and the company's goal to some degree
is to keep you as excited as possible.
And so, to some degree, that means
that you can kind of move and your place of interest.
Now, I think what people usually find
is that after a certain point there,
they start thinking about aligning themselves
to the company's larger interest.
Now how can I support this larger area
that the company is going in?
It means that your projects will move faster,
it means that you'll move faster,
it means you'll get more support,
more other people will be interested in joining
your torch-bearing cause.
But it really is kind of your own discretion.
I don't think I understood how much that would
be true about being here, but it certainly is.
UMESH SHANKAR: Just to kind of follow up
on that very briefly, I think like the usual process by which
this kind of thing happens is, you start small,
like you build a prototype.
And the real process by which things get kind of sorted out
and what ends up making it and what doesn't is,
like, there's a marketplace of people's time
because people's time is really valuable.
And if your idea is really good, and you convince
people to work on it, then your project will grow.
And if you can convince people that your idea's good,
maybe it's not that good.
Because maybe it is and we're missing an opportunity,
but I think, on the whole, the system works pretty well.
And it works very organically, that means
that you have to probably invest more
time in learn skills of persuasion and things
that you might otherwise to get it off the ground.
That's not necessarily a bad thing, either.
So yeah, small things can become big.
If anything, I think maybe one mistake
that people sometimes make in trying
to start new projects is thinking too small.
It's hard to get people to want to sign up to something which,
even if it all works out great, still ends up being small.
That's just not that exciting.
So having a big vision for even something that
starts out small really ends up being key.
And that's really, I think, a lot of Google's all about.
GIDEON MANN: I just want to piggyback on
that I totally agree.
And I think one feedback I get from-- I
met with Jennifer Rexburg, who's a prof at Princeton,
and she was like, it's amazing, I meet all these engineers
at Google and they're all so able to communicate.
And I think, partly, that's not the interview process.
I think, partly, it's the work process.
You're forced in kind of your daily process
to communicate with other people.
That is kind of what will propel you and communicate
about your ideas.
And so, just as Umesh was saying,
it's like, that communication, persuasion, building
the vision, that is kind of the part that
will drive you on your project.
So agenda-wise, I think we're getting
close to starting the Q&A session.
I've noticed that we have more and more questions,
so I would like to encourage it even more.
AUDIENCE: This is a segue [INAUDIBLE].
I understand that you are all PhDs in computer science
[INAUDIBLE] as far as I can tell,
seem to work on areas within Google that
follow from graduate research context
in a specific area for your dissertations.
And so I was wondering, in part because I am a math PhD, not
a computer science PhD, about the adjustment process
that some people have [INAUDIBLE] Google, as friends
of mine have PhDs in biophysics and then
they start to work on video chats.
One example that is true.
Because I think it's kind of an adjustment process
to be a database researcher in academia,
and then a database researcher at Google,
and it's another kind adjustment process to be a mathematician
and then to work on [INAUDIBLE].
What kinds of [INAUDIBLE]?
Just the kinds of resources and learning
curve that people face in that?
OKSANA YAKHNENKO: So, most likely, if you are,
what was your example again?
If you're a mathematician, well, if you're a mathematician, most
likely, you will not be hired as a front end developer.
Mostly because, as far as I know,
and maybe actually, people who work in the HR
maybe will be able to comment a little bit more on that,
but that is as far as I know, they
try to put you on a project where you really
can contribute.
Because, yeah.
STU: May I try answering that one?
I'm not on the panel, I apologize.
[INAUDIBLE]
We have, [INAUDIBLE] one of the earlier comments.
Last time I asked, we had some 3000 people [INAUDIBLE],
some of them are none of the above. [INAUDIBLE].
A whole bunch of psychology.
[INAUDIBLE] a few medical doctors, Juris doctors,
[INAUDIBLE].
But we have over 2500 in the computing cluster, which
I will define as math, science, and engineering.
And then we have another 300 or 400 in biophysics, [INAUDIBLE]
which is as far out as you can get.
And [INAUDIBLE] hiring outside research.
Yes, we do take a look at skills.
If you're going to come in as an engineer,
you'd probably [INAUDIBLE] real programmer.
And then we actually do a sorting hat
with a serious effort to find a plausible place for what
you know.
We're assuming that you're a Ph.D. [INAUDIBLE].
And yes, of course there are courses
on this, that, and the other. [INAUDIBLE].
And your colleagues will contribute [INAUDIBLE]
but the goal is we can be doing something useful about a month
after you show up [INAUDIBLE] in courses and learning
[INAUDIBLE] how to make the elevators work.
But after that, we expect you'll be
doing something kind of useful in a month,
and be valuably up to speed in [INAUDIBLE].
And by useful, I mean a team [INAUDIBLE]
depending on you to actually do something. [INAUDIBLE].
MICHIEL BACCHIANI: More questions.
AUDIENCE: The problem between academic research
and corporate research is that every very smart idea
you guys have [INAUDIBLE] is the competitive advantage.
So How does that conflict with publishing?
GIDEON MANN: I'll take that one.
I had in mind to directly address this
when Michiel was talking about publishing.
I think that it is very clear that the fruits of all
this labor is material advantage.
And that it is valuable work, it's
important work, and that it makes
a difference at the company.
And so I think that everything that gets published,
people think about, OK, well, how much are we actually giving
away?
And I would say the things that are, in particular, I would
say Urs is probably the gatekeeper.
And if he feels like we're giving too much away,
he'll say, let's maybe either delay it
until the competitors have got in close enough,
or let's hide some of the most key pieces,
not so that it's useless, but just so
that we don't give away the bank.
And so, I think that, yes, clearly it has value,
and clearly we don't want to give away.
And so I think that.
But in my experience, to give an example,
we published a paper on distributed optimization.
And they didn't want us to directly say
the number of cycles, or the amount of time things
had taken up on that racks or on the machines.
So we converted it to some arbitrary scale,
and that was enough to satisfy people
that we weren't giving details away.
So I think that there's always room for negotiation,
even within that space.
AUDIENCE: [INAUDIBLE].
GIDEON MANN: So, my impression is that Google's general
position on patents is that they wish there weren't.
They didn't exist.
Google is involved in so much patent litigation
that I think it would be beneficial to the company
if the whole patent system was more evolved.
Maybe Stu has a slightly different perspective.
STU: True, but it ain't going away.
So yes.
Somebody will look to the publication [INAUDIBLE]
might have been patented and submitted, in which case,
some [INAUDIBLE].
We would be delighted to disarm, we just can't.
GIDEON MANN: But I think to--
STU: Have that conversation.
GIDEON MANN: But, to that point, I think that it's not--
STU: It rarely delays you more than a couple weeks
to actually get the lawyers [INAUDIBLE]
and [INAUDIBLE] probably one paper in 100 [INAUDIBLE].
GIDEON MANN: Oh, but I think part of the question was,
how important to your career is patent filing,
and I think it's--
STU: Very little.
GIDEON MANN: Yes,
MICHIEL BACCHIANI: There's a question over here.
AUDIENCE: Obviously, [INAUDIBLE] products here [INAUDIBLE].
So, excuse me, so [INAUDIBLE].
Because people somehow appear to be [INAUDIBLE].
EUGENE WEINSTEIN: Yes definitely.
So you mentioned 3,000, right?
So Google is, what, 50,000 now?
STU: Has 2500 PhD's, and nobody ever
uses the title unless they're a psychologist
or a lawyer or a medical doctor.
So unless you actually ask, you probably
won't know [INAUDIBLE] to get a PhD [INAUDIBLE]
actually pretty damn good.
AUDIENCE: [INAUDIBLE].
MICHIEL BACCHIANI: So I think it's not so much that it'll
give you an edge, let's say, in title
or in how much people respect your value, your opinion.
It won't get you that.
It's more the ability that you'll
have to bring that research kind of flavor to the work
that you do.
And I think in turn, because you will
be able to show that kind of skill,
in turn, that will get you the respect
and buy in from your colleagues.
It won't be the fact that you have the Ph.D. There was also
the adjustment aspect of the questions
OKSANA YAKHNENKO: I think one thing
that you'll learn as a Ph.D. Student,
and it's mostly because you spend, like, five years working
on one problem, is you really learn how to problem solve.
So it's not necessarily the title that gives you the edge,
but it's the fact that you, in the process of getting the PhD,
you learn how to solve problems.
So think about it.
If you come to Google, you get thrown
into an environment where most likely,
you will be working on something that is or is not related
to your Ph.D. Or to your research project directly,
you still know how to approach a problem, how to solve it,
how to evaluate your results.
And I think that's like a great skill to have, for anybody.
And I think most of people who actually work at Google
have that skill already.
So in terms of adjustment, I don't know.
GIDEON MANN: I'll speak to adjustment.
I was a post-doc, and I think my biggest adjustment coming here
was that Google is not very hierarchical.
And so, it's like when you're a graduate student or a post-doc,
you have your adviser, and your adviser
has a final say, to some degree.
I mean, about what you do.
It's really not like that here.
It's a very big company, there are
a lot of very different opinions,
and as an individual engineer, you
have a huge amount of autonomy and a huge of ability
to persuade that is kind of separate from your manager
and from your team.
So one of the ways that the engineering groups work
is that there's this interesting kind
of a triad that decides how things happen.
It's the manager of the team, it's the tech lead of the team,
and the PM, potentially, of the team.
And that triad is also subject to the whims
of all the people on the team.
And so it's like a very complicated process
how things get done.
And I think that understanding that model, which
is different from the academic model,
I think that is the biggest adjustment,
and I think a healthy adjustment.
I think academia is very rigid.
But I think that's a big adjustment.
OMAR BENJELLOUN: Just one more comment
I want to make on that question.
I think there's a very strong computer science
culture at Google, and engineering culture, but also
computer science.
So if you come with a view of problems
which is sort of academic or researchy,
or wants to think about problems in a fairly rigorous way
and model them property and solve them property and prove
that you have the right solution,
this is something that's very respected.
And so, I think at Google more than anywhere else,
doing things right and solving the problems sort
of at a fundamental level is something
that is valued and respected.
AUDIENCE: I have a a comment first and then a question.
[INAUDIBLE] question.
The comment, well, you asked, or somebody [INAUDIBLE]
the idea that people [INAUDIBLE] publishing.
Well, the fact is, I'm in software engineering.
I go to software engineering conferences.
I publish in software engineering.
And you see [INAUDIBLE].
I mean, it's not a value judgement,
it's just a fact if I look at the program.
So people come away saying, OK, there's
lots of PhD's at Google, there are
lots of smart people at Google.
They're not [INAUDIBLE] in software engineering.
What they do in software engineering [INAUDIBLE] Well,
why is that?
And so, I think that's where people get the idea.
It's a very pragmatic [INAUDIBLE].
MICHIEL BACCHIANI: The comment I have, and this
is my personal view of it, but I think
the panel said it to some extent.
But people join here, and it's a very different environment.
So they have this engineering team mates
that really want to get something going.
And at some point, you have the system out there,
and it serves people, and there's traffic going in,
and you have the signals coming back
from that which you want to optimize.
So to some extent it's sort of like taking a break
and doing the paper publications,
and sort of formally write it up and doing the measurements
in some kind of controlled way, which isn't usually frequent
the way that an application is deployed, where it's just,
you get more and more traffic.
A lot of the things are not very well controlled in that sense.
It makes it harder to do the paper
and sort of becomes a distraction.
I think they get so roped up into it.
AUDIENCE: I was saying [INAUDIBLE]
I was responding to the question that was asked, where do people
get this idea from?
I'm saying, that's where people get the idea from.
MICHIEL BACCHIANI: I mean, to some extent, I bring it up
because I think the misperception is that there's
some kind of corporate shut down on any kind of publication
because that's not what we value.
I think that's very unfair to address us like that.
AUDIENCE: OK.
So, my question for you, though, which is completely [INAUDIBLE]
is you spend a lot of time comparing Google to academia.
Now, I'm somebody who's, my father's attended
[INAUDIBLE] which, the minute I moved there, became AT&T Labs.
And I spent a lot of time at Google Research [INAUDIBLE].
So as somebody who spent time in AT&T Labs, I
ask you how you compare [INAUDIBLE] now,
of course, you went there in a different life
stage. [INAUDIBLE] According to this is a brand new PhD.
[INAUDIBLE] How would you compare life
at AT&T Research versus life at Google?
MICHIEL BACCHIANI: So, when I joined AT&T Research,
I gave you the context of getting
hired at AT&T. I remember I was done with grad school
and had lined up five or six interviews.
And the first interview I did was AT&T
because whoever was my contact there was very proactive
and got that going, so it was becoming the first interview
that I did.
And I visited one day and had the interview process.
And it was actually so much fun that I came back the next day,
and this is '99, right?
So the market is hot, I guess.
And they're worried00 I didn't even factor in my mind,
but they were eager to get people.
So the next morning, I get a phone call saying,
like, I still have to do the paperwork and the approval
or whatever, but you're going to get an offer.
At that point, I literally canceled
all my other interviews because I liked it that much.
So I loved AT&T Research When I joined,
it was really a fun place to be.
But I also felt that in my years there, I mean,
there were nasty things because the company was going down
the drain.
But AT&T Research in itself was
AUDIENCE: You have no idea how far down the drain--
MICHIEL BACCHIANI: But I mean, AT&T research in itself,
I think, was the greatest place.
Because all my colleagues there were famous,
so there was no attitude.
They was sort of like, do good work.
There was a lot of interesting people to talk to.
There was a collaboration within there,
but the rest of the company was alien.
In fact, I remember making fun of the rest of the company
where you'd have these town hall meetings,
and we didn't really understand what people are talking about,
because it's like, the rest of the company, to some extent.
So I think the contrast is, like, all of this
is AT&T research.
The entire company.
It's very open, collegial, sort of collaborative,
many aspects to it.
So I think that's the best comparison.
AUDIENCE: [INAUDIBLE].
GIDEON MANN: I'll take that.
And so in 2009, I had this project I wanted to work on.
I found a guy, [INAUDIBLE] Silverman,
who was at NYU HD program.
I convinced him that we should work on this,
and as part of the project, [INAUDIBLE]
in spreadsheets, machine learning,
he checked out the spreadsheet's curve, he built it locally
and he embedded our software into the spreadsheets.
I thought it was the coolest crap I'd ever seen.
And then he produced this demo which
showed what it would be like to have this code in actually
the spreadsheet.
It was like a different group, a group that
sat-- I think they may have at that point
sat on the same floor as we did.
But we didn't know them at all.
And so we built the prototype, he and I [INAUDIBLE]
mostly his work.
And then he showed the demo to the group,
and it was a very exciting moment,
because we're able to show them their software doing something
that they hadn't expected before.
And so that particular group kind of
laid dormant for a very long time.
And I think recently, we went back
and built a lot of those pieces in a different direction.
But I think that pathway is a very common pathway.
You see someone else's code, you check it out, you build it,
you modify it, you talk to them, you try to get their feedback,
you try to plan a way forwards, and it kind of moves this.
And that's just saying it's organic.
It's like, you can see just about any code in that company.
You can send an email to just about anybody
and they will respond.
Actually, the more senior are, the quicker they respond.
And I don't know why it works that way, but it seems to.
As far as collaboration, internal collaboration,
I think it is a wonderland.
It's fantastic.
people are very eager, they're very interested,
and it's a nice place for that aspect of it.
AUDIENCE: This one small anecdote.
I worked in IBM research for quite a while,
and IBM Research and IBM Software Group
were two completely separate entities.
And we had a prototype we wanted to build into the product,
and it was a nightmare to get their hands on code.
And there was envy, and protective, and politics.
I mean, here, we have access to almost all of the code,
you can talk to almost everybody,
mean there is not the question of what forums are there
for communication.
The company is the forum for communication.
OKSANA YAKHNENKO: And just to add to that,
we have tons of resources to even like, you know,
finding who's working on related projects.
For example, everybody has like an internal page,
and anybody in the company can actually
access your internal page.
All the projects have internal websites
with, like, members, what they work on,
what they're interested in, there's
a mailing list for that.
Then there are mailing lists on pretty much anything
you can imagine, so there's like a mailing list for machine
learning, and there's a mailing list
for some internal specific tools,
or like, mailing lists for some of the products
that are being developed.
It also happens from, well, in my experience
anyway, is that, when I joined the company,
I already knew a lot of people, not necessarily in New York,
but in other offices.
And just like, people I've met at conferences, or like people
who were in the same field.
And remote communication is like super easy.
You literally go to a conference room, you press the button,
and that person is on the screen in front of you.
Magic.
And then, as Gideon said, people are really, really willing,
and really supportive.
If you want to set up a meeting with somebody,
you don't even have to, like, sometimes you can just put it
on the calendar, and they will show up.
It's like, it's magic.
EUGENE WEINSTEIN: Also just to add one note about what venues
there are for fostering collaborations.
So we have a thing called beer and demos,
which is, it's on Friday, right?
Right, so you can show up to this room,
It's very much like this, and you
can have beer and watch people demoing their half working,
or quarter working, or it might work, kind of demos.
And if you have interest, if you're interested in the work,
then you can come up to them and create
a collaboration that way.
We also have millions of tech talks, people, both internal
and external speakers coming to present their work.
And you know, there are visiting faculty,
there are other people that are around
and want to share their ideas with you
and are interested in creating collaboration.
So there's many venues that we haven't thought of.
It Sorry?
MICHIEL BACCHIANI: So you could be needing for 10 minutes
already, so any questions that you think
are really, really important to be answered by the panel rather
than the person that will probably
be sitting at your table?
AUDIENCE: Ask a question?
MICHIEL BACCHIANI: Yes absolutely
AUDIENCE: [INAUDIBLE].
STU: I suspect that some of our offices
in other countries [INAUDIBLE].
MICHIEL BACCHIANI: One of my favorite quotes
is, some people in other places, probably for good reasons.
Final question?
AUDIENCE: So, [INAUDIBLE] it's most important, in response
to the [INAUDIBLE] I was wondering
what the extent of that was and how common [INAUDIBLE].
UMESH SHANKAR: Well, in some sense,
it's part of the price of success.
You build something and it's not being used,
you don't pay the tax, but that's not really
where people want to end up either.
And part of it, too, is I think you sort of build
in the cost of maintenance to your utility function
of how good your thing is.
Where that just becomes part of how
you decide if a system is good.
Right?
It's not too good, even if it just works,
or even if it works well.
It's how easy is it to modify over time?
How easy is it to maintain over time?
How much of your resources or your team's resources
is it taking?
And so, you get better at it.
Sort of naturally, things often evolves to a state
where you spend your time either on doing new stuff
or making the things that you already have more efficient
and taking less maintenance burden so you
get to a good state again.
So yeah, it definitely various, I think.
Working on infrastructure, your definition of success
is making people that use your stuff successful in a sense,
and that's what my team's work on
is largely in the infrastructure area.
So we probably pay a larger tax that most,
but some of what-- I sort of alternate
between calling it a tax and learning.
Because a lot of the times, the things
that we spend a lot of time with are basically customer service.
So a new team is integrating with some technologies
that we've built.
We've built security and privacy infrastructure,
and they'll have a novel scenario, right?
They have different data access patterns from ones
we've seen before, they're doing some mash up
of kind of more internal facing systems
with externally facing systems, and that's a new thing,
and they're querying data in a different way,
all this kind of stuff.
OK, we haven't seen it before, right?
And so it's like, ah man, we've got
to put down the code that we're writing
and help them figure out the right thing to do.
Maybe we have to change our code,
maybe they have to change their design,
maybe we just come up with something clever.
But the net effect of those things
is that, when you design the next version of your system,
you've already taken those into account.
So I think maybe I was a little harsh
in calling it always a tax, exactly.
There is certainly an element of that, right?
You end up doing some stuff kind of a lot, and it's like ugh.
Got to update something again, right?
Or some machine broke somewhere and now
you've got to fix something.
So that kind of stuff happens, right?
But I said over time, the trend is towards, OK, well,
if some machine broke for the 10th time
and I'm having to spend the 10th time to fix it,
maybe I need to engineer around that problem.
And now you have a much more robust system too.
And in terms of customer service,
maybe you build your system in a way
that people don't have to ask as many questions,
and it's just more natural and easy for them to use.
And that, too, is progress, so yeah, you
try to find a good equilibrium.