And I'm here to tell you about something cool

that we've been working on in the TensorFlow team

for the past few months.

This is mostly work done by Google Brain in Cambridge.

And I'm here presenting because, well, I'm not in Cambridge.

So this project is called AutoGraph.

And what it is, is essentially, a Python compiler

that creates TensorFlow graphs for you.

So I'm sure, since you walked into this room,

you probably used TensorFlow before.

And you might have some idea about how it works.

And the way it works, under the hood,

is you build this graph to represent your computation.

And here, there is a multiply, an add, and a relu.

This is a very simple neural network.

And the nice thing about having a TensorFlow graph

is that once you have this graph,

you can deploy it to a server.

You can train it on a TPU.

You can run it on your mobile phone with TensorFlow.js.

You can run it on a Raspberry Pi.

There's all sorts of things you can do with it.

You have-- we have compilers that take these graphs

and produce highly-optimized code

for all sorts of platforms.

The downside of this graph is that you end up

writing code like this, which you see five lines of code,

one of which has the actual computation that you

want to run.

And in case you're too far to read, it's the fourth one.

Everything else is boilerplate.

And some of this boilerplate is easy to deal with,

like these place holders.

But some of the boilerplate is not.

For example, if you want a TensorFlow model that

has control flow inside the model,

you have to take your mental idea of what code

you want to write, which probably looks

like the example on the left, where you have a Python "if,"

and turn it into the code on the right, which is what

we need to generate a graph.

And the reason why we need this is

that our computation graph, because it can be put on a TPU,

and put on a phone, and run independently

of the code that produced this graph,

has to have information about both branches

of your conditional.

So we need to somehow tell TensorFlow everything that

could possibly happen in this program.

And so you end up with code that's

very redundant in this way.

And we have answers for this.

And we're working hard to make your life easier

if you want to use TensorFlow.

One project I've spent a lot of effort

personally on is eager execution.

And the idea of eager execution is

to make your life easier by saying,

you don't have to do graphs at all.

And you can take this idea fairly far.

But eventually, you hit some limits.

Right?

If you're-- with eager execution enabled,

you have a very intuitive interface.

You just write Python code.

You use Python control flow, Python

infrastructures-- all the things you're already familiar with.

And it's nice.

But you don't get this automatic parallelism and distribution

and scale that Frank was telling you

about with all the TPU pods, because you still

have to run Python code.

And processors aren't getting faster.

And Python is inherently single-threaded.

We have a few ways of getting around this.

Like, from eager execution, we have

defun-- this primitive that lets you build little graphlets

and run them.

So within that graphlet, you have all the performance

of graph mode.

But you still can have eager execution in your outer loop.

We can also have py_func, which lets

you have full graph code that can run on clusters

of multiple GPUs, but with eager code

in the middle that's doing some complicated control flow stuff.

And these tools are useful.

Both defun and py_func are differentiable.

They run on all sorts of devices.

They give you a lot of flexibility.

But it's a little unsatisfying that you have

these seams in your program.

And you need to think about, what's going to be a graph?

What's not going to be a graph?

What if we could do better than this?

And AutoGraph is this tool that we've been working on,

in the Google Brain Cambridge office, that

tries to do better than this.

And what it lets you do is write eager-style, Python code

and have a compiler turn this into the boring, very

complicated, graph-style code.

Because if you think with me for a second,

if you look at the transformations that

need to happen to turn the code on the left

into the code on the right, that looks like something

that can be automated.

And AutoGraph is, indeed, the thing that

automates this transformation.

It's pretty easy to use now.

It's in TensorFlow.contrib.

And when 2.0 comes out, it will be in Core.

You just import AutoGraph from contrib.

And you decorate your function with autograph.convert.

Then, when you call it, under the hood,

AutoGraph will generate and run this more complicated code

for you.

And these are not academic concerns.

There's a lot of code out there that has control flow in it.

So for example, the implementation

of dynamic RNN in TensorFlow is very hard

to read if you're not familiar with how TensorFlow works.

But I can write one, here, in the slide.

And it's pretty easy.

If you squint, you have a "for" loop

that loops over my sequential data, applies an RNN cell,

and has some logic in it to mask the outputs,

because the different sequences in the minibatch

can have different lengths.

But this is not--

this is 11 lines of code.

There's no magic here.

But if you run it through AutoGraph,

you get the actual code that you see inside tf.dynamic_rnn,

which is substantially harder to read.

And not just RNNs--

in this-- I mean, this RNN is a basic building block

that we use in all sorts of sequence-to-sequence models,

when we're generating text, when we're generating images,

when we're turning data into other data.

And this is just one example of this kind

of control-flow-rich programs that you often want to write,

but that writing in normal Graph TensorFlow

can be more painful than it needs to be.

So what AutoGraph supports is a subset of the Python language

that covers what I hope is what you most need

to run in a TensorFlow program.

So we can do print.

And I don't know how many of you have ever

failed to use tf.print, because I do about once a week.

It's really hard, right?

Also tf.assert-- it's very easy to have your assertion

silently dropped in the void.

AutoGraph is also composible.

So you can have arbitrary Python code,

with classes and functions and complicated call trees.

And it will turn the whole thing into a graph for you.

You can have nested control flow, with breaks and continues

and all sorts of stuff.

And we can make that work.

And we're still working on it to make it better.

So another nice example, I think,

to visualize what AutoGraph is actually doing, is using what

has been unanimously decided to be the best interview

question in the history of humanity, which is FizzBuzz--

this thing where you loop over the numbers.

And you print "Fizz" for some of them, "Buzz" for some of them,

"FizzBuzz" for some of them.

And otherwise, you just increment your counter

and print the number.

And you can see this is like 10 lines of Python code,

pretty straightforward.

But you should try to turn it into in TensorFlow code.

And we ran it through AutoGraph, because I'm too

lazy to write this code myself.

It's not pretty.

And if you think this is it, this is not.

This is it [LAUGHS].

So this all looks nice.

But I think you would be remiss to believe me if you didn't

understand how this works.

So I want to spend a few minutes and tell you

what we're actually doing to make this possible.

And there's many ways of conceptualizing this.

But I think, really, the core of AutoGraph

is we're extending the notion of what's possible with operator

overloading.

And I don't know if you're familiar with operator

overloading.

It's a feature in Python that lets you customize

a little bit of the language.

And we use this heavily in TensorFlow.

So for example, when you write c = a * b in TensorFlow,

we don't actually run the Python multiplication operator.

What actually runs is the thing on the right,

c = tf.multiply(a, b).

And the way we turn one-- the code on the left

into the code on the right is by having this Tensor class,

where a and b are tensors.

This Tensor class defines this _ _nul_ _ method, which Python,

then, when it sees the operator, it rewrites to a call

to the multiply method.

And then, we can use this multiply method

to make whatever code we want run when you type the star.

And this is pretty straightforward

and shouldn't be big of a surprise.

But what Python doesn't let you do is this.

There's no way in Python to override

the behavior of the "if" operator

to run some code you want to run.

So ideally, if Python let us override _ _if_ _,

we would be able to make all sorts of graph rewrites that

take control flow into consideration.

Sadly, we can't.

So what we're trying to do, really,

is overwrite the syntax that Python doesn't

let us override the syntax.

And the way we do this is we read the Python abstract syntax

tree of your code.

We process it using Python's [INAUDIBLE] modules.

And we just run a bunch of standard compiler passes.

We rewrite loops so that they have a single exit.

We capture all the variables that you assign

in loops and conditionals.

We unify the variables that you mutate in either branch

of a conditional.

We do all these boring transformations

that need to happen, so that we can take the code that you want

to write, that has advanced control flow,

into the code that TensorFlow wants you to write.

And if you've been following me so far,

you might have a question in your head, which

is that, if you've written TensorFlow program--

AutoGraph didn't exist.

You look at your code.

It's full of "ifs," and "while" loops, and "for" loops,

and things like that.

And you probably don't want any of those things

to end up in a graph, because a lot of this

is just configuration.

You're choosing what optimizer to use.

Or you're choosing how many layers your network has.

And you have-- these things are very easy to express in loops.

But you might think that if we rewrite

every loop and conditional in your code

to be a TensorFlow loop or conditional,