ALEXANDRE PASSOS: I'm Alex, and I'm here

to tell you about how you're going to build graphs

in TensorFlow 2.0.

And this might make you a little uncomfortable,

because we already spent quite some time earlier today

telling you that in TensorFlow 2.0,

we use eager execution by default.

So why am I taking that away from you?

And I'm not.

You still have your eager execution by default,

but graphs are useful for quite a few things.

The two ones that I care the most about personally

are that some hardware, like our TPUs,

really benefit from the kind of full program optimization

that we can get if we have graphs.

And if you have graphs, you can take your model

and deploy it on servers and deploy it on mobile devices

and deploy it on whatever thing you want,

make it available to as many people as you can think of.

So at this point, you're probably, eh, I

remember TensorFlow 1.0.

I remember the kind of code I had to write to use graphs,

and I wasn't proud of it.

Is he just going to tell me that I have to keep doing that?

And no.

One of the biggest changes we're doing with TensorFlow 2.0

is we're fundamentally changing the programming

model with which you build graphs in TensorFlow.

We're removing that model where you first add

a bunch of nodes to a graph and then rely on session.run

to prune things out of the graph,

to figure out the precise things you want

to run in the correct order and replacing it

with a much simpler model based on this notion of a function.

We're calling it tf.function, because that's the main API

entry point for you to use it.

And I'm here to tell you that with tf.function,

many things that you're used to are going to go away.

And I dearly hope you're not going to miss them.

The first one that goes away that I really

think no one in this room is going to miss

is that you'll never have to use session.run anymore.

[APPLAUSE]

So if you've TensorFlow with eager execution,

you know how it works.

You have your tensors and you have your operations,

and you pass your tensors to your operations,

and the operations execute.

And this is all very simple and straightforward.

And tf.function is just like an operation except one

that you get to define using a composition

of the other operations in TensorFlow however you wish.

Once you have your tf.function, you can call it.

You can call it inside another function.

You can take its gradient.

You can run it on the GPU, on the TPU, on the CPU,

on distributed things, just like how

you would do with any other TensorFlow operation.

So really, the way you should think about tf.function

is we're letting you define your own operations in Python

and making it as easy as possible for you to do this

and trying to preserve as many of the semantics of the Python

programming language that you already know and love

and when you execute these functions in a graph.

So obviously, the first thing you would think

is that is it actually faster?

And if you look at models that are

large convolutions or big matrix multiplications,

large reductions, it's not actually any faster,

because you get executions plenty fast.

But as your models get small, and as the operations in them

get small, you can actually measure

the difference in performance.

And here, I show that for this tiny lstm_cell with 10 units,

there is actually a tenfold speed up

if we used tf.function versus if you don't use tf.function

to execute it.

And as I was saying, we really try to preserve as much

of the Python semantics as we can

to make this code easy to use.

So if you've seen TensorFlow graphs,

you know that they are very much not polymorphic.

If you built a graph for float64,

you cannot use it for float32 or, God forbid, float16.

But tf.function-- but Python code

tends to be very free into the types of things it accepts.

With tf.function, we do the same.

So under the hood, when you call a tf.function,

we look at the tensors you're passing as inputs

and then try to see, have we already made

a function graph that is compatible with those inputs?

If not, we make a new one.

And we hide this from you so that you can just

use your tf.function as you would use normal TensorFlow

operation.

And eventually, you'll get all the graphs you need built up,

and your code will run blazingly fast.

And this is not completely hidden.

If you want to have access to the graphs

that we're generating, you can get them.

We expose them to you.

So if you need to manipulate these graphs somehow or do

weird things to them that I do not approve,

you can still do it.

But really, the main reason why we changed this model is not

to replace session.run with tf.function,

it's that by changing the promise for what

we do to your code, we can do so much more for you

than we could do before.

With the model where you add a bunch of notes to a graph

and then prune them, it's very hard for the TensorFlow runtime

to know what order do you want those operations to be executed

in.

Almost every TensorFlow operation is stateless

so that doesn't matter.

But for the few ones where it does matter,

you probably had to use control dependencies

and other complicated things to make it work.

So again, I'm here to tell you that you will never

have to use control dependencies again

if you're using tf.function.

And how can I make this claim happen?

So the premise behind tf.function

is that you write code that you'd like to run eagerly,

we take it and we make it fast.

So as we trace your Python code to generate a graph,

we look at the operations you run,

and every time we see a stateful operation,

we add the minimum necessary set of control dependencies

to ensure that all the resources accessed

by those stateful operations are accessed in the order you

want them to be.

So if you have two variables and you're updating them,

we'll do that in parallel.

When you have one variable and you're updating it many times,

we'll order those updates so that you're not

surprised by them happening out of order or something

like that.

So there's really no crazy surprises

and weird, undefined behavior.

And really, you should never need

to explicitly add control dependencies to your code.

But you'll still get the ability of knowing

what order things execute.

And if you want something to execute before somebody else,

just put that line of code above that other line of code.

You know, how you do in a normal program.

Another thing that we can dramatically

simplify in tf.function is how you

use variables in TensorFlow.

And I'm sure you've all used variables before.

And you know that while they're very useful--

they allow you to share state across devices,

they let you persist, checkpoint,

do all those things, it can be a little finicky.

Things like initializing them is very hard, especially

if you're using your variables of any kind

of non-trivial initialization.

So another thing that we're removing from TensorFlow

is the need to manually initialize variables yourself.

And the story for variables is a little complicated,

though, because as you try to make

code compatible with both eager execution and graph semantics,

you very quickly find examples where

it's unclear what we should do.

My favorite one is this one--

if you run this code in TensorFlow 1.x,

and you session.run repeatedly, the result,

you're going to get a series of numbers that goes up.

But if you run this code eagerly,

every time you run it, you're going

to get the same number back, because we're

creating a new variable, updating it, and then

destroying it.

So if I wanted to turn this code-- wrap

this code with tf.function-- which one should it do?

Should it follow the 1.x behavior or the eager behavior?

And I think if I took a poll, I would probably

find that you don't agree with each other.

I don't agree with myself, so this is an error.

Nonambiguous at creating variables,

though, is perfectly OK.

So as you've seen in an earlier slide,

you can create the variable and capture it

by closure in a function.

That's a way a lot of TensorFlow code gets written.

This just works.

Another way you can do is like write your function such

that it only creates variables the first time it's called.

This is incidentally what most libraries in TensorFlow

do under the hood.

This is how Keras layers are implemented,

how the TF 1.x layer is implemented,

Sonnet, and all sorts of other libraries

that use TensorFlow variables.

They try to take care to not create variables

every time they're called, otherwise you're

creating way too many variables, and you're not actually

training anything.

So code that behaves well just gets turned into function,

and it's fine.

And if you've seen this, I didn't actually

need to call the initializer for this variable

that I'm creating, and it's even better.

I can make the initializer depend

on the value of the arguments to the function

or the value of other variables in arbitrarily

complicated ways.

And because we control--

we add the necessary control dependencies

to ensure that the state updates happen in the way you

want them to happen.

There is no need for you to worry about this.

You can just create your variables,

like how you would use in a normal programming language.

And things will behave the way you want them to behave.

Another thing that I'm really happy about tf.function

is our autograph integration.

And if anyone here has used Control Flow in TensorFlow,

you probably know that it can be awkward.

And I'm really happy to tell you that with Autograph,

we're finally breaking up with tf.cond and tf.while_loop.

And now, you can just write code that looks like this--

so if you see here, I have a while loop,

where the predicate depends on the value of a tf.reduce_sum

on a tensor.

This is probably the worst way to make a tensor

sum to 1 that I could think of.

But it fits in a slide.

So yay.

If you put this in a tf.function,

we'll create a graph and we'll execute it.

And this is nice.

This is great.

But how does it work?

Under the hood, things like tf.cond and tf with our while

loop are still there, but we wrote this Python compiler

called Autograph that rewrites Control Flow expressions

into something that looks like this--

yes.

Something that looks like this, which is not