ELIE BURZSTEIN: If you are new to our session

on cutting edge TensorFlow, my name is Elie.

And today with Josh, Mike, and Sophien,

we have four exciting projects for you.

First, I will talk about Keras Tuner.

Then, Josh will talk about probability programming.

Next, Mike will talk about TF-Ranking.

And finally, Sofien will show you how to use TensorFlow.

Graphics

Let me start by telling you about Keras Tuner, which

is a framework we initially developed internally

to bring new [INAUDIBLE] model to faster to production.

Before getting started, let me ask you a question.

How many of you have ever spent time optimizing your model

performances.

Right, I see quite a few hands raised.

This is not surprising because getting the best model

performances is not easy.

Getting the best model performances

is hard because there are many parameters such as the learning

rate, the batch size, and the number of layers

which influence the model performances.

Moreover, those parameters are interdependent,

which make finding the optimal combinations by hand

very challenging.

This is why we rely hypertuning to automatically find

the best combinations.

However so far, hypertuning have been

known to be not easy to use.

So if hypertuning is essential to get

the optimal performances, this begs the question,

can we make it easy as one, two, three?

And the answer is yes.

This is why today I am happy to introduce

to Keras Tuner which is a tuning framework made for humans.

Keras Tuner is a tuning framework

designed to make the life of AI practitioner

as easy as possible.

It also helps hypertuner algorithm creators and model

designers by providing them with a clean and easy to use API.

For AI practitioner, which is most of us,

Keras Tuner makes moving from a base model

to a hypertuned model quick and easy.

Let me show you how this is done by converting a basic MNIST

model to a hypertuned one.

We will only have to change a few lines of code.

Here's our basic MNIST model that is a TensorFlow

Keras controlled API.

This is something I'm sure all of you have seen already.

As you can set on the side, all parameter are fixed.

For example, our learning rate is set to 0.001.

So let's transition it to a hypertunable model

in three steps.

First, we wrap up our model in a function.

Second, we define hyper-parameter ranges

for the parameter that we would like to optimize.

Here for example, we're going to optimize the learning

rate which is one of the most important parameters

to hypertune.

Finally as the last step, we replace our fixed parameters

with our hyper-parameter range.

And we're done.

Our model is ready to be hypertuned.

This is as easy.

Besides offering an intuitive API Keras Tuner,

we'll also provide you with state of the art hypertuning

algorithm, tuneable architectures which

are ready to go, and an automatic experimental

recording which make it easy for you to analyze, share,

and reproduce your results.

Originally, I started developing Keras Tuner to quickly try it

on new models for [INAUDIBLE] purposes

including the one we developed to protect Gmail

against malware and phishing.

Band detection is one of the core building

block we need to protect your inbox against phishing email

that impersonates the brand you love.

This is why today, our [INAUDIBLE] example

would be to show you how to build a simple and yet

accurate brand logo classifier as logo identification is

one of the critical components to detect brand spoofing email.

The first thing we need to do is to load our dataset.

In that example, we have about the 150 icons

from various brands, including the ones displayed on the side.

We also need to set a few variables

such as batch size and the number of icons

we're going to use [INAUDIBLE].

Next, our data set is very small so we'll

rely on data augmentation to create enough training data.

This slide shows you a few examples of the augmented icons

we're going to feed to the classifier as training

input, the output being the brand names.

We are going to save the real icon as our validation dataset

to make sure that our classifier degenerates well.

To establish a baseline, let's first

train a ResNet101v2 which is one of the most common and well

known model architectures.

As you can see on the [INAUDIBLE] graph,

our model did converge but the actuality on real icon

is not great.

We barely reached 79% accuracy.

And it's also quite big with 44 million parameters.

Well, that's OK.

We can use Keras Tuner to find a better model which

is smaller and more accurate.

So to do that, the first thing we need to do

is to create, as the MNIST model a model function and input

TunableResNet.

TunableResNet is a tunable version of ResNet

that we will provide with Keras Tuner

as one of the architectures which are ready to tune.

Next, you add a few layers on top of it

and combine the model.

Then well, we initialize the tuner

and we give it $500 to spend on tuning to find the best model.

And we ask it to maximize evaluation accuracy.

Finally, we have to launch the tuning

and wait for the results.

So did it work?

Well, yes it did.

Actually, Keras Tuner found a way better model.

Our new model have now 100% accuracy

and only takes 24 million parameters.

So we get a faster and more accurate

model thanks to hypertuning.

Keras Tuner works with many of the tools you love,

including TensorBoard, Colab, and BigQuery,

and many more to come.

One more thing.

Hypertuning takes a long time so to make everything

more convenient, we will be releasing

alongside with Keras Tuner an optional cloud service that

will allow you to monitor your tuning on the go

whether it's from your phone or from your laptop.

Here is a screenshot of an early version of the mobile dashboard

to give you a sense of what to expect.

So the design is not final, but the UI

will show you how long before you're tuning complete,

as well as offer you a visual summary of the model trained

so far so you can know how your tuning is going.

Thank you for attending today.

We are really excited about Keras Tuner.

And you can sign up today for the early access program

by heading to g.co/research/kerastunereap.

[APPLAUSE]

Josh is now going to talk about [INAUDIBLE] programming

and TensorFlow.

JOSH DILLON: OK.

Hi, I'm Josh.

And today, we'll be talking about everyone's

favorite topic-- probability.

So let's just start with a simple example.

So suppose we're trying to predict these blue dots--

that is, the Y-coordinate from the X-coordinate.

And Keras makes this pretty easy to do.

As you can see here, we have a dense neural network

with one hidden layer outputting one float.

And that float is the predicted Y-coordinate.

And we've chosen mean squared error as a loss, which

is a good default choice.

But the question is, how do we make our loss function better?

What does a better loss function look like?

And how would we even know?

How would we evaluate the fit of this model?

And so we would like to be able to specify the negative log

likelihood as a generic loss, but then

encode the distributional assumptions in the model.

And furthermore, if we're doing that,

then wouldn't it be nice to get back

an object for which we can query--

ask for the mean, the variance, the entropy, et cetera.

And the answer is we can do this.

Using a TensorFlow probability distribution layer,

we can say that we want the neural network to output

a distribution, basically.

And the way this works is as you can

see, the second to the last layer here outputs one float.

That float is interpreted as the location or mean

of a normal distribution.

And that's how we can implement linear regression.

And as you see here, the fit is this sort of red line.

But what's cool is, we're actually

outputting a distribution.

Right?

So you can take this output and just

ask for the entropy or the variance of the prediction.

So that's nice.

But now that we've done this, there's

sort of something that looks a little fishy here.

We're learning the mean, but not the standard deviation.

It seems like maybe a missed opportunity

to improve our model.

And that missed opportunity is now self-evident

that we're learning a distribution directly-- sort

of an idea that was hidden in what was otherwise

the mean square error.

So to learn standard deviation, it's just

another one or two line change.

Instead of outputting one float, we now output two.

One is interpreted as the mean as before, the location.

The other when passed through a soft plus function,

is now interpreted as the standard deviation.

And what you see is we're now able to get

this sort of green line and the red lime--

green being the standard deviation,

red being the mean fit from before.

As you can see, sort of the green line

diverges as X increases.

And so that suggests that our data--

the variability of Y, actually-- changes

as a function of X. Statisticians

call this heteroskedasticity.

But I'd just like to think of it as learning known unknowns.

There was variability present in our data.

And because we took a more probabilistic view

of our model, it was pretty easy to see how we should fit that.

So that seems pretty cool.

But I guess the question is, now that we're

thinking about sort of known unknowns

or aleatoric uncertainty, what about unknown unknowns?

Do we even have enough data to accurately make the claim

that this is the standard deviation

and mean of this regression problem?

And if we don't, how might we get there?

And the answer is--

or an answer-- is to be Bayesian.

Rather than to just fit the weights, if instead we

think about weights is being drawn from a distribution

and try to find what might be the best posterior

distribution, then we can actually

capture some degree of unknown unknowns.

That is, keeping track of how much evidence

we have or don't have to make the prediction we want to make.

So in practice, this boils down to something

that looks a lot like learning an ensemble.

As you can see here, there are numerous random

draws each corresponding to a line.

But what's cool is computationally,

we only pay a small additional overhead for fitting

what is otherwise an infinite number of models.

So that seems pretty cool.

But we seem to have lost the aleatoric uncertainty--

the known unknowns.

And so can we get it back?

Yes.

Since all of this is modular, you

can simply specify whatever assumptions you want to make.