Welcome to this Tensorflow session about rain enforcement.

Luring s So this is you, Han.

I'm Martin.

And today we would like to build a neural network with you.

Do you know about neural networks?

If I say a serious question, if I say soft max or across entropy, raise your hand if you know what that is.

E like 1/2 1,000,000,000.

Okay.

Very quick primer.

This is a neural network.

Okay, Layers off neurons.

Those neurons, they always do the same thing they do awaited some off all of their inputs, and then you can layer them in layers.

The neurons in the first layer they do awaited some off.

Let's say pixels.

If we are analyzing images, then you don't see in the second layer will be doing weighted.

Sum some off outputs from the first layer.

And if you're building a classifier, let's say here probably you want to classify those little square images into airplanes and no airplanes.

You will end up on the last layer, which has as many neurons as you have classes.

Yeah, if those weights or configured correctly, we'll get to that that one of those neurons will have a strong output on certain images and tell you this is an airplane or the other one we have.

We'll have a strong output that you will tell.

You know, this is not an aeroplane.

Okay, there is one more thing in this.

There are activation functions.

So a little bit more details about what a neuron does for those who like it.

You have the full yips right here.

You have the full transfer function here.

So you see, it's a weighted sum.

Plus something.

Go, Tobias.

That's just an additional degree of freedom.

And then you feed this through activation function.

And in your own networks, that is always a non linear function.

And to simply five for us here on Lee to activation functions that counts.

So for all the intermediate layers, the white laters.

Ready?

That's the simplest function you can imagine.

You have it on the graphic.

Okay, just this.

It's nonlinear.

We love it.

Everyone uses that.

Let's not go any further on the last layer, though.

If we are building a classifier, typically what you use is the soft max activation function, and that is a new exponential, followed by a normalization.

So here I have a different classifier that classifies in 10 classes, and you have your waited sums coming out of your 10 final neurons.

And what you do in Soft Max is that you elevate all of them to the exponential, and then you compute the norm off that vector of 10 elements, and you divide everything by north.

The effect of that, since exponentially it's very steeply increasing function is that it will pull the winner apart.

I made a little animation to show you like this.

This is after South Max.

This is before some flex.

So yours, you see much clearer which off those neurons is indicating the winning class.

Okay, that's why it's called soft plaques.

It pulls the winner and parks like Max, but it doesn't completely destroy the rest of the information.

That's why it's a soft version of Max.

Okay, so just those two activation functions with that, we can build the stuff we want to build.

So now, coming out over a neural network, we have our let's say, here, 10 final neurons producing values which have been normalized between zero and one.

We can say those values are probabilities.

Okay, The probability off this image being in this class, How are we going to determine the weights in those weighted sums?

Initially, it's all just, you know, random.

So initially, our network doesn't do anything actually so useful.

We do this through supervised learning.

So you provided images which you have labeled beforehand.

You know what they are.

And on those images, your network is going to output this set of probabilities.

But you know what the correct answer is because you are doing supervised learning, so you will include your correct answer.

In a format that looks like what the network is producing, it's the simplest encoding you can think of.

It's called one harting coating, and basically it's a bunch of zeros was just 11 in the middle at the index off the class you want so here to represent the six, I have a vector of zeros with the one in the sixth position, and now those vectors were very similar.

I can compute the distance between them and the good of the people who studied this in a classifier, they tell us, don't choose any distance used across entropy distance.

Why, I don't know.

They're smarter than me.

I just follow in the interest of the cross entropy distances computed like this.

So you multiply element by element wth e elements of the vector, the known answer from the top by the logarithms off the probabilities you got from your neural network and you then sum that up across the fact this is the distance between what the network has predicted and the correct answer.

That's what you want.

If you want to trade a neural network, you get that it's called a natural function or lost function.

From there on, Tensorflow can take over and do the training for you.

We just need another function.

So in a nutshell, those are the ingredients in our part that I want you to be aware off.

You have neurons.

You don't do waited sums.

You have only to activation functions that you can use either the Railly activation function on intermediate layers or if you build a classifier on the last layer.

Soft max.

And the other function that we are going to use is the cross entropy error function that I had on the previous line.

Okay, all good on this bit of code, this is how you would write it in Tensorflow.

There is a high level A P I in Tensorflow called layers where you can in Stan.

She ate an entire layer at once.

You see, I mean, Stan, she the first layer here has 200 neurons and is activated by the radio activation function.

It's this layer here and this also in Stan.

She hates the weights and biases for this layer in the background.

You don't see that it's in the background.

I have a second later here, which is this one is just 20 neurons, again ready activation function on this.

I need to deal.

Do this little final layer, so it's a dense layer again with two neurons, and even if you do don't see it on the screen, it is activated by soft max, but you don't see it because I use its output in the cross entropy function, which has so slacks built in.

So it is a self next layer.

It's just that you don't see Ridge itself backs in the coat.

Finally, this here is my better function, which is the distance between the correct answer here.

One hot encoded and the output from my neural net.

Once I have that, I can give it to tensorflow taken optimizer.

Ask it to optimize this loss and the magic will happen.

So what is this?

Magic Tensorflow will take this at our function.

Differentiated relatively to all the weight's on All the bios is all the trainable variables in the system, and that gives it something that is mathematically pulled, a great agent.

And by following this radiant, it can figure out how to adjust the weights and biases in the neural network in a way that makes this error smaller.

That makes the difference between what the network predicts and what we know to be true.

Smaller.

That's supervised Lord.

So that was the primary.

Now why?

What do we want to build today?

We would like with you to build a neural network that plays the game off punk.

But just from the pixels of the game, it's notoriously difficult to explain to a computer the rules of the game and the strategies and all that.

So you wouldn't you want to do way of all that and just get the pixels and find some learning algorithm that will where it will learn to play his game.

And of course, that is not a goal in itself.

Because figuring out how to program a paddle to win punk is super easy.

You just always stay in front of the ball, you know, and you win all the time.

And actually we will train again.

Such a computer controlled agent.

The goal here is to explore learning algorithms because this has applications, hopefully way beyond punk.

So this looks like a classifier.

We just learned how to build a classifier.

You know, we have the big cells, We will build a neural network and it has three possible outcomes.

This is a position in which you want to go up, Stay still or you want to go down.

Okay, let's try to do this by the book in a classifier.

So we have a single intermediate layer off neurons activated with ready off course, and then a last layer withers three neurons activated by soft max.

We use the cross entropy loss, so you have the function here.

It's the distance between the probabilities that this policy network, it's called the Policy Network, predicts probability off, going up still or going down in the correct move.

Uh, but what is the correct move?

I don't know.

You know, you have no Martha, your executive right here.

Unlike in the supervisor endings problem.

We don't know what's correct.

Move to play here.

However, the environment requires that we keep making moves off going up or stains the or moving down to progress again.

So what we're gonna do is work on the same photo, which means picking one of the three possible moves off, moving a pedal up since the or going down randomly, but not completely randomly, though we're going to pick the movie based on the output of the network.

So, for example, the network stays the output for ability off going up 0.8.

Then we should make it so that there's 80% of chance we choose to move up as our next.

All right, so that we know how to play the game.

The policy, that's what gives us probabilities were all along loaded dice and pick from that.

And we know what next move toe Blake.

But initially, this network is initialized with random waits, so it will be playing random moves.

How is that?

How does that inform us about the correct move to play.

I need to put the correct moving my formula.

That's right.

So in this case, we really want to correct move to mean moves that will lead to winning.

You don't know that until someone has scored a point.

And so that's what we're gonna do only when someone, either our pedal with abundant pedal, has scored the point.

Do we know we have played well or not?

So whenever somebody stores were going to give ourselves a reward, if our repair those scores, the point will give ourselves a positive one reward point.

And if the opponent paddles florist that we give ourselves a negative one reward points, and then we're going to structure our lost function slightly differently than before.

Um, over here once again, over here you see some aloe sounds very much like the crows, and she'll be function.

We still before now in the middle.

Here is the main difference.

Where, instead of the correct labelling a supervised learning problem, we're just going to put the simple the movie in there move.

We played in the movie that we're having to play and well, some of the moves removes the helpers.

And so that's why every lost, long lost value is multiplied by the reward out front.

This way, moves that eventually lead to a winning point will get encouraged and moves that leads.

We're losing.

Point will be discouraged over time.

Okay, so you do this for every move.

I can see how, with this little modification, it could lead to some learning.

But up putting back my mathematicians hat on.

I see a big problem here.

You have the sample move.

That is a picking operation, a sampling operation.

You pick one out of three that is not defensible and to apply, you know, the minimization and radiant descent and all that.

The last function must be different.

Chewable.

You're right, Marty.

I see the hard question here.

Simple move here.

It actually does depend on the models, ways and biases.

But the simple in operation is not defensible.

And so we're not going to differentiate that.

Instead, work going to look at a simple moves us if they're Constance and then we play many games across many, many moves to get a lot of data treating all of these simple moves as if their constant comes the labels and only differentiating the probabilities that our output by the model in blue here on the screen in those probabilities directly depend on the models, ways and biases.

And we can differentiate those with respect to the ways of bias is this way.

We still get a very radiant and can apply grading descent techniques.

Oh, okay.

So you kind of cheated the part that is problematic.

Just regarded as constant.

You're going to blame any games with the same neural network.

Accumulate those plate moves, accumulate those rewards.

Whenever you know that you scored a point, you accumulate those rewards and and then you plug that in and you only different shade relatively to the predicted probabilities.

And yes, you're right.

That still gives you a radiant.

That depends on weights and bios is we should be able to do that.

Okay, I get it.

This is clever.

So this will actually brain probably very slowly here.