tutorials.

But no.

There is one more because I am leading to something.

I am going to-- you will soon see in this playlist

a section on convolutional neural networks.

But before I get to convolutional neural networks,

I want to look at reasons why a convolutional layer.

I have to answer this question like, what is a convolution?

I've got to get to that.

But before I get to that, I want to just see why

they exist in the first place.

So I want to start with another scenario

for training your own neural network.

That scenario is an image classifier.

Now you might rightfully be sitting

there saying to yourself, you've done videos

on image classifiers before.

And in fact, I have.

The very beginning of this whole series

was about using a pre-trained model for an image classifier.

And guess what?

That pre-trained model had convolutional layers in it.

So I want to now take the time to unpack what that means more

and look at how you could train your own convolutional neural

network.

Again, first though, let's just think

about how we would make an image classifier

with what we have so far.

We have an image.

And that image is being sent into an ML5 neural network.

And out of that neural network comes either a classification

or regression.

And in fact, we could do an image regression.

And I would love to do that.

But let me start with a classifier

because I think it's a lot simpler to think about

and consider.

So maybe it comes out with one of two things,

either a cat or a dog and some type of confidence score.

I previously zoomed in on the ML5 neural network

and looked at what's inside, right?

We have this hidden layer with some number

of units and an output layer, which, in this case,

would have just two if there's two classes.

Everything is connected, and then there are the inputs.

With post net, you might recall, there were 34 inputs

because there were 17 points on my body,

each with an xy position.

What are these?

Let's just say, for the sake of argument,

that this image is 10 by 10 pixels.

So I could consider every single pixel

to be an individual input into this ML5 neural network.

But each pixel has three channels,

and R, G, and B. So that would make 100 times three inputs,

300 inputs.

That's reasonable.

So this is actually what I want to implement.

Take the idea of a two layer neural network

to perform classification, the same thing I've

done in previous videos, but, this time, use as the input

the actual raw pixels.

Can we get meaningful results from just doing that?

After we do that, I want to return back to here

and talk about why this is inadequate or not going

to say inadequate but how this can be improved on

by adding another layer.

So this layer won't--

sorry.

The inputs will still be there.

We're always going to have the inputs.

The hidden layer will still be there.

And the output layer will still be there.

But I want to insert right in here

something called a convolutional layer.

And I want to do a two dimensional convolutional

layer.

So I will come back.

If you want to just skip to that next video,

if and when it exists, that's when I

will start talking about that.

But let's just get this working as a frame of reference.

I'm going to start with some prewritten code.

All this does, it's a simple P5JS sketch

that opens a connection to the web cam,

resizes it to 10 by 10 pixels, and then

draws a rectangle in the canvas for each and every pixel.

So this could be unfamiliar to you.

How do you look at an image in JavaScript in P5

and address every single pixel individually?

If that's unfamiliar to you, I would refer

to my video on that topic.

That's appearing over next to me right now.

If you go take a look at that and then come back here.

But really, this is just looking at every x and y position,

getting the R, G, B values, filling a rectangle,

and drawing it.

So what I want to do next is think about,

how do I configure this ML5 neural network,

which expects that 10 by 10 image as its input?

I'm going to make a variable called pixel brain.

And pixel brain will be a new ML5 neural network.

I should have mentioned that you could find the link to the code

that I'm starting with, in case you

wanted to code along with me, both the finished code

and the code I'm starting with will

be in this video's description.

So to create a neural network, I call the neural network

function and give it a set of options.

One thing I should mention is while in all the videos

I've done so far, I've said that you

need to specify the number of inputs

and the number of outputs to configure your neural network.

The truth is ML5 is set up to infer

the total number of inputs and outputs

based on the data you're training it with.

But to be really explicit about things

and make the tutorial as clear as possible,

I'm going to write those into the options.

So how many inputs?

Think about that for a second.

The number of columns times the number of the rows times

R, G, B. Maybe I would have a grayscale image.

Maybe I could just make it I don't

need a separate input for R, G, and B. But let's do that.

Why not?

I have the 10 by 10 in a variable called video size.

So let's make that video size times video size times three.

Let's just make a really simple classifier that's

like I'm here or not here.

So I'm going to make that two.

The task is classification.

And I want to see debugging when I train the model.

Now I have my pixel brain, my neural network.

Oops.

That should be three.

Let's go with my usual typical, terrible interface,

meaning no interface.

And I'm just going to train the model based on when

I press keys on the keyboard.

So I'll add a key press function.

And then let me just a little goofy here,

which I'm just going to say when I press the key,

add example key.

So I need a new function called add example.

Label.

So basically, I'm going to make the key that I press the label.

So I'm going to press a bunch of keys

when I'm standing in front the camera

and then press a different key when I'm not standing

in front of the camera.

Now comes the harder work.

I need to figure out how to make an array of inputs

out of all of the pixels.

Luckily for me, this is something

that I have done before.

And in fact, I actually have some code

that I could pull from right in here,

which is looking at how to go through all the pixels

to draw them.

But here's the thing.

I am going to do something to flatten the data.

I am not going to keep the data in its original columns

and rows orientation.

I'm going to take the pixels and flatten them out

into one single array.

Guess what?

This is actually the problem that

convolutional neural networks will address.

It's bad to flatten the data because its spatial arrangement

is meaningful.

I'll start by creating an empty array called inputs.

Then I'll loop through all of the pixels.

And to be safe, I should probably

say video dot load pixels.

The pixels may already be loaded because I'm

doing that for down here.

And I could do something where if I'm drawing them,

I might as well create the data here.

But I'm going to be redundant about it.

And I'm going to say--

ah, but this is weird.

Here's the weird thing.

I thought I wasn't going to talk about the pixel array

in this video and just refer you to the previous one.

But I can't escape it right now.

For every single pixel in an image in P5JS,

there are four spots in the array, a red value,

a green value, a blue value, and an alpha value.

Alpha value for transparency.

The alpha value, I can ignore because it's

going to be 255 for everything.

There's no transparency.

If I wanted to learn transparency,

I could make that an input and have 10 by 10 times 4.

But I don't need to do that here.

So in other words, pixel zero starts here, 0, 1, 2, 3.

And the second pixel starts at index four.

So as I'm iterating over all of the pixels,

I want to move through the array four spaces at a time.

There's a variety of ways I could approach this,

but that's going to make things easiest for me.

So that means right over here, this

should be plus equals four.

Then I can say the red value is video dot pixels index

I. The green value is at I plus one.

And the blue value is at I plus two.

And just to be consistent, I'm going

to just put a plus zero in there so everything lines up nicely.

So that's the R, G, and B values.

Then I want those R, G, and B values

for this particular pixel to go in the inputs array.

The chat is making a very good point,

which is that I have all of the stuff in an array already.

And all I'm really doing is making a slightly smaller array

that's removing every fourth element.

I could do that with the filter function

or some kind of higher order function

or maybe just use the original array.

I'm not really sure why I'm doing it this way.

But I'm going to emphasize this data preparation step.

So I look forward to hearing your comments about

and maybe reimplementations of this that just

use the pixel array directly.

But I'm going to keep it this way for right now.

So I'm taking the R, G, and B and putting them

all into my new array.

Then the target is just the label,

a single label in an array.

And I can now add this as training data,

pixel brain add data inputs target.

Let's console log something just to see that this is working.

So I'm going to console log the inputs.

And let's also console log the target,

just to see that something is coming out.

So, a, yeah.

We can see there's an array there.

And there's the a.

And now if I do b, I'm getting a different array with b there.

So I'm going to assume this is working.

I could say inputs dot length to make sure

that that's the right idea.

Yeah.

It's got 300 things in it.

OK.

Next step is to train the model.

So I'm going to say, if the key pressed is T,

don't add an example but rather train the model.

And let's give it train it over 50 epochs

and have a callback when it's finished training.