Charlie Gerard PARISS: I have been enjoying all of these

presentations from backstage and I'm like reading everything backwards.

But it's cool.

All right.

So, next we have strike a pose  gesture recognition in JavaScript with Charlie Gerard.

So, can we give some encouragement and give some clapping and  whoo!

Charlie!

[ Applause ] CHARLIE: Thank you.

Thanks so much for the introduction.

And thanks, everybody, for the warm welcome.

We'll see if you feel the same way at the end of the talk.

But that was nice.

So, yes, my name is Charlie.

I'm a developer at Atlassian.

I live in Sydney in Australia.

You might not know Atlassian, but maybe the product, Jira.

But what I'm going to talk about today has nothing to do with Jira.

And outside of my job as well, I'm a Google developer expert in web technologies and a

Mozilla TechSpeaker.

There's been a few other speakers part of these groups in the past few days.

If you have questions, feel free to ask us.

But the title of this talk was strike a pose.

And I don't know if there's any Madonna fans.

I forgot.

I should say that first.

So, I really like human computer interaction.

With Jira, nothing to do with that.

But at home, I like to explore ways to interact with technology and interface with devices

in ha different way.

Maybe not with your computer or your phone, or maybe using the phone a different way.

So, this is why I called this talk strike a pose because it's from the song Vogue.

And I don't know if you know it, but vogueing is a type of dance where you don't follow

certain steps.

You just express yourself in different gestures and it's who you are and mow with dance.

But we're not going to dance today.

Not what we're gonna do.

But we are going use our gestures to play a game of Street Fighter.

At the end, that's the goal.

Yes.

So, to do this, there's a few different ways.

But the way we're going to talk about today, you need this to get started.

So, we're going to use Arduino.

I'm really glad some of you did workshops yesterday to use hardware.

You already know a little bit what I'm going to talk about.

This is probably not the one you use.

This is a maker 1000 that connects via Wi Fi.

On top that have, we're going to be holding that piece of hardware and using machine learning

to find patterns in the data that we're tracking in our movements and we're going to do absolutely

everything in JavaScript.

So, for Arduino, yay!

For Arduino, Johnny Five.

And for TensorFlow, we're going to use TensorFlow JS.

So, step one, gathering data.

At first we have nothing.

We have an idea.

We want to be able to play Street Fighter with our body movements.

What I mean by that is when I want to punch I'm going to actual punch.

There's three movements.

I have punch, and  let's just call them like that.

So, gathering data.

We're going to do that with the Arduino.

So, I used a Wi Fi board because if you use a basic board like the UNO, it has to be timed

to the computer and I want to be able to move around.

Play from anywhere.

If I wanted to, I could play from the back of the room as well.

But the Arduino is going to render the program, but there's no built in sensor to track motion.

We are going to add an NPU1650, an accelerometer and gyro scope.

This is good to track movements.

I'm going to become an NPU1650 right now.

How does an accelerometer work?

If you can track motions in three different axis, X, Y and Z.

And accelerometer, left to right, that would be the X axis and then the Y axis and then

the Z axis.

But if you use the accelerometer just by itself, you're not walking like this through life.

We use the gyro scope to get the changes of rotation.

We have to track three more axis, but with rotation.

X axis, rotation, and Y and Z.

And in the end, we're going to have six points of data to track over time as we are moving.

But there is one last thing that I did as well, which is, oh, sorry, a button.

And I did a button because I want to track the data only when I am doing a gesture.

You don't have to do that, but as a first prototype, it was easier to only get the data

from my gesture to train the algorithm to only  to only find patterns in the data when

I'm holding the button.

When I'm gonna hold the button, do the gesture.

And when I release it, take into consideration that data coming from the sensor.

So, this is for the hardware.

But in terms of code we start by always requiring a few normal.

Johnny Five, the system, and the client to set up our board.

As it is a Wi Fi board, you have to connect to the IP address of the Arduino and a certain

board to communicate with your computer.

And then once the board is ready, once that is set up, we set the pin that you want, in

this case, A0, and then we create a file, like a stream, to be able to write data to

that file.

In this code sample, I had coded a name of the file, example punch zero.

Because you have to record data a lot of times.

So, punch zero, punch one, punch two.

And this is how I spend my personal time.

I do that at home.

After you open the stream to write to a file, you have to instantiate with a sensor the

NPU1650, give it the pins to which it's connected.

And as soon as you have change of data from the, as soon as you're moving, you can track

the file.

And when you're holding the button down and writing that data to the file.

Once we release the button, we end that stream, meaning that the rest of the data coming from

the sensor, we're not adding it to any files.

In the end, we expect stuff that look like this.

And I say expect because this is not exactly what I got.

But this is what I wanted.

So, this is real data, but not come from the Arduino.

And I will talk about that a little bit later.

This is actually not at all what I got.

But I really wanted that because I knew that it kind of work when the I built the really

first prototype of this.

What I got instead was something like this.

So, we still have six points of data from the accelerometer in the gyroscope.

But it's not personalized enough.

Don't worry about reading it, but looking at it later, you see that the points of data

look exactly the same.

And over time, there's not that much of a change coming from the sensor.

Maybe it's my sensor or code, I'm not sure.

I just know in the end you can train the algorithm, you get your model.

But the thing is the accuracy of this data is not good.

I have to copy that later.

And this is one thing if you get into machine learning, it's not only about how much data

you get, it's also about how good it is.

I was getting a lot of data from the Arduino.

But the quality of it was not good enough for the model to be accurate.

Okay.

So, now we recorded our data many times in other files.

But you can't really feed that to TensorFlow yet because at the moment it's just lines

in files.

So, it's not really useful.

So, we have to do a bit of data processing to transfer the data into a way that TensorFlow

can work with.

So, if we imagined that I have the nice file that I wanted.

And start with lines.

But the first step to transform it into something that we can use it.

And change it from features and label.

I think this is probably one of like the main terms in machine learning.

Features will be the characteristic of your gesture.

So, all the data that's in the file.

And the label will be  so, you can't really work with the strings.

And it has to be numbers.

So, we're just converting the name of the gesture to the number.

If you have a gesture array with the punch and punch is index zero.

You set the label to zero.

But that's only for one file.

So, one sample.

What we want is to do that with all our files.

So, we're going to end up with a massive array of objects that represent our gestures in

features and label.

But that's not it neither.

Because that's the first step to kind of put the data in a way that we understand it a

bit better.

We can work with objects.

But TensorFlow doesn't work with objects.

TensorFlow works more with stuff that look like arrays, but it's more take tensors, but

let's just talk about arrays.

And at the moment, we have objects, that's not good.

We need to split between features and label.

So, we're going to have here.

We're going to end up with something like this.

I'm going to go line by line, that's like what?

The labels have zero, one or two, three gestures, punch, round, and upper cut.

And we will have a multidimensional array of features to map gestures to each other.

So, the first level of our multidimensional array for labels is going to be mapped to

our first level of features as well.

And if we go a bit deeper, it means that the first punch that I record is going to be mapped

to the data that I get from an in features array.

And you're going in, the second one mapped to the second one in the first level of the

array.

And you go through that one by one.

So, we're getting a bit  I don't know if you noticed, I didn't show you any code to

transfer the data because it has nothing to do with TensorFlow yet.

At moment, it's if you know lots of array methods and pushing objects and stuff like

that.

You do that with normal JavaScript functions.

We're not using any TensorFlow methods.

But we are getting more towards something that TensorFlow can work with because it's

only multidimensional arrays and just numbers.

No strings, no numbers.

We're getting closer.

And now the thing  now we need to move on to converting to tensors.

Think of tensors as a data type that TensorFlow can work with.

This is where I'm going to show a little bit more of code that is TensorFlow specific because

we have to use the built in methods to be able to transform our multidimensional arrays

into tensors.

We don't opt to look at the labels and features.

And I had an array of zeros and two, and one to map all the gestures.

But with this, the algorithm is not going to really try to really look at the data and

think of a gesture.

It's probably going to look at, well, treat previous ones that you gave me were punches,

I'm going to assume this is a punch.

But if you shuffle the data, you can force the algorithm to try to understand  what

makes a punch and an upper cut and things like that.

So, once the data is shuffled, we start using  well, in this case it's a tensor 2D from TensorFlow

where what is going on here is that we're using our shuffled features in the second

parameter of the function is going to be the shape of the tensor.

And that means that it's going to be expecting the data to be coming in a certain way.

And our first argument is the number of samples by gesture.

If a punch gesture 20 times, I'm going to use the number 20.

And the total number of data per file, the second one in the array is the number of,

let's say, data input in a file.

So, you know how I'm tracking six different values for my gestures, X, Y, Z for accelerometer

and gyroscope.

You have to multiply the two numbers and that will be the shape of your tensor.

Specifically the amount of data in one gesture that it's going to be looking for.

And then labels, tensor.

So, this is gonna be a tensor 2D because the label is an integer, zero, one or two.

And we move from a tensor 2D to a tensor 1D.

Really we have the data type that TensorFlow can work with.