lot of statistical models, from the matched pairs t-test to linear regression. And for

the most part, we've used them to model data that we already have so we can make inferences

about it.

But sometimes we want to predict future data. A model that predicts whether someone will

default on their loan could be very helpful to a bank employee. They're probably not

writing scientific papers about why people default on loans, but they do care about accurately

predicting who will.

Many types of Machine Learning (ML) do just that: build models to predict future outcomes.

And this field has exploded over the past few decades. Supervised Machine Learning takes

data that already has a correct answer, like images that have been labeled as “cat”

or “not a cat”, or the current salary of a company's CEO, and tries to learn how

to predict it. It's supervised because we can tell the model what it got wrong.

It's called Machine Learning because instead of following strict rules and instructions

from humans, the computers (or machines) learn how to do things from data.

Today, we'll briefly cover a few types of supervised Machine Learning models, logistic

regression, Linear Discriminant Analysis, and K Nearest Neighbors.

Intro

Say you own a microloan company. Your goal is to give short term, low interest loans

to people around the world, so they can invest in their small businesses. You have everyone

fill out an application that asks them to specify things like their age, sex, annual

income, and the number of years they've been in business.

The microloan is not a donation, the recipient is supposed to pay it back. So you need to

figure out who is most likely to do that.

During the early days of your company, you reviewed each application by hand and made

that decision based on personal experience of who was likely to pay back the loan.

But now you have more money and applicants than you could possibly handle. You need a

model--or algorithm--to help you make these decisions efficiently.

Logistic regression is a simple twist on linear regression. It gets its name from the fact

that it is a regression that predicts what's called the log odds of an event occuring.

While log odds can be difficult, once we have them, we can use some quick calculations to

turn them into probabilities, which are a lot easier to work with. We can use these

probabilities to predict whether an individual will default on their loan.

Usually the cutoff is 50%. If someone is less than 50% likely to default on their loan,

we'll predict that they'll pay it off. Otherwise, we'll predict that they won't

pay off their loan.

We need to be able to test whether our model will be good at predicting data it's never

seen before. Data it doesn't have the correct answer for. So we need to pretend that some

of our data is “future” data for which we don't know the outcome.

One simple way to do that is to split your data into two parts.

The first portion of our data, called the training set, will be the data that we use

to create--or train--our model. The other portion, called the testing set, is the data

we're pretending is from the future. We don't use it to train our model.

Instead, to test how well our model works, we withhold the outcomes of the test set so

that the model doesn't know whether someone paid off their loan or not, and ask it to

make a prediction.

Then, we can compare these with the real outcomes that we ignored before.

We can do this using a what's called a Confusion Matrix. A Confusion Matrix is a chart that

tells us what actually happened--whether a person paid back a loan--and what the model

predicted would happen.

The diagonals of this matrix are times when the model got it right. Cases where the model

correctly predicted that the person will default on the loan is called a True Positive. “True”

because it got it right. “Positive” because the person defaulted on their loan.

Cases where the model correctly predicted that a person will pay back the loan are called

True Negatives. Again “true” because it made the correct prediction, and “negative”

because the person did not default.

Cases where the model was wrong are called False Negatives--if the model thought that

they would not default--and False Positives--if the model thought they would default.

Using current data and pretending it was future data allows us to see how this model performed

with data it had never seen before.

One simple way to measure how well the model did is to calculate its accuracy. Accuracy

is the total number of correct classifications--Our True Positives and True Negatives--divided

by the total number of cases. It's the percent of cases our model got correct.

Accuracy is important. But it's also pretty simplistic. It doesn't take into account

the fact that in different situations, we might care more about some mistakes than others.

We won't touch on other methods of measuring a model's accuracy here, but it's important

to recognize that in many situations, we want information above and beyond just an accuracy

percentage.

Logistic regression isn't the only way predict the future. Another common model is Linear

Discriminant Analysis or LDA for short. LDA uses Bayes' Theorem in order to help us

make predictions about data.

Let's say we wanna predict whether someone would get into our local state college based

on their high school GPA. The red dots represent people who did not

get in, green are people who did.

If we make a couple of assumptions, we can estimate the GPA distributions of people who

did, and did not get their acceptance letter.

If we find a new student who wants to know if they will get in to your local state school,

we use Bayes Rule and these distributions to calculate the probability of getting in

or not.

LDA just asks, “Which category is more likely?” If we draw a vertical line at their GPA, whichever

distribution has a higher value at that line is the group we'd guess.

Since this student, Analisa has a 3.2 GPA, we'd predict that she DOES get in. Since

it's more likely under the “got in” distribution.

But we all know that GPA isn't everything. What if we looked at SAT Scores as well.

Looking at the distributions of both GPA and SAT scores together can get a little more

complicated. And this is where LDA becomes really helpful.

We want to create a score, we'll call it Score X, that's a linear combination of

GPA and SAT scores. Something like this: We, or rather the computer, want to make it

so that the Score X value of the admitted students is as different as possible from

the Score X value of the people who weren't admitted.

This special way of combining variables to make a score that maximally separates the

two groups is what makes LDA really special.

So, Score X is a pretty good indicator of whether or not a student got in. AND that's

just one number that we have to keep track of, instead of two: GPA and SAT score.

For this sample, my computer told me that this is the correct formula:

Which means we can take the scatter plot of both GPA and SAT score and change it into

a one-dimensional graph of just Score X.

Then we can plot the distributions and use Bayes Rule to predict whether a new student,

Brad, is going to get into this school.

Brad's Score X is 8, so we predict that he won't get in, since with a score X of

8, it's more likely that you won't get in than that you will.

Creating a score like Score X can simplify things a lot. Here, we looked at two variables,

which we could have easily graphed. But, that's not the case if we have 100 variables for

each student. Trust me, you don't want your college admissions counselor making admissions

decisions based on a graph like that.

Using fewer numbers also means that on average, the computer can do faster calculations. So

if 5 million potential students ask you to predict whether they get in, using LDA to

simplify will speed things up.

Reducing the number of variables we have to deal with is called Dimensionality Reduction,

and it's really important in the world of “Big Data”. It makes working with millions

of data points, each with thousands of variables, possible.

That's often the kind of data that companies like Google and Amazon have.

The last machine learning model we'll talk about is K-Nearest Neighbors.

K-Nearest Neighbors...or KNN for short...relies on the idea that data points will be similar

to other data points that are near it.

For example, let's plot the height and weight of a group of Golden Retrievers, and a group

of Huskies:

If someone tells us a height and weight for a dog--named Chase--whose breed we don't

know...we could plot it on our graph.

The four points closest to Chase are Golden Retrievers, so we would guess he's a Golden

Retriever.

That's the basic idea behind K-Nearest Neighbors! Whichever category--in this case dog breed--has

the more data points near our new data point is the category we pick.

In practice it is a tiny bit more complicated than that. One thing we need to do is decide

how many “neighboring” data points to look at.

The K in KNN is a variable representing the number of neighbors we'll look at for each

point--or dog--we want to classify.

When we wanted to know whether Chase was a Husky or a Golden Retriever, we looked at

the 4 closest data points. So K equals 4. But we can set K to be any number.

We could look at the 1 nearest neighbor. Or 15 nearest neighbors. As K changes, our classifications

can change. These graphs show how points in each area of the graph would be classified.

There are many ways to choose which k to use. One way is to split your data into two groups,

a training set and a test set. I'm going to take 20% of the data, and ignore

it for now.

Then I'm going to take the other 80% of the data and use it to train a KNN classifier.

A classifier basically just predicts which group something will be in. It classifies

it. We'll build it using k equals 5.

And we get this result: Where blue means Golden Retriever. And red means Husky.

As you can see, the boundaries between classes don't have to be one straight line. That's

one benefit of KNN. It can fit all kinds of data.

Now that we have trained our classifier using 80% of the data, we can test it using the

other 20%. We'll ask it to predict the classes of each of the data points in this 20% test

set. And again, we can calculate an accuracy score. This model has 66.25% accuracy. But

we can also try out other K's and pick the one that has the best accuracy.

It looks like using a k of 50 hits the sweet spot for us. Since the model with k equals

50 has the highest accuracy of predicting Husky vs. Golden Retriever. So, if we want

to build a KNN classifier to predict the breed of unknown dogs, we'd start with a K of

50.

Choosing model parameters--variables like k that can be different numbers--can be done

in much more complex ways than we showed here, or could be done using information about the

specific data set you're working with . We not going to get into alternative methods

now, but if you're ever going to build models for real, you should look it up.

Machine Learning focuses a lot on prediction. Instead of just accurately describing our

current data, we want it to pretty accurately predict future data.

And these days, data is BIG. By one estimate, we produce 2.5 QUINTILLION bytes of data per

day. And supervised machine learning can help us harness the strength of that data.

We can teach models or rather have the models teach themselves how to best distinguish between

groups like will pay off a loan and those that won't. Or people who will love watching

the new season of The Good Place `and those that won't.

We're affected by these models all the time. From online shopping, to streaming a new show

on Hulu, to a new song recommendation on Spotify. Machine learning affects our lives everyday.

And it doesn't always make it better we'll get to that. Thanks for watching. I'll see you next time.