Stanford University.

>> Okay everyone.

We're ready.

Okay well welcome to CS224N in Linguistics 284.

This is kind of amazing.

Thank you for everyone who's here that's involved and also the people who don't fit

in here and the people who are seeing it online on SCPD.

Yeah it's totally amazing the number of people who've signed up to do this class

and so in some sense it seems like you don't need any advertisements for

why the combination of natural language process and

deep learning is a good thing to learn about.

But nonetheless today, this class is really going

to give some of that advertisement, so I'm Christopher Manning.

So what we're gonna do is I'm gonna start off by saying

a bit of stuff about what natural language processing is and what deep learning is,

and then after that we'll spend a few minutes on the course logistics.

And a word from my co-instructor, Richard.

And then, get through some more material on why is language

understanding difficult, and then starting to do an intro to deep learning for NLP.

So we've gotten off to a rocky start today,

cause I guess we started about ten minutes late because of that fire alarm going off.

Fortunately, there's actually not a lot of hard content in this first lecture.

This first lecture is really to explain what an NLP class is and say

some motivational content about how and why deep learning is changing the world.

That's going to change immediately on the Thursday lecture because for

the Thursday lecture is then we're gonna start with sort of vectors and

derivatives and chain rules and all of that stuff.

So you should get mentally prepared for

that change of level between the two lectures.

Okay, so first of all what is natural language processing?

So natural language processing, that's the sort of computer scientist's name for

the field.

Essentially synonymous with computational linguistics which is

sort of the linguist's name of the field.

And so it's in this intersection of computer science and linguistics and

artificial intelligence.

Where what we're trying to do is get computers to

do clever things with human languages to be able to understand and

express themselves in human languages the way that human beings do.

So natural language processing counts as a part of artificial intelligence.

And there are obviously other important parts of artificial intelligence,

of doing computer vision, and robotics,

and knowledge representation, reasoning and so on.

But language has had a very special part of artificial intelligence,

and that's because that language has been this very distinctive properties of

human beings, and we think and go about the world largely in terms of language.

So lots of creatures around the planet have pretty good vision systems,

but human beings are alone for language.

And when we think about how we express our ideas and go about doing things that

language is largely our tool for thinking and our tool for communication.

So it's been one of the key technologies that people have thought

about in artificial intelligence and it's the one that we're going to look at today.

So our goal is how can we get computers to process or

understand human languages in order to perform tasks that are useful.

So that could be things like making appointments, or buying things, or

it could be more highfalutin goals of sort of, understanding the state of the world.

And so this is a space in which there's starting to be a huge amount of commercial

activity in various directions, some of things like making appointments.

A lot of it in the direction of question answering.

So, luckily for people who do language, the arrival of mobile has just been super,

super friendly in terms of the importance of language has gone way way higher.

And so now really all of the huge tech firms whether it's Siri,

Google Assistant, Facebook and Cortana.

But what they're furiously doing is

putting out products that use natural language to communicate with users.

And that's an extremely compelling thing to do.

It's extremely compelling on phones because phones have these dinky

little keyboards that are really hard to type things on.

And a lot of you guys are very fast at texting, I know that, but

really a lot of those problems are much worse for a lot of other people.

So it's a lot harder to put in Chinese characters than it is to put in

English letters.

It's a lot harder if you're elderly.

It's a lot harder if you've got low levels of literacy.

But then there are also being new vistas opening up.

So Amazon has had this amazing success with Alexa, which is really shown

the utility of having devices that are just ambient in the environment, and

that again you can communicate with by talking to them.

As a quick shout-out for Apple, I mean, really,

we do have Apple to thank for launching Siri.

It was, essentially, Apple taking the bet on saying we can

turn human language into consumer technology that

really did set off this arms race every other company is now engaging on.

Okay, I just sort of loosely said meaning.

One of the things that we'll talk about more is meaning is a kind of a complex,

hard thing and it's hard to know what it means to understand fully meaning.

At any rate that's certainly a very tough goal which people refer to as AI-complete

and it involves all forms of our understanding of the world.

So a lot of the time when we say understand the meaning,

we might be happy if we sort of half understood the meaning.

And we'll talk about different ways that we can hope to do that.

Okay, so one of the other things that we hope that you'll get in

this class is sort of a bit of appreciation for human language and

what it's levels are and how it's processed.

Now obviously we're not gonna do a huge amount of that if you really wanna

learn a lot about that.

There are lots of classes that you can take in the linguistics department and

learn much more about it.

But I really hope you can at least sort of get a bit of a high level of

understanding.

So this is kind of the picture that people traditionally have given for

levels of language.

So at the beginning there's input.

So input would commonly be speech.

And then you're doing phonetic and

phonological analysis to understand that speech.

Though commonly it is also text.

And then there's some processing that's done there which has

sort of been a bit marginal from a linguistics point of view, OCR,

working out the tokenization of the words.

But then what we do is go through a series of processing steps

where we work out complex words like incomprehensible,

it has the in in front and the ible at the end.

And that sort of morphological analysis, the parts of words.

And then we try and

understand the structure of sentences, that syntactic analysis.

So if I have a sentence like 'I sat on the bench',

that 'I' is the subject of the verb 'sat', and the 'on the bench' is the location.

Then after that we attempt to do semantic understanding.

And that's semantic interpretation's working out the meaning of sentences.

But simply knowing the meaning of the words of a sentence isn't

sufficient to actually really understand human language.

A lot is conveyed by the context in which language is used.

And so that then leads into areas like pragmatics and discourse processing.

So in this class, where we're gonna spend most of our time is in that middle

piece of syntactic analysis and semantic interpretation.

And that's sort of bulk of our natural language processing class.

We will say a little bit right at the top left where this discussion,

speech signal analysis.

And interestingly, that was actually the first place where deep learning

really proved itself as super, super useful for tasks involving human language.

Okay, so applications of Natural Language Processing are now

really spreading out thick and fast.

And every day you're variously using applications of

Natural Language Processing.

And they vary on a spectrum.

So they vary from very simple ones to much more complex ones.

So at the low level, there are things like spell checkings, or

doing the kind of autocomplete on your phone.

So that's a sort of a primitive language understanding task.

Variously, when you're doing web searches,

your search engine is considering synonyms, and things like that for you.

And, well, that's also a language understanding task.

But what we are gonna be more interested in is trying to

push our language understanding computers up to more complex tasks.

So some of the next level up kind of tasks that we're actually gonna want to have

computers look at text information, be it websites, newspapers or whatever.

And get the information out of it, to actually understand the text well enough

that they know what it's talking about to at least some extent.

And so that could be things like expecting particular kinds of information, like

products and their prices or people and what jobs they have and things like that.

Or it could be doing other related tasks to understanding the document,

such as working out the reading level or intended audience of the document.

Or whether this tweet is saying something positive or

negative about this person, company, band or whatever.

And then going even a higher level than that, what we'd like our computers

to be able to do is complete whole level language understanding tasks.

And some of the prominent tasks of that kind that we're going to talk about.

Machine translation, going from one human language to another human language.

Building spoken dialogue systems, so you can chat to a computer and

have a natural conversation, just as you do with human beings.

Or having computers that can actually exploit the knowledge of the world

that available on things like Wikipedia and other sources.

And so it could actually just intelligently answer questions for

you, like a know everything human being could.

Okay, and we're starting to see a lot of those things actually being used

regularly in industry.

So every time you're doing a search, in little places, there are bits of

natural language processing and natural language understanding happening.

So if you're putting in forms of words with endings,

your search engine's considering taking them off.

If there are spelling errors, they're being corrected.

Synonyms are being considered, and things like that.

Similarly, when you're being matched for advertisements.

But what's really exciting is that we're now starting to see much

bigger applications of natural language processing being commercially successful.

So in the last few years, there's just been amazing,

amazing advances in machine translation that I'll come back to later.

There have been amazing advances in speech recognition so that we just now

get hugely good performance in speech recognition even on our cell phones.

Products like sentiment analysis they have become hugely commercially

important, right?

It depends on your favorite industries but there are lots of Wall Street Journal

firms that every hour of the day are scanning news articles looking for

sentiment about companies to make buy and sell decisions.

And just recently, really over the last 12 months,

there's been this huge growth of interest in how to build chatbots and

dialog agents for all sorts of interface tasks.

And that sort of seems like it's growing to become a huge new industry.

Okay, see I'm getting behind already.

So in just a couple of minutes,

I want to say that corresponding things about deep learning.

But before getting into that,

let me just say a minute about what's special about human language.

Maybe we'll come back to this, but

I think it's interesting to have a sense of right at the beginning.

So there's an important difference between language and

most other kinds of things that people think of when they do signal processing

and data mining and all of those kinds of things.

So for most things, there's just sort of data that's either the world out there.

It has some kind of, pick up some visual system for it.

Or someone's sort of buying products at the local Safeway.

And then someone else is picking up the sales log and saying,

let me analyze this and see what I can find, right?

So it's just sort of all this random data and

then then someone's trying to make sense of it.

So fundamentally, human language isn't like that.

Human language isn't just sort of a massive data exhaust that you're trying to

process into something useful.