Placeholder Image

字幕列表 影片播放

  • JOSEPH JAY WILLIAMS: So I just want to mention, my background

  • is in cognitive science.

  • So I'm really interested in doing experiments in terms of

  • understanding how people learn.

  • And to get a sense of the way that fits in with the research

  • landscape, because there's a ton of research on learning

  • and many insights.

  • I guess the best concept is thinking in terms of

  • qualitative analyses.

  • So these rich investigations that you see from a

  • sociological perspective.

  • Education schools that will take a student and really take

  • a very close look at what they're learning.

  • Explore how they're understanding of algebra and

  • all the difference misconceptions they can have.

  • Then there's also things that are much more like randomized

  • control trials.

  • So policy workwear.

  • You might take a bunch of students in the school and

  • give them a treatment where they

  • undergo a training program.

  • And then you can see if that actually has an effect

  • compared to control group.

  • And that's a very large scale.

  • There's longitudinal studies, which, again, are very long

  • time scale.

  • Collecting measures, like people's grades or students'

  • outcomes as they progress through traditional education.

  • And of course there's working computer science and

  • Educational Data Mining, where you take very large data sets

  • in terms of observations, then try and induce what's going on

  • with learners.

  • As is terms of the way cognitive science fits in with

  • this, I think it's in between something like a randomized

  • control trial, longitudinal study and some light

  • qualitative analysis.

  • Because most of the experiments we do on a short

  • time scale.

  • And it really involves precisely controlling what

  • someone's learning in a particular situation and

  • trying out different forms of instruction and then assessing

  • how learning is occurring after that.

  • And you might think that from such micro experiments you

  • can't learn much.

  • But that's actually the expertise

  • in cognitive science.

  • And I think it's ready insightful.

  • It's obviously using a lot of insights.

  • But I think it's ready well suited for online education,

  • where often you want to ask questions that intermediate

  • between again, quality of assessment of what's going on,

  • and running a full-out randomized control trial where

  • you give people two different versions of the course.

  • There are questions about how you should frame instruction,

  • what kind of video you should show someone, and the kind of

  • strategies you should teach them.

  • So it just lets you know where it sits in there.

  • And what's also nice about cognitive science as an

  • approach is it's pretty interdisciplinary.

  • In a sense, you've got psychology, which is heavily

  • focused on experiments.

  • Philosophy, in terms of really trying to do good conception

  • analysis of what problem you're facing and classifying

  • different kinds of learning.

  • Linguistics, anthropology, as I mentioned, neuroscience, and

  • of course, AI.

  • And what's also nice, I think, is that it's also a bit easier

  • for people in cognitive science perhaps to talk with

  • the kind of researchers at Google who are interested

  • often in things like modeling and machine learning.

  • So to give you a bit of a preview of [INAUDIBLE]

  • cover.

  • So I'm going to talk about two ways you

  • could think about learning.

  • One that I think gives us a lot of rich insights into how

  • to improve education.

  • And I'm going to talk about very quickly three finds in

  • cognitive science that I find have been

  • particularly powerful.

  • It's thinking about what can you do before someone starts

  • learning in terms of framing their learning as

  • an answer to a problem?

  • What can you do during learning in terms of

  • requesting explanations from the learner?

  • And then what can you do after learning?

  • A new and interesting finding here is that you can actually

  • use assessments as instructional tools.

  • Having people take a test can actually be more powerful for

  • learning than having them study material again.

  • And then having looked at some of what we know about how to

  • promote learning of a concept or some set of knowledge, it's

  • worth asking, well, what knowledge can we teach someone

  • that's actually going to have the biggest impact

  • practically?

  • And that's why I think thinking about this idea of

  • what do people learn that's going to help them learn more?

  • So it's not just on this concept but across a range of

  • situations.

  • And then I'll talk about how you can change people's

  • beliefs and have a massive effect on their motivation.

  • You can teach people strategies for learning that

  • then have trickle down effects on all the content they may

  • come across.

  • And then finally, I'm going to talk about online search,

  • which is something I really started to think about

  • seriously recently since looking [INAUDIBLE]

  • of a power searching course.

  • And I actually think this is a really fascinating topic

  • that's right at the junction of things that are really

  • important to industry and the business world, and also

  • really important to education.

  • And I think that's actually a great place to focus, because

  • it allows you to get the benefits of private innovation

  • as well as academic work.

  • And also it means that you can use insights across areas.

  • When you're teaching a business person about search,

  • you're also learning something about how you could help a

  • child to be more inventive in the way they discover

  • information on the internet.

  • How is my speed in terms of talking?

  • Because with an accent, it probably sounds

  • like twice as fast.

  • And I speak twice as fast, so you're hitting a lot of powers

  • being raised there.

  • OK.

  • And one thing I'll cover at the end is I put on the

  • website just a list of resources that I found really

  • useful in terms of knowing what the literature is that

  • has shown really impressive effects on learning.

  • When you go in Google Scholar, I mean literally there are

  • thousands of papers there.

  • And I think that can be daunting.

  • And it's just easy to start with your

  • problem and work on that.

  • But there's always tons of really

  • interesting relevant work.

  • And so I tried to put aside some of the resources that I

  • found most useful.

  • So I have hyperlinks to some of the papers there and brief

  • explanations of what they're about.

  • And also information about other things that I'll talk

  • about at the end.

  • So in terms of learning, I think one kind of challenge,

  • even after studying learning for all my adult life, I still

  • think that there's this kind of intuition or this intuitive

  • theory that I have about learning that holds me back,

  • or misleads me when I'm making instructional decisions.

  • I think it's something that's sort of common

  • across many of us.

  • In the Cambridge Handbook of Learning and Sciences they

  • refer to this idea that when you learn something, you're

  • just dropping it into a bucket.

  • Learning is transferred from the teacher to the student.

  • So let's think about some content.

  • For example, learning from a video, like from a power

  • searching online video, a Khan Academy video.

  • Or reading text, which is how we absorb most of our

  • information.

  • Or you could think about learning from an exercise.

  • So a student solving a math problem, or someone attempting

  • to process a financial statement or do some budgeting

  • or solving a problem of managing

  • relationships with coworkers.

  • Anytime you can see those icons I need to represent, you

  • can stick your personal content in there.

  • So if you have a favorite example of learning or one

  • that's particularly relevant to your everyday experience or

  • your job, just feel free to insert that every time that

  • flashes out.

  • But I'll try to illustrate it with examples.

  • Because again there's good evidence that that's an

  • important thing to do for learning.

  • So I think there's this intuition that what learning

  • is about is just adding information.

  • So it's almost as if we have this model where

  • the mind is a bucket.

  • And so what does it mean to learn?

  • Well it means to take information and drop it in

  • that bucket.

  • Whether it's videos, text, solving stuff, just put it in.

  • And later on you can come back and get it.

  • But actually I think a much better way of thinking about

  • learning is that it's like taking a piece of information,

  • integrating it into the internet, and assuming it had

  • a web page.

  • So that galaxy is the internet.

  • And you can guess what those big giant planets are.

  • So how might you think about this analogy shedding more

  • light on learning?

  • Well under this kind of view, when you're adding content to

  • the internet, first of all, you have to

  • think about three stages.

  • Well what do you do before the content arrives?

  • Or what do you do in linking it to what's already known?

  • So it's really important now.

  • You're not just dropping in a bucket.

  • You're actually linking it to web pages that already exist

  • by citing them.

  • You're putting on a particular domain.

  • And so I think this is very analogous to the challenge we

  • face as learners of how do you get information to link up to

  • what people already know?

  • And in what kind of knowledge are you linking

  • new concepts to?

  • And I think that some things are obvious

  • when we look at it.

  • But it's definitely not at the forefront, I think, of a lot

  • of instructional decisions.

  • Also if we think now even of processes people engage in

  • while they're learning.

  • Everyone on the internet, time is limited, they're not going

  • to read your whole page.

  • If you're lucky they will even read the first few lines.

  • So we have to ask questions like, how do you structure the

  • information on a web page so that the really core

  • concepts jump out?

  • What are the key principles?

  • Or depending on the person, is the web page structured so

  • they can get to the information they need?

  • And so this is analogous to when we're processing

  • something like a video or a bunch of text, it's actually

  • just flat out impossible to remember all that information.

  • So what are the cognitive operations you're engaging in

  • that pick out some information as being more

  • important than others?

  • Some relationships or some principles or some details of

  • whatever content you're learning.

  • And then finally, what's the last part of getting a web

  • page off of the internet?

  • Again, I think you guys all have a better advantage in

  • answering these audience type [INAUDIBLE] questions.

  • But it's actually being able to find it.

  • And this is probably the most underlooked part of what

  • learning involves.

  • Because anytime you learn something, it's only going to

  • be learning if it influences your mind in a way that, at

  • some point in the future, you're going to act

  • differently because of it.

  • So you have to retrieve that information somewhere.

  • So for example, the final stage of learning is it's got

  • to be set up in a way that it's got the right cues or the

  • right connections that when someone goes looking for that

  • information, they're actually going to find it.

  • For example I might be really interested in things about

  • mathematics from a video.

  • And you could say that I know it.

  • But when I am actually faced with a situation where I have

  • to solve or differentiate something, am I actually going

  • to remember that fact at that moment?

  • Or am I going to remember something else I learned about

  • differentiation, or something else about functions?

  • And so really a key challenge in learning is the instruction

  • has to actually ensure that people are putting information

  • in the right way that they can access it later.

  • And actually a key part of that is going to be that after

  • instruction takes place, it's actually really important to

  • give people practice in testing them, and sort of

  • accessing that information.

  • What could be called retrieval of practice.

  • OK.

  • So one thing I wanted to hammer home at this point, in

  • terms of why it's really important to think about how

  • hard learning is and why these two models kind of matter, is

  • that there's this phenomenon called transfer, or this

  • distinction which is that learning might just be, again,

  • a piece of information.

  • A transfer is when you take an abstract principle from one

  • context and use it in another one.

  • So transfer is extremely rare.

  • And I think this is probably the biggest, most important

  • thing I feel like I know now that I didn't

  • know eight years ago.

  • Is that I assume that if I am told something and I don't

  • remember it, well, I just forgot it.

  • I have to practice a bit more.

  • But actually it's a much more profound problem than that.

  • Some people make this statement that almost nothing

  • that we learn in one context is going to get transferred to

  • a very different context.

  • And so actually the way we do most of our learning is by

  • really learning things in very specific contexts and slowly

  • generalizing them out.

  • So for example, let's think of a problem.

  • Let's say that you're a general

  • trying to invade a castle.

  • And you've got 100,000 people.

  • And you know that you can take it if you send them

  • all at the same time.

  • Unfortunately the person in the castle knows that as well.

  • And they've mined the roads so you can only

  • send 20,000 at a time.

  • So that's just not enough.

  • You can't send 20,000 people to take the castle.

  • So how do you solve this problem?

  • You need 100,000 to arrive at the castle.

  • But you can only send 20,000 along any

  • single road at a time.

  • OK.

  • Yeah.

  • There are multiple roads.

  • So the idea is that you divide your force up, and

  • they go and do it.

  • Here's a separate problem.

  • It's going to use a different kind of technique.

  • You're a physician.

  • And you need to kill a tumor in someone's stomach.

  • And you've got really high frequency rays that can

  • destroy the tumor.

  • But it's also going to take out a large part of the

  • person's flesh.

  • And you've got low frequency rays.

  • But they're not actually going to kill the

  • flesh or the tumor.

  • How do you solve that problem?

  • Yeah.

  • So here I gave you guys an advantage.

  • But what's shocking is that if you have people, you can bring

  • someone right in front of you.

  • They'll do the problem, turn the page to the next problem.

  • 20% of people will actually transfer that spontaneously.

  • You could even help them and get them to elaborate the

  • principle, do all kinds of things when they're learning

  • their first problem.

  • They don't hit the transfer spontaneously.

  • If you tell them, I think the one before is relevant, then

  • they might stop and really think and

  • they're like, oh yes.

  • I see how to solve it.

  • But that kind of transfer, you obviously can't have someone

  • running around telling you what's relevant to what.

  • I think it really, in my mind, highlights something that

  • seems so obvious, like the common principle there.

  • It does not happen for people at all.

  • It just doesn't come across.

  • It's almost like they learn something about generals and

  • invasion, and they learn something

  • about medical practice.

  • They didn't learn that abstract principle, and it's

  • really tough to get people to.

  • In fact, I can bet I could give you guys similar versions

  • of this problem and bet in your life in a week, in a few

  • weeks, and you'd be surprised how high a rate

  • you'd fail at that.

  • They've even done really compelling studies where they

  • take business students.

  • And they have them learn a negotiation strategy.

  • For example, we all think negotiation is about

  • compromise.

  • But actually one much better strategy is to find out what

  • the other person wants that you don't really care about

  • and what you want that they don't really care about and

  • trade off on that.

  • So we could have an orange, and if we're fighting you

  • could take half, I could take half.

  • But maybe I want the peel because I have to

  • flavor up some food.

  • And you actually want the inside because you're hungry.

  • That's a trade off negotiation strategy.

  • So what's interesting is when you take MBA students and

  • teach them this strategy, they do everything that you want

  • your students to do.

  • They understand the strategy, they can explain it to you,

  • they can give you other examples of the strategy.

  • They really get it.

  • Except when they actually have to use it.

  • If you put them in a face-to-face negotiation, they

  • fail to use the strategy.

  • So that's really compelling.

  • Because even in a situation where someone's done

  • everything that we like students to do, and students

  • almost never get there all of the time, it's still the case

  • that they're going to fail the transfer.

  • And so I think that really highlights the importance of

  • thinking carefully about what sort of processes you have

  • people engage in while they're learning, and also having

  • assessments that can really help you understand if people

  • have transferred or not.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: Oh sorry, yeah.

  • I'll clarify.

  • So they could tell the strategy to you and they could

  • write out and explain the principle.

  • But if two minutes later they were then in a situation where

  • they had to negotiate with someone face-to-face as part

  • of a team, 20 to 30% would do it.

  • 20 or 30, but it was very low.

  • AUDIENCE: Was anyone asking the question, what do you want

  • that I don't want?

  • JOSEPH JAY WILLIAMS: Exactly.

  • And it's shocking, because they really did

  • understand the strategy.

  • But face-to-face, it's almost like you have a different set

  • of concepts when you're dealing with people and a

  • different set of concepts when you're learning something in

  • your class.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: Yeah.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: I guess it's probably both true, but I

  • would say I lean towards the second one.

  • The way some people write about it, your mind is not

  • designed for transfer.

  • It's not designed.

  • You've got concrete concepts that correspond to different

  • things, and if you want transfer you have to do

  • something special with them.

  • Actually that's the original paper that showed the general

  • and the medical study.

  • This is one that showed it with business students.

  • And actually this guy wrote a book that sort of

  • put transfer on trial.

  • Where he pretty much argues, he thinks transfer almost

  • never occurs.

  • And so what we need to do is teach people specific skills

  • in specific contexts.

  • Obviously that's an extreme position but that's just to

  • let you know the tune of how people think about this.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: Yeah.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: Oh sure.

  • The question was, are these theories of transfer also true

  • if while they're learning about the concept they demote

  • mock negotiations?

  • Or the other point was maybe if they do three different

  • negotiations across different contexts.

  • And so those things definitely help and they make a big

  • difference.

  • The really key principle I'd seen is that there's a big

  • focus in business and medical education

  • on case-base reasoning.

  • And actually there's good evidence that how people

  • structure abstract concepts around a specific case is

  • really useful.

  • But the experiment by Genter, Loewenstein, and Thompson,

  • what this showed is that even if it helps to have a specific

  • case, you still don't necessarily get

  • the abstract principle.

  • So what is really nice about the experiment is, they would

  • give people two cases, one after the other, as was

  • suggested, versus giving the two cases side by side, and

  • actually tell people what are the similarities and

  • differences between these two cases.

  • In this story, it was about business, but let's say a

  • story about general, the story about the medical doctor,

  • which agents are playing what role?

  • And what's the relationship that exists here?

  • And so when you actually have that pretty extensive process,

  • which is really good at helping people really get

  • abstractions in a way that's telling it to them and words

  • that just doesn't, when they actually have to process this

  • and align items, that actually makes a big difference in

  • abstraction.

  • But it's not necessary enough for them to just see cases.

  • That helps.

  • But you've really got to add something extra onto it.

  • In comparison, I would say one of the super processes that

  • we've seen for getting people to really grasp an abstract

  • concept that they wouldn't get in words or they won't get

  • just from examples.

  • OK.

  • Well, in terms of thinking about if someone's about to

  • learn some content, what are things that

  • we're going to do before?

  • Are they going to watch a video about some training they

  • have at work?

  • Or are they going to read a text about mathematics?

  • So what we typically would do is give them that information.

  • And we might ask one or two questions or tell them what

  • they're going to study.

  • You're going to learn about math.

  • But pretty much you give them the information very much in

  • line with putting it into their head like it's a bucket.

  • But actually there's this really interesting set of work

  • on what's called Problem Based Learning.

  • And so the idea here is that with a very minimal change you

  • can influence how people are going to process

  • that video or lesson.

  • If I ask them a question before that's going to make

  • them interpret everything they learn as a

  • solution to a problem.

  • Instead of learning, for example, reading a bunch of

  • text about solving algebra equations, if that was

  • prefaced with if you have two variables in an equation and

  • you're trying to figure out how to isolate one, how

  • would you do it?

  • The student will probably not actually know how to do that.

  • They'll fail.

  • They'll try all kinds of things that are wrong.

  • But the mere fact that you got them thinking about that

  • problem, that means that as they go through the lesson,

  • they are encoding all that information as a solution to

  • that problem.

  • For example, you can think of cases where people have told

  • you something and you take it for granted or encode it, but

  • if at first you'd first been trying to solve a problem and

  • then got that piece of information in response, that

  • would've been much more effective for learning.

  • And so what's really nice about this manipulation is

  • that it's very minimal in that all you've got to do it just

  • take whatever content you have and proceed

  • it with some questions.

  • For example, if you're about to learn, let's say, about

  • power search, you could ask someone as well, how do you

  • find this kind of content?

  • Or if you're learning mathematics, is it possible to

  • construct this kind of relation and then just having

  • people think about that will then lead them to process

  • everything they see after differently?

  • It's also more motivating for students in that they're not

  • just absorbing facts, but they feel like they're actually

  • learning with some kind of purpose in mind.

  • In terms of thinking about what you can do during

  • learning, and here I'm going to focus on the work I've done

  • on explanation just because it's what I know best in terms

  • of presenting.

  • So the idea here is that we all have this intuition that

  • it's really good if we actively process information.

  • Or if we have ever tried to explain a concept to someone

  • else, something about actually explaining it to them might

  • actually resort to something new.

  • But despite that, it's not so often that instructional

  • videos or texts have questions embedded in them or activities

  • that force you to generate explanations.

  • And so there's a good bit of evidence that this does

  • actually help learning.

  • If you can prompt someone as they're learning to generate

  • explanations, they understand it more deeply and they're

  • more likely to transfer the principles on the situation.

  • The question that was interesting is understanding--

  • well, not to be too mad about it--

  • but why is it that explaining why helps you learn?

  • And this is really relevant in terms of thinking, when are

  • you going to ask learners for explanations?

  • You can't ask someone all the time.

  • Or is this some kind of content you should ask for?

  • Or what kind of knowledge is going to be targeted when you

  • ask for an explanation?

  • So it's got really important practical

  • implications in that sense.

  • So now we contrast just throwing someone into learning

  • and letting them use whatever strategies come to them, with

  • actually prompting them while they're learning to explain

  • why they're doing something, to explain why a fact is true

  • instead of just accepting it.

  • And so we looked at this in three contexts.

  • The first was in terms of learning artificial

  • categories.

  • So even though there's lots of work on explanation and rich

  • educational settings, this is the first study that's

  • actually in a lab context with artificial materials where we

  • can really control things very precisely.

  • Even if we have a sense an experience is helping, it's

  • very hard to pin down exactly what kind of knowledge people

  • are gaining.

  • Is it just that explaining is making

  • people pay more attention?

  • So the mere fact that you have to answer a question means

  • that you've got to pay attention to what you're

  • learning instead of dozing off or looking up information.

  • You've got to spend more time processing it.

  • It could also be that because you have to provide an

  • explanation you're more motivated.

  • There's a standard social context where you have to

  • produce information that could possibly be relevant to

  • someone else.

  • And so if those are how experience works, then that

  • suggests one way in which we'd use it as an

  • instructional tool.

  • But the idea that we wanted to test is actually that

  • experience effects is a lot more selective.

  • That there's something about explanation that doesn't just

  • boost your general processing or attention, but actually is

  • constraining you or forcing you to search for underlying

  • principles or patterns or generalizations.

  • And so this is a question we could test more readily when

  • we look at a constrained context like learning

  • artificial categories.

  • We also examined explaining people's behavior.

  • So for example, if you notice a friend donates to charity,

  • that wasn't me, right?

  • OK, that's fine.

  • If you notice a friend donates to charity, you could just

  • encode this fact about them.

  • You might make other predictions from it like,

  • they're going to have less money in their pocket at the

  • end of the day and probably won't treat you for drinks.

  • Or you could try and explain it.

  • Well why did that person give to charity?

  • And again, that's a different way of learning

  • about another person.

  • And so are facts just to boost your general attention to what

  • they're doing?

  • Or is it actually going to do something deeper like make you

  • search for an underlying principle?

  • And the final context we looked at this in was actually

  • trying to learn something more educational.

  • So the idea here was that we had people learn a bit about

  • the concept of variability.

  • And this is an example.

  • I'm actually going to talk more in depth so that I don't

  • spend too much time on this part, but I'll sum up the main

  • conclusions.

  • Across all of these domains the basic form was for

  • different materials and different kinds of knowledge,

  • but the basic form was asking a why question.

  • Explaining category membership, explaining why

  • someone did something, or explaining

  • the solution or problem.

  • And that would be contrasted with letting people choose

  • their own strategies, so being able to study freely.

  • And we also contrasted it with asking people to say out loud

  • what they were thinking.

  • Because it could be that the main effects of explanation

  • are actually having you clarify your thoughts.

  • Once you state it explicitly you can see what you were

  • thinking, what was missing, what the gaps

  • in knowledge were.

  • And so we're trying to pick apart that effect from whether

  • there's something else we're explaining, driving you to

  • search for underlying principles.

  • And then following the study, we gave people learning

  • measures the tapped exactly what they've learned.

  • So to talk about going into that third one in more depth,

  • I'll just run through the procedure so that

  • you can get a sense.

  • So the idea here is that people have to learn a

  • university's ranking system from examples.

  • So you're told that a university has a way of

  • ranking students from different classes.

  • And you're going to see examples of students who have

  • been ranked by last year's officer.

  • And you have to learn, well, what's the

  • system for doing it?

  • What information are they using?

  • So for example, you've got information about someone like

  • their personal score, their class average, the class

  • minimum score, the class deviation.

  • So for example, this might be John.

  • He got an 86% in history.

  • You might see what the top score is, what the class mean

  • standard deviation is.

  • Then there's another student.

  • Then in physics, he got about 80%.

  • Again, you see the top score, the class mean, the class

  • standard deviation.

  • And so, you'd actually be told which one was ranked higher.

  • So for example you'd be told the person on the left was.

  • And so I'm trying to think about how we

  • can learn about this.

  • There are a bunch of different ways you could approach it.

  • You could pay attention to lots of examples, and then

  • that would inform future situations.

  • You automatically get an intuition for

  • how people are ranked.

  • You could try and look for different patterns.

  • So for example, maybe people are ranked higher just if they

  • had a higher score.

  • The university is pretty egalitarian across courses.

  • You could think it's actually how far they are above the

  • class average.

  • So for example, this person's 6% above the class average,

  • whereas this person's 4%.

  • So even though they got a much lower score, they might still

  • actually be ranked higher.

  • Actually sorry, this is consistent in this case, but

  • the reason this person's ranked higher is not because

  • they got a higher score.

  • It's actually because they're farther from the average.

  • You could also think it's actually just a matter of

  • being close to the maximum score.

  • If you're close to the top person then you deserve to be

  • ranked higher no matter what.

  • And then finally, if you think about what statistics would

  • suggest, there's this idea that you should actually look

  • the distance from the average, but weighted by whatever the

  • standard deviation is.

  • And so the idea here is that what you really want to know

  • is, how far above most of the other students in the class

  • was each of these people?

  • And that'll give you a measure of whether you should rank

  • someone highly or not.

  • How many standard deviations above were

  • they, what is the Z-score?

  • And so we gave students a bunch of these observations.

  • They saw five ranked pairs in total, and the actual true

  • rule was using the number of standard deviations someone

  • was above the average.

  • But the other rules were pretty salient.

  • And in pre-testing, we find that a lot of

  • students endorse them.

  • Whoever got the highest score should be ranked higher.

  • Whoever's closest to the maximum, and so on.

  • And so some of the observations just didn't fit

  • with those rules.

  • So this might fit sometimes with the higher raw score, but

  • sometimes they don't.

  • They might match the close to maximum rule, but not always.

  • And so the idea here was, if explaining really forces

  • people to seek underlying principles, then explain these

  • sort of observations and the ones that conflicted with some

  • rules and others should drive them to actually find this

  • underlying principle, the rule above deviation that actually

  • applies to all the observations.

  • And so this is what we found.

  • This is showing you people's accuracy in classifying ranked

  • pairs before and after and how much improved.

  • And what you see is that people who had to explain had

  • a much bigger increase in accuracy, of about 40%,

  • compared to those who were writing their thoughts.

  • That's interesting because writing your thoughts is a

  • pretty elaborative activity.

  • You have to express what you're

  • thinking about the materials.

  • You have to process them deeply.

  • But explaining [INAUDIBLE]

  • this broader principle.

  • And that's just an illustration because across

  • all those contexts where there was learning about categories,

  • explaining how people find a broad generalization that

  • counted membership for all the robots.

  • If it was explaining people's behavior, again, you were more

  • likely to discover a general pattern like, young people

  • donated to charities more often than older people.

  • So this suggests that, in terms of thinking about

  • educational contexts, when you want to ask someone a why

  • question, it's not necessarily a good idea to just always be

  • prompting them, or just trying to boost their engagement.

  • You actually want to be selective in terms of when

  • they're given examples they can point them towards an

  • underlying principle, then you need to prompt

  • them to explain why.

  • Or if you think they don't have enough knowledge to

  • induce the principle yet, then it might actually make more

  • sense to engage in other activities, like helping them

  • acquire new concepts or have more facility with

  • understanding what the different facts are.

  • And then they might reach a stage where they can actually,

  • with prompting, be able to induce

  • the underlying principle.

  • What's another interesting thing is that in the second

  • line of experiments, we found that explaining actually

  • boosted the extent to which people used their knowledge in

  • finding underlying principles.

  • So the idea was that in the absence of explaining you

  • might have a lot of relevant knowledge that might suggest a

  • deviation is a good rule or something else is.

  • But you don't actually bring it to bear.

  • If you explain though, and you don't have that knowledge,

  • well then you might find some pattern but it's not

  • necessarily the correct one.

  • So the idea is that explaining of prior knowledge, interact

  • or guide what patterns people are searching for, and what

  • they're likely to detect.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: You mean in general, or with respect--

  • AUDIENCE: You said [INAUDIBLE]?

  • JOSEPH JAY WILLIAMS: So we ran it online and we just asked

  • people afterwards how familiar with it they were.

  • And it was a pretty big range.

  • Some people had never heard of it before, but a decent number

  • had at least come across the concept before.

  • And those people though, they still did ready

  • badly on the pre-test.

  • So in a lot of experimental situations you can look at

  • interesting effects of learning.

  • Because the truth is that learning just doesn't really

  • accrue that well the first time around.

  • So often there's relearning the material.

  • And even with extensive practice, it's hard to get

  • people to get that idea of the Z-score principle.

  • And we didn't find differences between low and high people

  • who had low and high exposure.

  • But again that might be because we need

  • to broaden the sample.

  • Because it was online we might just look on people who just

  • didn't have a lot of knowledge.

  • They wouldn't sign up for a math experiment maybe.

  • Also in terms of thinking about explanation's effect, it

  • really was selective.

  • And actually what we found is that it didn't help you

  • remember information better.

  • Sometimes even impaired it.

  • But especially if there wasn't a reliable pattern present, or

  • if you couldn't actually find it because you just didn't

  • have enough knowledge to, explain actually impaired

  • learning in some cases.

  • Because it drove people to keep searching for a principle

  • that didn't exist.

  • Or one that was only partially reliable.

  • And so this is something you might think about in terms of

  • when people make these mini generalizations, or they make

  • these kind of bugs that's sort of seem half right.

  • Children do this a lot in algebra for example.

  • And it might actually be from trying to seek explanations

  • and construct some kind of pattern.

  • But it's just a case that it's characterized as cases they've

  • seen so far.

  • But it's not actually going to help

  • them with future learning.

  • And so again that's another reason to think carefully

  • about well, when are you going to prompt someone for an

  • explanation?

  • You need to at a point where it's not going to hinder their

  • learning or be worse.

  • On the flip side, it's not saying explanation is bad,

  • it's just saying it's sort of like a powerful selective tool

  • that if you use in the wrong cases,

  • will actually be harmful.

  • And then since most of the workers have dealt with adults

  • we also looked at the effects of explanation in

  • five-year-olds learning causal relationships.

  • And we found very similar results.

  • Even though they were young and their ability to verbalize

  • are pretty limited, it was still able to, when prompted

  • to explain, discover underlying patterns.

  • And especially to bring prior knowledge to bear, and

  • actually use it to discover patterns that they wouldn't if

  • they weren't explaining.

  • And in terms of thinking how to take these kinds of

  • findings online, one really good context is thinking about

  • things like online mathematics, exercises, like

  • the one that Khan Academy has in their framework.

  • So that's a mini shrunken version but basically people

  • will get a problem statement, and they'll try and solve it

  • and type in an answer.

  • And along the way if they wanted hints, clicking on a

  • hint button gradually exposes the actual solution.

  • And so in the literature this is called a worked example.

  • And it's sort of infamous that sometimes they can be much

  • better for students learning than just trying things out

  • and not succeeding.

  • But it's really hard to get people to process them deeply,

  • to actually not just skim over it but understand each

  • component, each step, and how it relates to something more

  • general, so to get the principle.

  • As the last one case explained could be really helpful.

  • And so there are a number of different ways we're thinking

  • of approaching this.

  • But the simplest version is something like, when someone

  • enters a correct answer, or even if they're following a

  • solution, you can prompt them to try and think, well,

  • explain why that solution's correct.

  • Or explain why that person took the step they did.

  • Even though it seems, of course, you just saw the

  • correct answer, you should be thinking

  • about why it's correct.

  • This is actually not necessarily that common.

  • I mean, it's not guaranteed.

  • A lot of students may be trying to memorize the answer.

  • They may be trying to explain other aspects of the solution.

  • There are many things that people may be doing.

  • And when you do detailed analyses of children's

  • strategies, you find incredible variation.

  • And so that idea, just having that prompt to focus them on

  • understanding how that particular answer, or that

  • particular solution is just an instance

  • of a general principle.

  • What's the general strategy that was used here?

  • What's the key concept?

  • Now one thing that could be problematic is even though

  • this is promising because you can get people to learn just

  • by being prompted, so you don't necessarily have to give

  • them feedback, when people don't get feedback they often

  • tend to feel like they're throwing

  • explanations into the void.

  • They're just talking and it's disappearing.

  • And so one thing I'm going to try is try and couple this

  • really closely with actually giving them an explanation, a

  • correct one, or almost correct one.

  • And tell them that's another student's or teacher's

  • explanation.

  • And so the idea is not only change a task from do

  • something that the teacher or instructor's asking you to do,

  • type something in, to generate an explanation, and then

  • you're going to be comparing it to what someone else said.

  • Actually the idea here is that at this point you could ask

  • them to do things like, once they've got an explanation

  • written down and they see someone else's, well, what if

  • you grade both of them?

  • Do you think that was a good one on a scale from one to 10?

  • Was your explanation good on a scale from one to 10?

  • You could have them elaborate on what they thought might be

  • better or worse about an explanation.

  • Or you could even just have them make ratings like a

  • similarity or dissimilarity.

  • So it's not very common to ask for these kinds of ratings

  • that's in a MOOC or online.

  • And I think it could be because often if you just ask

  • a question, and at random, 1 to 10 rating, people may not

  • give informative answers.

  • But if you use the kind of assessments that psychologists

  • have really honed down on for understanding things like what

  • concepts people think are similar, or people's ability

  • to predict when they make errors.

  • I think there's some really useful information which we

  • get out of this.

  • And what's even better is you're getting information

  • about what students think, and you're helping them learn at

  • the same time.

  • You have to comparing what they degenerate into what

  • someone else generates.

  • It could be very powerful.

  • There's a good reason to think it would be very

  • powerful for learning.

  • Another thing that would fit in really well here is that

  • there's work by people at Khan Academy, like

  • [? Carolyn Rosie ?]

  • and David Adamson.

  • And what they've got is they've found a way to

  • leverage the minimum amount of AIU you can get into a web

  • based system to get lots of students learning.

  • And so the idea here is that when people might type in

  • something like this, like explain why you

  • think that's correct.

  • They're placed in a small chat room with a

  • couple of other students.

  • So then this is not embedded in the context of necessarily

  • a Khan Academy exercise.

  • But it's embedded in the context of trying to solve a

  • problem together.

  • And so someone has to provide an explanation.

  • And then the chatbot can then say things like well, John,

  • what do you think about that person's explanation?

  • How would you say it in your own words?

  • Do you disagree?

  • And so these are simple questions you can ask people.

  • But they've been shown to have a really powerful effect on

  • students' learning.

  • The term for this class of strategies, like accountable

  • talk moves.

  • And the idea is if I get people to really think about

  • what they're saying and how it differs from other people's

  • ideas, you can get a lot of learning.

  • And once you have that chatbot in there, it can also act as a

  • sort of instructional tutor, but one that doesn't need a

  • lot of attention, a lot of time.

  • Because students will often carry a lot of the

  • conversation themselves.

  • And the key thing is just doing things like adding

  • prompts to keep it going.

  • Or matching a student's explanation to something else

  • in it's database.

  • And then not saying, and just asking a student, is

  • this what you meant?

  • And so it's a great way that very simple technology you can

  • give an answer and the answer's probably not going to

  • be taught right because NLP isn't there.

  • But then actually the student's having a learning

  • experience because they're now trying to correct what you

  • said and put exactly what they were trying to get across.

  • And it's a really exciting line of work to look at.

  • So in terms of thinking about the benefits of explanation, I

  • think it's a great way to have instructor guidance, because

  • you can ask people questions that

  • guide how they're thinking.

  • While at the same time having learners generate information

  • or generate knowledge.

  • So they're actually constructing it.

  • It's also especially on online courses useful because you can

  • do learning without any feedback.

  • So you might have to do things to get people to actually

  • generate explanations, but it's a great way of actually

  • maximizing learning without necessarily having, even if

  • you don't have the resources to give people

  • individualized feedback.

  • And again it's really key, because as I mentioned,

  • transfer is really challenging to people, so the fact that

  • explanations help him understand these abstract

  • principles can have a very powerful

  • effect on their learning.

  • So in terms of thinking of what you can--

  • do you have a question?

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: That's a planned study, that's a

  • planned study, yeah.

  • So--

  • AUDIENCE: [INAUDIBLE]?

  • JOSEPH JAY WILLIAMS: Yeah.

  • I think we're really excited to look at--

  • oh sorry, the question was, did we find effects of having

  • them grade other people's explanations?

  • I was just saying that that's a planned study.

  • Because we can update tons of things in the lab when using

  • software [INAUDIBLE].

  • But when we have to deal with a platform that has its

  • constraints, unfortunately it takes a lot of adaptation.

  • Just on a side point, actually, one way I think that

  • research psychologists can be really relevant to people

  • actually doing practical work, like in edX and of Coursera,

  • in Khan Academy is they have a lot of constraints in terms of

  • the system they have to build has to scale,

  • getting things done.

  • But actually I think something like a sand-boxing addition to

  • any of those platforms will be very useful.

  • For example there's like 50, I mean literally 50 softwares

  • that don't require programming experience.

  • You can author an experiment really quickly.

  • It won't be as beautiful or as glossy as maybe, we think,

  • Khan Academy's perfected.

  • But it would be a great way to test out very quickly and

  • cheaply lots of different strategies for learning.

  • And then once you figure out which ones seem to work really

  • well for learners, that's what you put your

  • development time into.

  • Into actually investing it.

  • For example with Google's Open Course Builder, maybe there's

  • functionality you're thinking of implementing.

  • Well before doing that, you could try and add an addition

  • to a power searching course where some people will opt

  • into take in a different software, and then we can

  • replicate tons of things in terms of the questions they

  • asked or the examples they see or the format of video

  • presentation.

  • And then when you see which ones are really effective

  • that's some functionality we can then

  • build into the platform.

  • So in terms of thinking what people can do after learning,

  • after the study is done that actually has a

  • huge effect on learning.

  • I think a really interesting idea here is to use

  • assessments as instructional tools.

  • So normally we think of a test as just being to measure what

  • someone's done.

  • So it's like an inconvenience.

  • It's taking away time from learning.

  • And actually at Mackenzie, for example, especially when they

  • do training of soft skills, the instructional designers

  • complain that when they spend 20% of the time actually

  • testing people on this, it actually didn't think it was a

  • waste of time, because that was less instruction people

  • were getting.

  • I think that's something that maybe needs to be re-thought

  • in some ways.

  • Because there's now really good evidence that actually

  • testing people can have a really powerful effect on what

  • they learn.

  • If you think of the traditional model where you do

  • lesson one, lesson two, and lesson three.

  • And then maybe you do them again.

  • You drop some more information into the bucket.

  • Versus actually after each of those you actually add an

  • assessment.

  • So there's actually tons of reasons

  • testing can help learning.

  • For example if it gives you feedback, if it makes you more

  • motivated to study.

  • But I'm actually going to talk about the effects of testing

  • that are only due to taking the test.

  • So there's no feedback involved, and you're not

  • allowed to restudy.

  • So the idea is that something like mnemonic or there's

  • something cognitive about having to

  • generate information yourself.

  • Even though you get no new information

  • and you don't study.

  • And on a test you normally generate much less than what

  • was actually in the actual lesson.

  • So this refers to the testing effect.

  • As you can imagine, if you give a learner the option to

  • study something once, and then following that they have the

  • option of studying it again, versus they had the option of

  • studying it and then taking a test.

  • Most learners would opt for this first one.

  • And in fact, there's good data that they do.

  • And in a way it's sort of strange to think that taking a

  • test would be helpful because you can't recall the 200 facts

  • from the video.

  • You can only recall a subset of them.

  • But what's actually been found is if you look at immediate

  • testing, so let's say you do the two videos, you do a

  • version of test, and then you, let's say, do one more test

  • short after.

  • You actually don't find so much for difference, or you

  • find a slight advantage for spending extra time studying.

  • But if you wait an hour or a few days or weeks, what you

  • actually find is that studying twice is definitely much worse

  • than studying and being tested.

  • In fact the most recent paper in science found that you

  • could have people study four times in a row, and it wasn't

  • as good as studying and then testing.

  • So the key thing that's going on here is thinking about the

  • retrieval stage of learning.

  • Where it's not just about having

  • information in your mind.

  • It's actually able to get to get at it.

  • And when you do a test you're forced to form the cues or the

  • retrieval, pathways or connections that will help you

  • get back that information.

  • What's interesting is that in all these studies, even though

  • they were doing worse in the study/study condition.

  • They claimed that they were doing better.

  • And they expected their performance to he higher.

  • Another really interesting finding actually that I think

  • has gotten less attention but it's very powerful is what's

  • called the Mixing Effect.

  • So let's say you have questions of type A, B, and C.

  • Sorry, lessons A, B, and C. It's natural to put the

  • assessments for A, whatever 3x says you have right

  • afterwards, and so on for B and C.

  • But the idea behind the Mixing Effect is, don't make new

  • exercises that do anything different.

  • Or just scramble the order.

  • So if after you learn A, you get one question A. After you

  • learn B you get a question B, a question A, and a question

  • B. And then when you finally finish it out on C, you get a

  • mix of all of them.

  • And so this is especially dramatic because you haven't

  • increased how much exercises you have to make.

  • Learners do terribly when they're doing this and they

  • think they're doing really badly.

  • But it has profound effects on their ability to actually

  • generalize.

  • Because the idea is that when you do an exercise and them

  • move straight into having a bunch of questions, it's

  • really easy to know which principle applies.

  • When you mix them together it's really necessary for them

  • to figure out what the correct principles are

  • and where they apply.

  • And so even though they do much worse in the learning

  • stage they do far better in generalizing.

  • And this something you can easily do

  • in a textbook, right?

  • Take all the exercises and start mixing them around.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: Yeah.

  • Let me think a second.

  • So there's no study I can think of that said that.

  • Even though there might be one out there.

  • But I should have thought.

  • Because that would definitely be, as you

  • said, a great point.

  • AUDIENCE: The idea is to--

  • before you [INAUDIBLE]?

  • JOSEPH JAY WILLIAMS: Yeah.

  • That's a really good point, actually, yeah.

  • So I'll check the quiz.

  • And so this is the quiz [INAUDIBLE] on Mixing Effects.

  • So I'll take a look at the paper and see if these

  • [INAUDIBLE].

  • But that's true, that would work really well.

  • And it's not something, I think, that would necessarily

  • be obvious in this perspective.

  • But if you put together the problem-based learning work,

  • then it does make a lot of sense.

  • So basically I think what's really exciting about this

  • work is, that basically when you have online education you

  • can't have a real world lab.

  • We have to be in a lab of psychologists because you need

  • control and you have to be able to measure

  • what people are doing.

  • But now that people have moved online, you've got a real

  • world environment that's exactly like a lab context.

  • You can do experiments, randomize assignments,

  • precisely deliver and control.

  • And you can have really quantitative measures of

  • learning that are automatically connected.

  • And so I hope that in the future what that means is that

  • you get a lot more cognitive scientists and experiment

  • psychologists actually working directly in

  • online education platforms.

  • Whereas normally it's incredibly difficult to do

  • education research.

  • You can't have a teacher around and assign instruction

  • to different students.

  • They just can't do it, you know?

  • Whereas you can have two different videos show up.

  • What's also really nice, then, is that we're doing research

  • in a context with really rapid ecological validity.

  • And you're actually doing research on a product.

  • Where when you're done with your theoretical research it's

  • actually been improved.

  • It's better for learning.

  • And so what you can have is a lot of

  • evidence-based decisions.

  • You can take that product and deliver

  • it directly to someone.

  • So there's perfect fidelity to treatment.

  • Whereas a lot of research conclusions, even if they work

  • really well when the research is carried out, it's hard to

  • disseminate it, because the way someone else implements it

  • isn't guaranteed to be true to that.

  • There's great scalability because you've got these

  • implements in online system.

  • And especially you've got the knowledge really wrapped up in

  • the precise things that the researchers decide to ask.

  • And of course iterative improvement where it's

  • constantly getting better.

  • So given that all of these methods are improving

  • learning, I'm coming towards the end so I'm going to touch

  • on these things really quickly.

  • But the idea is, if you've got these ways of teaching a

  • concept that seem to be effective, well, what concepts

  • do you want to teach that are going to

  • have the maximum effect?

  • And any kind of concept that improves people's future

  • learning is a good candidate.

  • So compare with your motivation, learning

  • strategies and online search.

  • And so in terms of increasing motivation, the idea here is

  • that one really powerful way to motivate people is actually

  • by targeting their beliefs about intelligence.

  • So do you agree, for example, that your intelligence is

  • something very basic about you that you

  • can't change very much?

  • Or do you think no matter how much intelligence you have,

  • you can always change it quite a bit?

  • And so Carol Dweck writes that as being a fixed theory of

  • this malleable theory of intelligence.

  • And it's pretty shocking what this predicts in terms of

  • people who start off with equal IQ's will diverge.

  • Because people with fixed theory

  • avoid taking hard problems.

  • People with a malleable theory always ask questions, they're

  • happy for challenges, they don't think it means something

  • negative about them.

  • And so what are the effects of actually teaching people a

  • malleable theory?

  • And so this is work that's done by PERTS, a project for

  • education that researches [INAUDIBLE] at Stanford.

  • That's Dave Paunesku and Carissa.

  • Actually Carissa's is right here, she just

  • wanted to sit it.

  • And so what they did is trying to instill all these insights

  • into a single lesson, where people will be taught that the

  • brain's malleable.

  • Then they would have some questions and make them think

  • about how it applies to their life.

  • And you can contrast the control commission where the

  • same lesson will teach people about the brain, but not

  • necessarily emphasize yet that your brain changes every time

  • you learn a new fact and that intelligence is constantly

  • being improved, do exercises, and then try and take some

  • measurements of what people's mindsets are.

  • And so this is great in terms of actually being able to see

  • what effect this has on students.

  • And instead of this whole system of actually running in

  • high schools and middle schools and so on.

  • So it's got all those great features of a real world lab

  • that I mentioned.

  • But in addition to asking about mindset, they also just

  • collect the student's grades.

  • And normally I wouldn't collect measures on

  • [INAUDIBLE] grades, because it's so hard to impact

  • something that has so many variables fitting into it.

  • And there's a lot that goes on.

  • But actually what they found in this randomized controlled

  • trial is that just two lessons of about 45 minutes actually

  • boosted students' GPA.

  • Now you might think grades are not the best measure of

  • things, but if someone knows how to change grades they

  • really understand something really deep about the

  • psychological structure of what students are doing.

  • And you can imagine these kinds of effects applying to

  • all kinds of contexts.

  • For example in a workplace in terms of changing people's

  • beliefs about hard work or about what it means to get

  • negative feedback.

  • And so in terms of how I was thinking it would be good to

  • bring cognitive and social psychology together in this

  • way is just applying some of these principles like

  • preparatory questions during the actual learning of the

  • materials, have explanations, and after, questions that

  • repeatedly have applied the concept.

  • And so with Dave and Carissa, I have developed some initial

  • measures of how people might apply mindsets.

  • And we'd like to create video versions for MOOCs and Khan

  • Academy and that you can give to someone.

  • But we haven't gotten to that yet.

  • But what I've done is taken those cognitive principles and

  • applied them to the text they had.

  • And try and take this simple idea of a mindset, but teach

  • people it in a way that they're more likely to

  • remember it and apply it to situations in everyday life.

  • And so that's actually running now, but in so many lab

  • studies at least it is having an effective where even if my

  • inset is better than the control for getting people to

  • believe in a growth mindset and make judgments about other

  • people that are consistent with the malleable theory, the

  • effect of addition of [? Khan ?]

  • [INAUDIBLE]

  • is increasing that further.

  • Well, that's backward.

  • But we have our fingers crossed that it's going to

  • turn out successfully, and especially

  • if it impacts grades.

  • Another thing that is really interesting is there's so many

  • ways this mindset idea can be implemented into courses.

  • For example changing feedback in exercises.

  • It's natural to tell someone who does well

  • that they did great.

  • And it's even better to tell them they're smart.

  • But the problem is when you tell someone that they're

  • smart, you're emphasizing something internal and fixed

  • about them, a quality.

  • And so there's actually work showing that that can actually

  • undermine children's abilities.

  • By praising them for their intelligence they can actually

  • start to become very resistant of trying things out or being

  • right about things, showing that their dumb.

  • Whereas praising children for their effort, or praising them

  • for the way they've set up a problem or approached

  • something or tried to find out more is actually a lot more

  • effective in the long run.

  • As you can imagine, Khan Academy exercises in workplace

  • evaluations try to frame things from the perspective of

  • improving or people's ability to be malleable can have a

  • really powerful effect on learning.

  • And so just quickly to go through, I guess I'll just

  • skip over this.

  • So basically I'd like to apply the same concept to learning

  • strategies.

  • And so the idea here is to integrate work not just from

  • cognitive psychology, but also ideas about mindset and habit

  • and behavior change, which is it's own separate literature.

  • Get people to stop thinking about learning in terms of

  • memorizing facts but actually in terms of understanding.

  • And in particular that what it means to learn a concept is to

  • be able to understand it, to be able to teach

  • it to someone else.

  • So the idea would be within a MOOC where they collect grades

  • automatically.

  • It's a great situation to introduce training videos on

  • learning strategies.

  • Like whenever you're studying something, imagine you're

  • going to be teaching it to someone.

  • Or when you finish something, imagine if you had to explain

  • what you learned to someone else.

  • And how technology can prompt people to do this.

  • And then see if it actually does have an

  • effect on final grades.

  • And MOOCs provide a great way to actually do that data, that

  • kind of longitudinal study that you

  • just couldn't do before.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: What?

  • Oh, sorry.

  • OK.

  • A MOOC is a Massive Open Online Course.

  • So a course there on Google Power Search.

  • You have 20,000 people and you teach a course.

  • OK.

  • And online search.

  • I guess you guys are all pretty much on board with

  • online search being important.

  • But actually I think everyone knows it's good

  • to be able to search.

  • But often we think if it as just like a tool, like it's

  • for something like fishing.

  • Versus like a tool kit for problem solving.

  • And online search I think is powerful for teaching because

  • if a student has problems in the math problem, and they

  • just need a habit or they have the skills to be able to find

  • resources on the internet.

  • They can find a site that's better for them than maybe

  • their current textbook.

  • They could find a site that offers free tutoring.

  • If they ever later on in life need to solve a similar

  • problem, they can go online and find those resources.

  • And so I think online education that teaches

  • students better search skills or especially to search for

  • knowledge as opposed to just being able to find product

  • reviews online.

  • It can be really powerful in terms of learning.

  • The fact that there's Google Scholar, it simply means that

  • if I hear a medical claim and I don't know if it's true, I

  • can go and look on Google Scholar and just

  • see what turns up.

  • I mean, I don't understand exactly just the journal, but

  • if I find a journal article relevant to it,

  • it tells me a lot.

  • So these kind of strategies could actually be a great way

  • to improve scientific literacy.

  • Even of course being able to pursue a project, errors that

  • are not under your current interest is also a great power

  • of getting students in the habit of online search.

  • Well then you're allowing it to a bit more independently

  • driven if they want to learn about a

  • topic that wasn't covered.

  • And you can even imagine using online search to do things

  • like, what are good learning strategies?

  • Or, how can I improve my grades?

  • We don't think to go and Google that online but it's

  • amazing what you will find on Google these days.

  • So in terms of just--

  • OK, I'm over, but the one point I'd make is cognitive

  • science doesn't just have to use instruction design.

  • I think it ready interfaces nicely with the machine

  • learning and work that people want to do.

  • Right now we've got binary answers on

  • multiple choice questions.

  • And it's true it's big data in that it's on the order of tons

  • of information.

  • But if you have a narrow bandwidth, you just can't that

  • get much out of it.

  • And so I think subtle changes to these things can make a big

  • difference in what we can get out of machine learning and

  • data mining.

  • For example, having people rate the plausibility of each

  • of four multiple choice questions can raise our binary

  • score into four numbers that each correspond to a different

  • kind of concept.

  • Just including those are standard practice.

  • For example, you just have them read one to seven

  • [INAUDIBLE].

  • And so be like, everyone understands.

  • Having people predict their accuracy is also a really

  • powerful thing, because there was a study that [INAUDIBLE]

  • teaches students, and then saw how it correlated with their

  • later performance in undergrad and the SAT college

  • performance.

  • But it didn't correlate as well as how accurate students

  • were in predicting which questions they got wrong and

  • which they got right.

  • So this study had you do an essay question and say how

  • likely are you to get that right.

  • And the fact that you were good at judging what you're

  • were getting wrong or right was a much better predictor of

  • undergrad success than actually the SAT itself.

  • And so if you're looking for an individual difference in

  • learners, that would be a great one.

  • How good they are at understanding gaps in their

  • own knowledge.

  • We think similarity.

  • It seems really simple, but that's again something that's

  • just been shown across tons of studies to reveal something

  • very deep about people's minds.

  • And I think also grading explanations.

  • So again that would be a much richer data set than is

  • currently available for actually understanding

  • what's going on.

  • And that's where you might see data turn into

  • the best data yet.

  • So let's say I give you two concepts and I say, how

  • similar is this mathematical concept to this

  • mathematical concept?

  • Well two explanations, and I say, well,

  • how similar are these?

  • How different?

  • And that's actually a really powerful way of getting to

  • what people think or believe.

  • OK.

  • And I'm happy to talk about this some more.

  • But I've tried to put together the resources that have been

  • useful with me on a website in terms of citations.

  • And if you're really interested in improving

  • education, this is a great book in education policy that

  • looked at all the systems across the world and tried to

  • see what insights that they have for US education.

  • And I think especially online education is different.

  • Obviously there's a lot that's already known.

  • So we could benefit from a lot of that.

  • All right.

  • Thank you.

  • [APPLAUSE]

  • And do people have any questions?

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: So the question was, I was talking

  • about subjects to explain by drawing on knowledge that they

  • already had, and how did I measure it?

  • There are many different ways people take to it.

  • Mostly past studies like you said administered a pre-test.

  • The way I did it was actually I manipulated knowledge

  • experimentally.

  • So they explained or didn't.

  • And there's another experimental factor, I gave

  • them extra training or I didn't.

  • And so that's a way we can know exactly what

  • they've been learning.

  • AUDIENCE: That's one of the benefits of [INAUDIBLE]

  • is that they have learning without feedback.

  • But you mentioned that explanations were [INAUDIBLE].

  • Or those people could have had an idea of what you were

  • thinking, have a way of learning without feedback.

  • JOSEPH JAY WILLIAMS: Yeah.

  • So I think that's a really good point.

  • I think that's why I tried to see well, what is exactly is

  • an effective explanation?

  • Because then, once we have a bit of a better handle on what

  • explaining is doing, if it's driving people towards

  • principles, then we have a better sense of, OK, it's not

  • a good idea to ask this person to explain at

  • this point in time.

  • We should wait to ask them to explain when

  • they have more knowledge.

  • Or it would be a bad idea to ask them to explain why these

  • two problems are correct.

  • Because that might lead them to a local generalization that

  • only applies to those two math problems.

  • For example with the statistics example, it's

  • pretty reasonable to say people with higher scores

  • would be ranked more highly, but that's just accounting for

  • the observations you've seen so far.

  • It won't hold in the general case.

  • So I think features like that in terms of how you set up the

  • environment and which cases you ask people to explain

  • would be important variables to look at.

  • So I think then more research or using research that's

  • already there would be useful for that.

  • Another thing to keep in mind is actually that sometimes it

  • can actually be helpful for people to produce a wrong

  • explanation.

  • And there's a paper that suggests this and has some

  • evidence from the recording they did but it wasn't tested

  • experimentally.

  • Which is that once you've produced a misconception and

  • qualified it, it's then a lot easier to recognize it.

  • Because then you're not just implicitly

  • thinking through things.

  • You've actually made a statement.

  • Like, I think that that person's ranked high because

  • they got a higher grade.

  • And then when you see data that actually goes against

  • that, you can then revise your belief.

  • And for example, that's how a lot of children's learning

  • might work.

  • Where you could say, my hypothesis' case is flat.

  • And I'm not going to do something if I

  • know a lot of data.

  • Or you could say I'm going to try a lot of things out, and

  • then just be really open to revise them quickly.

  • AUDIENCE: [INAUDIBLE].

  • JOSEPH JAY WILLIAMS: Yeah.

  • So on that resource page I have some reference of the

  • [INAUDIBLE].

  • And I'll let you know some more.

  • But Eric Mazur, who's a physicists at Harvard, has

  • pushed this idea of, it's not a flipped classroom, but where

  • every class actually is structured around questions in

  • the sense that he'll start a lecture.

  • After five minutes he'll put a question on the board.

  • And people will start discussing it.

  • They'll turn to each other and they'll say what they think.

  • Then they'll argue for, they'll argue against.

  • Then he'll pass around and talk with them and they'll get

  • back together.

  • And so I think that's got a lot of really

  • good lessons in there.

  • And also, this idea I think I

  • mentioned, reciprocal teaching.

  • Trying to teach someone a concept and have them teach it

  • back to you.

  • And I think that in terms of online environments, what

  • would be really powerful would be having people generate free

  • responses, and then again letting them see other

  • people's explanations, and use those as a base for

  • comparison.

  • And so that's something that I think you could get people to

  • be a lot more engaged.

  • Plus get the benefits of generating information, plus

  • it gets the benefits of things that are

  • like the right answer.

  • So that's what I'm really excited about

  • also trying that out.

  • A big thing that I think that people do now is because they

  • think that they have to give someone an explanation, there

  • are very few free responses.

  • So like the cognitive tutors, which are these adaptive

  • programs, will actually give people a multiple choice

  • instead of explanations.

  • And I think this simple manipulation of not having

  • multiple choice, having a free response, and then putting the

  • multiple choice can make a really big difference in terms

  • of what people learn.

  • You had a question?

  • AUDIENCE: You're running out of time.

  • JOSEPH JAY WILLIAMS: Oh, OK.

  • AUDIENCE: And so I think if anybody had questions for

  • Joseph, feel free to ask them after the talk.

  • JOSEPH JAY WILLIAMS: Thanks for having me.

JOSEPH JAY WILLIAMS: So I just want to mention, my background

字幕與單字

單字即點即查 點擊單字可以查詢單字解釋

A2 初級

Cognitive Science如何改善線上學習(How Can Cognitive Science Improve Online Learning?)

  • 251 33
    Why Why 發佈於 2013 年 03 月 29 日
影片單字