It was last year

A paper called "Concrete Problems in AI Safety"

Which is going to be related to the stuff I was talking about

before with the "Stop Button". It's got a bunch of authors; mostly from Google Brain

Google's AI research department, I guess..

Well a lot of it's AI research, but specifically Google Brain and some

people from Stanford and Berkeley and opening iEARN. Whatever... it's a

collaboration between a lot of different authors

The idea of the paper is trying to lay out a set of problems that we are able to

currently make progress on like if we're concerned about this far-off sort of

super intelligence stuff.. Sure; it seems important and it's interesting and

difficult and whatever, but it's quite difficult to sit down and actually do

anything about it because we don't know very much about what a super

intelligence would be like or how it would be implemented or whatever....

The idea of this paper is that it... It lays out some problems that we can tackle now

which will be helpful now and that I think will be helpful later on as well

with more advanced AI systems and making them safe as well. It lists five problems:

Avoiding negative side effects, which is quite closely related to the stuff we've

been talking about before with the stop button or the stamp collector. A lot of

the problems with that can be framed as negative side effects. They do the thing

you ask them to but in the process of doing that they do a lot of things but

you don't want them to. These are like the robot running over the baby right?

Yeah, anything where it does the thing you wanted it to, like it makes you the

cup of tea or it collects you stamps or whatever, but in the process of doing

that, it also does things you don't want it to do. So those are your negative side

effects. So that's the first of the research areas is how do we avoid these

negative side effects.. Then there's avoiding reward hacking, which is about

systems gaming their reward function. Doing something which technically counts

but isn't really what you intended the reward function to be. There's a lot of

different ways that that can manifest but this is like this is already a

common problem in machine learning systems where you come up with your

evaluation function or your reward function or whatever your objective

function and the system very carefully optimizes to exactly what you wrote and

then you realize what you wrote isn't what you meant. Scalable oversight is the

next one. It's a problem that human beings have all the time, anytime you've

started a new job. You don't know what to do and you have someone who does who's

supervising you. The question is what questions do you

ask and how many questions do you ask because current machine learning systems

can learn pretty well if you give them a million examples but you don't want your

robot to ask you a million questions, you know. You want it to only ask a few

questions and use that information efficiently to learn from you. Safe

exploration is the next one which is about, well, about safely exploring the

range of possible actions. So, you will want the system to experiment, you know,

try different things, try out different approaches. That's the only way it's

going to find what's going to work but there are some things that you don't

want it to try even once like the baby. Right, right.. Yeah you don't want it to

say "What happens if I run over this baby?" Do you want certain possible things

that it might consider trying to actually not try at all because you

can't afford to have them happen even once in the real world. Like a

thermonuclear war option; What happens if I do this? You don't want it to try that.

Is that the sort of thing that.. Yeah, yeah.. I'm thinking of war games.. Yes, yeah.. yeah. Global

Thermal Nuclear War . It runs through a simulation of every possible type of

nuclear war, right? But it does it in simulation. You want your system not to

run through every possible type of thermonuclear war in real life to find

out it doesn't work cause you can't.. It's too unsafe to do that even once. The last

area to look into is robustness to distributional shift. Yeah

It's a complicated term but the concept is not. It's just that the

situation can change over time. So you may end up; you may make something.

You train it; it performs well and then things change to be different from the

training scenario and that is inherently very difficult. It's something

humans struggle with. You find yourself in a situation you've

never been in before but the difference I think or one of the

useful things that humans do is, notice that there's a problem a lot of current

machine learning systems. If something changes underneath them

and their training is no longer useful they have no way of knowing that. So they

continue being just as confident in their answers that now make no sense

because they haven't noticed that there's a change. So.. if we can't

make systems that can just react to completely unforeseen circumstances, we

may be able to make systems that at least can recognize that they're in

unforeseen circumstances and ask for help and then maybe we have a scalable

supervision situation there where they recognize the problem and that's when

they ask for help. I suppose a simplified simplistic example of this is when you have

an out-of-date satnav and it doesn't seem to realize that you happen to be doing

70 miles an hour over a plowed field because somebody else, you know, built a

road there. Yeah, exactly. The general tendency of unless you program them

specifically not to; to just plow on with what they think they should be doing.

Yeah. It can cause problems and in a large scale heavily depended on , you know , in

this case, it's your sat-nav. So it's not too big of a deal because it's not

actually driving the car and you know what's wrong and you can ignore it

As AI systems become more important and more integrated into

everything, that kind of thing, can become a real problem.

Although, you would hope the car doesn't take you in plowed field in

first place. Yeah. Is it an open paper or does it

leave us with any answers? Yeah. So

the way it does all of these is it gives a quick outline of what the

problem is. The example they usually use is a cleaning robot like we've made this.

We've made a robot it's in an office or something and it's cleaning up and then

they sort of framed the different problems those things that could go

wrong in that scenario. So it's pretty similar to they get me a cup of tea and

don't run over the baby type set up. It's clean the office and, you know, not knock

anything over or destroy anything. And then, for each one, the paper talks about

possible approaches to each problem and

things we can work on, basically. Things that we don't know how to do yet but

which seem like they might be doable in a year or two and some careful thought

This paper. Is this one for people to read? Yeah, really good. It doesn't cover

anything like the range of the problems in AI safety but of the problems

specifically about avoiding accidents, because all of these are

these are ways of creating possible accidents, right? Possible causes of

accidents. There's all kinds of other problems you've been having in AI that

don't fall under accidents but within that area I think it covers everything

and it's quite readable. It's quite... It doesn't require really high-level because it's

an overview paper, doesn't require high-level AI understanding for the most

part. Anyone can read it and it's on archive so you know it's freely

available. These guys now working on AI safety, or did this then

They've hung their hat up. They've written a paper and they're hoping

someone else is gonna sort it all out. These people are working on AI

safety right now but they're not the only people. This paper was released in

summer of 2016, so it's been about a year since it came out and since then there

have been more advances and some of the problems posed have had really

interesting solutions or well.. Not solutions, early work, that looks like it

could become a solution or approaches new interesting ideas about ways to

tackle these problems. So I think as a paper, it's already been successful in

stirring new research and giving people a focus to build their AI safety research on

top of. So we just need to watch this space, right? Yeah, exactly..