Placeholder Image

字幕列表 影片播放

  • what's going on?

  • Everybody and welcomes you.

  • Part four of the M l in daylight.

  • Three tutorials.

  • Ah, where we left off we were creating for real.

  • Hopefully this time some training data gonna be pretty angry if this is not correct.

  • But we checked it last time, and I think I'm all set.

  • Um, Anyways, what I've got here is actually over 50,000 files and was 50 gigabytes of training data.

  • I think we're all set.

  • So I'm gonna go and stop all these from running, and we'll close these and we'll pop into here, come into training data, and let's sort by size, make sure looks like they're all between 2000 4 2000 and six.

  • Want to make sure none of them are just, like empty.

  • And then we can also sort by the name.

  • Looks like the largest is just under 5500 But pretty quickly, I mean, we only have a handful of 1000.

  • How light games.

  • Like where?

  • Where the Aye.

  • Aye, Somehow collected over 1000.

  • Which is pretty impressive because we're called.

  • We're only making one ship, so that starts at 5001 ship goes down to 4000 and we set this threshold to save as just basically 4100.

  • So did that ship collect anyhow, Light.

  • Now what we want to do is find out, um, where should we draw the line?

  • So s O, for example.

  • Um, it's come into this is just the data.

  • I don't really care about that.

  • Let's go over it.

  • Okay.

  • So I am, Um I guess we could take testing grounds.

  • Um, I think we'll just modify testing grounds is a couple of things I want to check.

  • Um, so the first thing would be let's do import O s.

  • And then all files equals O Estado lister training data.

  • Don't forget to put this in quotes and what I'd like to do import, Matt, plot lib DuPuy plot as peel tea and what I want to do.

  • Actually, this machine pride doesn't have Matt plot lips.

  • Let's go ahead and grab that.

  • Let me make sure Pip is to python that we're using.

  • Hurry up.

  • Okay.

  • Pit pit.

  • Been stall, Matt.

  • Lot lib.

  • So what, Grandma, plot lib.

  • Make sure we can.

  • We should be good, though.

  • I just wanna make sure no errors on.

  • What I'd like to do is plot, like a distribution of the score's kind of for two reasons.

  • One is I'm just curious right now.

  • But also, what we could do is keep this exact same threshold.

  • And then after we've trained a model, we can see, uh, is the distribution exactly the same as in, like, almost like, what's the average score of the games?

  • You also could do an average, but or both.

  • I'm curious to know, Uh, after we train a model, how much better did is our new model.

  • So coming back over here.

  • All files host out Lister trained data.

  • Let's just print Lin all files.

  • Run that really quick.

  • Let's see how quickly we can get through that pretty pretty darn fast.

  • So then four f because if we use file, I think that's Ah.

  • I thought file was a key word.

  • Anyway, I'm gonna keep with F for F in all files.

  • Um, so that should be the file name.

  • So then we just want to say, uh how light Hal item amount equals.

  • F dot split.

  • We'll split by the dash, and then we'll go with zero with elements.

  • So if we print, Hal, Iet amount will just break after the 1st 1 Brick, um, whom we could see 4100.

  • So then, um I'll just do how light amounts.

  • Here.

  • Uh, then we will delete Thio here, pal.

  • I amounts on DDE amount.

  • Uh, print.

  • Let's do print, Lynn, pal.

  • I underscore amounts also just so I can dev this to start, Let's just do, like, 500.

  • Cool.

  • Okay, so now we want a plot.

  • Hissed a gram.

  • I believe it's p l t dot hissed in Matt plot lib and then for a history, ma'am, you just have to pass exes.

  • So we want to pass, Hal.

  • I TTE amounts, and then it's like bins.

  • It might be a TSH bins or end bins.

  • We're gonna find out in a second.

  • I'm gonna say five and then peel tea that show Run that.

  • Whoo!

  • It worked.

  • Okay, so it's only 4100 because as we generate, I think we're just going in order.

  • So let's do, um, let's do 15th hated.

  • Let's do 1501st to see OK, so we can see clearly.

  • The most common is 4100 and it kind of goes down.

  • Uh, let's do them all.

  • I think I think that's looking pretty darn pretty darn gate.

  • Yeah.

  • I'm actually not sure why that was so difficult.

  • Why would that be so challenging for it?

  • Because we said we only wanted so many bins.

  • So why is the display so challenging for it?

  • Well, it's curious.

  • What about done wrong?

  • Um, I feel like it didn't actually put them in the bins.

  • Right.

  • Uh, hello.

  • Uh, like, pulling up real quick.

  • I don't know what I've done wrong here.

  • Um, it's too, uh, map plot live, hissed a gram pipe lot hissed.

  • That's correct.

  • See if we say you know, five bins.

  • It really should only have, like, five categories have been, uh, for clearly.

  • It wanted to label like a ba jillion.

  • Could you please not take five years to load and just, I don't know, load any of them.

  • These are all mad plot lived out or about this one.

  • This is a map lot live.

  • Yeah.

  • There we go.

  • Yeah.

  • How come?

  • How come they're bins?

  • Look normal.

  • Numb bins, X numbers.

  • How come miner is so ugly?

  • Uh, let him out.

  • Ben's equals five just to make sure Ben's is the key word.

  • It looks like that person passed.

  • It's still gonna be a pain.

  • Isn't it weird?

  • I don't know what I've done.

  • I thought with the bins, it would just show it would be like the range Almost.

  • But that doesn't appear to be the case.

  • Yeah, we got, like, all these 1,000,000 tick marks, Someone coming below what I've done wrong.

  • It's kind of a bummer.

  • I'll keep these around.

  • I'll keep the script.

  • Uh, the other thing we could do is, um I think it's from Is it statistics that has the mean operation from statistics import mean?

  • Um, for now, I'm just gonna comment this out, I guess because I don't think that's really working Print.

  • I mean, Hallet amounts.

  • What?

  • I was wrong.

  • Oh, dude.

  • Okay.

  • So I bet I know what I've done wrong up here.

  • Let's see.

  • So right now, these are not, uh this needs to be an integer, because right now it's a string.

  • We'll see if that fixed it.

  • Whoa.

  • Wow, man, what a dumb mistake.

  • Okay.

  • Okay.

  • So, uh, so this sort of helps.

  • I'd probably cut it off at about the 10,000 mark just to see what the real distribution is.

  • But yes, you can see it's a pretty significant or even maybe 2000 mark or something.

  • But you can see we got very few of these games and it takes up real fast.

  • Okay, Cool.

  • So then the other thing we really could do now is this.

  • And we get the mean, which is 42 02 So that's the average.

  • So later, after we train a model, we can come back and see what is the new meat.

  • Okay, so, um, so now that we've done that, um, I'm going.

  • What I'm gonna do is I'm gonna come in here training data old.

  • This is from before.

  • So over time, I'll probably keep it least the previous training data set.

  • Obviously, it's 50 gigabytes at some point.

  • We can't be doing that.

  • And soon we're gonna have multiple ships, and, uh, we're just gonna have a huge day sets so we can't keep too many of them.

  • Normally, I probably won't have 50,000 games, though that's a lot of games.

  • So really, we would just raise the threshold, Probably.

  • But for now, I do want to see if we can improve the mean game from one training example.

  • So testing grounds instead of calling that testing grounds Now what?

  • I'd like to call it I'm not sure why I had to save it.

  • Let me just test it real quick.

  • Here.

  • Cool.

  • Uh, what?

  • We're gonna call this instead?

  • Now we're gonna just copy paste.

  • I don't.

  • Why do we have another test?

  • Oh, this is the output.

  • Okay.

  • Coming back over here, I'm gonna get rid of test.

  • Has just taking up space.

  • Also sent a bought two.

  • I think we only needed that for testing as well.

  • I'm gonna delete that.

  • This I'm going to say, um uh, let's call this training data district distribution.

  • Cool.

  • Okay, so now let me just copy and paste either of these and we'll call this, uh, sent to train.

  • I don't know.

  • Anyway, basically is gonna train.

  • Um, we'll use this to be our training script.

  • Okay, So, to train, I'm gonna use tensorflow.

  • You could also use straight care off somebody's care us from tensorflow.

  • But you can do whatever you want.

  • If you don't know much about care oss, intense airflow and all that you could always go to python programming dot net.

  • Finally, it knows that one over you could type in care oz and then come down here and learn all about Cool.

  • Okay, So, uh, while we still can't get too mad plot This has been happening for, like, years were like, I just can't access Matt plot lib dot org's.

  • Sometimes I really wish they would figure out what the heck's going on there.

  • Anyway, I think we're done there.

  • Will open up a browser if we need it later.

  • So let's get started.

  • Import tensor flow as t.

  • I hate that sublime.

  • Does that to me.

  • Import tensorflow asked if we're going to use OS, we're gonna probably use numb, pious and P We're going to import time.

  • We are going to probably need random.

  • We need random toe like shuffle.

  • To my knowledge, there is no shuffle for secrets.

  • You can modify secrets to an extent toe have a shuffle, but that's not really the point of the secrets module.

  • So they don't have one.

  • Ah, we're also gonna from t Q d m import T Q t m, which I don't have ti que tm So it's going pip install T Q d m.

  • This just gives us a nice progress bar as you're doing things.

  • Since we have so many files as we iterated, it would be nice to know, where are we in this freaking process?

  • Um, okay.

  • Now from tenser.

  • In effect, Um, I'm not sure I can do this any faster than I can write it, but I might be able to, so it's gonna python permanent net.

  • Let's go back to that.

  • Care off Siri's, because you always just need, like, the same freaking imports.

  • So let's go down to how to use your train model.

  • Optimizing models, maybe lets you part three here.

  • So convolution all neural networks.

  • Let's see what the imports are to the statement thing.

  • Who did these awesome frickin pictures?

  • Yeah, basically all of this.

  • These like, sequential dense dropout activation.

  • Flatten calm, Studi.

  • Max pulling two d.

  • We need all that.

  • And I don't really see any point in typing all that out.

  • Really?

  • Sure.

  • This is faster.

  • Okay, so the only other thing that we want is gonna be from tensor float.

  • I dot care oz dot callbacks.

  • We want to import tensor board.

  • This is so we can display the beautiful, beautiful graphs as we go.

  • Okay, so I think we're ready.

  • So now, as we build out this model, we have, um, a few things that we need to do.

  • We we basically, uh the data set is too big toe load into memory.

  • First of all, we can chunk it down, probably and actually load the entire set onto a GPU, at least in the in the setting that it's in right now.

  • But I know if anybody's following the long and you don't have, say, a 24 gigabyte GP or a 12 or whatever and you can't fit in, the memory is important that we batch it in.

  • Now I know care.

  • Oz has some sort of generator, and you can load it in, and I've done stuff with that before.

  • But then when I need to start customizing stuff, that's where it starts getting unbelievably complicated.

  • So I just don't do it that way.

  • If you wanna, uh, use the generator and post how you would do it, go for it.

  • I just find it adds undo complexity to a problem.

  • And like in this case, I'm just trying to do like testing here.

  • I'm not trying to spend five days figuring out how I can create a custom generator that loads batches of files.

  • So anyway, not gonna do it, but yeah, if you want to do it, go for it.

  • Have fun.

  • Eso The first thing I'm gonna say is load train files, and I'm gonna set that to be false.

  • So basically true.

  • If we already have, uh, batch train files, I guess I'll call this train files.

  • This will make a little more sense when we get there, but basically, we're going to take these files because keep in mind, these are only one megabyte, right?

  • The training data.

  • So we're gonna want a load, you know?

  • Right.

  • 5100 200.

  • You could load quite a few, even like 1000 of these.

  • And it's no big deal, Probably depending on what side your memory is.

  • So But over time, the games are gonna get much, much bigger because right now we're only going for 50 turns and we only have one ship.

  • Now imagine later on.

  • We've got 500 turns, so it's already 10 megabytes and then imagine you've got 10 ships.

  • Well, that's 100 megabytes right there per game.

  • So, uh, so eventually it's gonna be even more important that we batch these files.

  • So that's what I'm going to do that from the very beginning eso for now to be false, but and then also like, if you're gonna train for more than one iPAQ or you want to train and then tinker something's and train again.

  • This what we can do is as we load and kind of combine everything and do the balancing all that stuff.

  • We only really need to do that one time.

  • So it would be useful to save it when we're done, because that's gonna take probably longer than the training is going to take, at least for a small model.

  • So then we're also gonna load lips meant to keep caps on load preview model again, I'm gonna set the false, but true.

  • If you want to transfer, learn from existing model.

  • Okay, Some like that.

  • Uh, I have to fix these.

  • Okay.

  • So, uh, now the final thing is, um, overtime, Like, right now, I just have a hal I TTE threshold.

  • So, for example, we could say, You know, Hal, I TTE threshold.

  • And initially it was 40.

  • 100.

  • Over time, we might want to change that.

  • So I'm gonna leave that for now on.

  • We'll come back to that that number, because again, we're probably end up changing that to something a little more dynamic as well.

  • Uh, so then also, we need a training chunk size.

  • And for now, I think we can set that to be even.

  • 100 is pretty good.

  • Probably 100 or even.

  • Maybe 200.

  • There'll be 200 megabytes more than you know, totally easy to fit onto Just about any GPU.

  • No problem.

  • We might even wanted that larger.

  • I'm gonna go with 500.

  • Um, Okay, so then if you have one pre ve Oops.

  • That's not what I wanted.

  • I want pre ve model name for now.

  • I'm just gonna leave that empty eggs.

  • We don't We don't actually have one.

  • Um then, um let's think you're, um I guess we would want a validation game count Will set that to be, like, how many games do we want to use for validation?

  • I think I'll say 50.

  • That means, like, 10% of each training chunk would be a validation.

  • 50 might be a little large, but I'll go 50.

  • That's fine.

  • Um Okay, so then, uh, we'll choose a name for this model.

  • So how are we gonna save it?

  • But, law, I'm gonna say this.

  • I'm gonna call it Phase One dash.

  • And then for now, let's just throw in its time dot time.

  • So that'll be the name of the model.

  • And then finally, a pox.

  • Let's just do one.

  • IPAQ, I don't really see any reason why we need to do a bunch.

  • Uh, then training data dir equals training data.

  • Fix this D cool.

  • Okay, so now what we want to do is generate over our, uh, our training data, and basically, we first need to get all of the files.

  • So training data names, uh, training.

  • Let's call this training file names.

  • And for now, that will be an empty list import or way before he got us.

  • That's not where you want to import it.

  • Four f and actually, let's just, uh oh, was it in here for f ine os dot lister?

  • So I don't really want to copy that exact thing sent to train.

  • Um, musty four f n o s dot list.

  • Dir uh, training data.

  • Dir.

  • Um Then what we want to say is basically the, um we'll say the she, uh let's say, Hal, I d equals.

  • And in that split, uh uh, this so it's a hallowed amount equals that split.

  • And then we'll ask the question if hal, I amount is greater than or equal to the hallowed threshold that we set.

  • If that's the case, then what we want to do is say, training file names.

  • Um, we want to Donna pend o s Stop, Pathak, join the training data.

  • Dir and, uh, that file names so f Okay, now, friend, let's just say after the threshold we have when, uh, training file names files.

  • Actually, we should call this games.

  • Uh, okay, so run that again.

  • Make sure all that works.

  • Kind of surprised how long that's taking.

  • You wouldn't expect that to take very long.

  • Okay, so we have all the game.

  • So what if we, um so clearly, 4100.

  • We don't need 52,000 games, especially for this first step.

  • So what if we said 4500?

  • We didn't see how few games we had after the, um, so 14.

  • 55.

  • So we want a little more than my target's about to thousands.

  • Let's try 4400.

  • Oh, wow.

  • So how about 44 feet?

  • Okay, I think we'll go with It's hard to say 40.

  • I don't know if it would be better to have the extra games.

  • I'll go.

  • I'll go with 4400 Just so we have plenty of games to learn from, I think that makes the most sense.

  • Okay, so with 32 35 games, that gives us a little over 150,000 samples to learn from.

  • Of course, most of those are just random and worthless, but hopefully some of them will be somewhat decent.

  • And that after 50 on Lee 50 turns, most of the moves or at least a large number of those moves, were actually probably pretty useful if we were able to gain 400.

  • How light from it.

  • With only one ship.

  • I think that's okay.

  • So, um, so at this point, we would want a random dot shuffle.

  • Those training founding sees you as you saw before they go in perfect order.

  • I'm not really sure we want to do that.

  • So random not shuffle training file names.

  • Okay.

  • And then at this point, um, I think what I want to do is probably stop it here, because the next point is really for us tow balance and then trained the data.

  • And I think if I'm gonna do that, I probably should split this off because the balance is definitely gonna take a considerable a man, a code.

  • The model shouldn't take too much.

  • In fact, we could probably copy and paste again from Mackerras Page and be good there.

  • Yeah, I think about for the balance.

  • I'm gonna split that away.

  • So anyways, I'm gonna stop here and in the next tutorial, what will do is we basically, you want to get to the point where you balance Now, this data probably is already pretty well balanced because it was a random choice and ah, pretty good random guys.

  • Were you secrets?

  • That choice?

  • So, actually, for the for this step, for the first step, we actually don't need to shuffle the date.

  • I'm sorry.

  • Balance the data because it is probably there is probably an equal amount of, like go up and left and right on down and don't move.

  • There's probably an equal amount of those because it was just randomly selected, for there might be a slight weight towards different things.

  • Like, for example, I think in Hal, I, you know, you might, depending on, you know, if the ship went left or right, Like most of the how light was over in one direction.

  • You know, you have way more turns or steps in that direction.

  • Also, probably still, like, hold still, maybe one of actually, the more common moves, because you're, like waiting on a plot of how light?

  • I don't really know.

  • Anyway, Regardless, we have to balance at some point, so might as well just throw in the code to do that.

  • Now, Um, so, yeah, that's what we'll do in the next tutorial.

  • So if you've got questions, comments, concerns, whatever, feel free to leave them below.

  • Um, yeah.

  • Otherwise I will see you guys in the next video where we balance create the model and hopefully start the training process.

  • And it shouldn't maybe in that training process, I'll just pause it while it trains.

  • It shouldn't take too too long to train, especially with only this many games.

what's going on?

字幕與單字

單字即點即查 點擊單字可以查詢單字解釋

B1 中級

查看數據--海立AI大賽中的深度學習第4頁 (Checking out Data - Deep Learning in Halite AI competition p.4)

  • 1 0
    林宜悉 發佈於 2021 年 01 月 14 日
影片單字