Just kind of doing unconventional things with neural networks.

And the type of model that we're going to start off with is the generative model.

So what I have here is a simple program, basically where I just asked the model, Hey, draw me a number of zero through nine.

I'll pick the number I'll say, Please draw me a five, for example, and that five becomes a sort of primer for the generative model.

The gender of models then generates me something like a five now.

The interesting thing about this model, though, is that first of all, this five is a totally unique new novel.

Five.

It's never it's not in the data set.

We've never seen it before.

And if I ask you to draw me yet another five, what it's gonna do is draw me yet another unique new novel.

We haven't seen it before.

Five and so So the reason why I think generative models air interesting is one that they're still like a really like area of research and a lot of people in the fields of deep learning, Um, kind of write off generative models.

I'm drawing a four, by the way, Just so you know, four.

Anyways, the people in the fields of machine learning tend to write off generative models because they don't quite fit or paradigms today of problems that we'd like to see solved.

So classifier is, on the other hand, with deep learning.

There's kind of two ways that class fires with deep learning have improved, and that's with incrementally better accuracy and results.

And then also with, um, there's our nice beautiful for you get the idea, um, and then also with solving problems where there's, like, millions of samples that we just could not train with support vector machines.

But arguably, I would say those results are kind of based more on, ah, just using GPS generally So g PG BYU's, um and so those two things kind of happened at the same time.

So anyways, so incrementally better results and then able to solve some classification problems that we've never had before.

But with generative models there, it just opens up on entire new door.

And so just cause we can't think of, uh, use cases for them, doesn't mean, we need to write them off immediately.

Um, I think probably part of the problem is a lot of times funding for research and stuff is done by companies and stuff like that that have a use case in mind.

And I think that's why we see these as being useless.

But anyways, um, no one's funding me.

Besides, you guys so python programming, and that's our support.

And, uh so anyways, we get to play around and do fun things.

So with that, let's get into the tutorial where we're going, t get to this point, but also do some other playing around with other things, Um, and just kind of poke around.

So what we've got what?

I'm working with his python 3.6 tensorflow 1.7, and then we're also going to at least start off using the following package.

So hello here.

So, um, the the commit for this exactly is for one E b f D.

Um, it doesn't look like it's to actively maintained, so maybe it won't be a problem, But if anything is not working, you can get the same python version, same tensorflow and same same package that we're gonna use here.

So go ahead and clone, download that and somehow got a copy of it.

And then what we're gonna do is we'll break out of this one, move that aside, and once you have the package will go ahead and extract it.

And let's just go over what this is.

So this is a character level generative model, and what's gonna do is you just give it any characters and then what it's gonna do is you're gonna say OK, with all these characters, I want you to you train on it and then be able to generate me something similar to what you've seen.

So it's It's like a classifier in the sense that you can produce.

Um, you know, it's there, there's an input and there's an output, and the goal is to fit the output to the input based on the training data.

So it's very similar to class fires in that sense, but the difference is a generative model can take a variable number of input, and it can produce a variable sized output.

And with tensorflow recurrent, we do have these dynamic, recurrent layers which are slowly becoming capable of doing this sort of thing.

Um, but not quite the same.

So I think the best way Thio to show you guys what generative models do is to just kind of use one and work with it.

So So this is the package we can check out.

Trained up.

I just kind of see what what's going on.

But basically, you're gonna pass in, You know, you're an astronaut pie.

It's gonna look in your data dirt for something, But we can We can use the command line arguments, Thio pass some other form of data.

We're gonna ask it to you know, we were gonna tell it where to save, where to put the tents are bored logs basically the size of the network, all that stuff.

But there's there's defaults for all of this.

Um, so you can kind of look through that once you start getting past sequence length, maybe number of at a pox.

Um, you You probably are are done changing things.

I wouldn't suggest you mess with these until wait on one.

But the main thing is to look out for or the size of the model, um, maybe batch size.

If you make a model too big that it can't fit in the memory.

You could decrease batch size.

If it's not too big, you could increase batch size to use more of your utilize more of your jeep.

You possibly, um, what was the other thing?

Oh, sequence.

Length.

Um, so how long of sequences?

Right now, it's gonna look at the last 50 characters each time it wants to generate.

One new character is gonna look back 50 characters.

So that's what's gonna do.

Also, we can see that there's maybe Cem Cem starting data.

So let's check out data slash tiny Shakespeare.

And basically, we're looking for input dot text, so it's good.

Let's go data Tiny Shakespeare input dot text And here's what we have.

So as we can see, this is some Shakespeare.

So it's a play, basically, and and there's a few things here that you want to kind of pick up.

One is, you see, there is some structure to it.

You've got basically what appears to be the structure of name colon.

New line sentence, New line, New line name, colon, New line sentence, New line, New line.

Since when?

Um so we have that also we can see that you basically got name, which should just be, like one or two words.

Most likely Ah, then the sentences could be highly variable.

Um, and then it's like old English, basically.

But it's clearly English.

Okay, Um, so yes, So that's our training set.

The other thing that I really think it's cool about generative models is the size of data that's required.

This is only one megabytes of data.

That's it.

So So the fact that if it works spoiler alert, it works.

If you haven't seen them spoilers already.

Um, that's really cool that you can do this with such a small amount of data.

So anyways, so that's our starting data set.

Let's go ahead and train the model.

So coming back into this main door where we've got trained up pie, let's go ahead and actually just run python trained up high.

So it's just gonna use all the defaults.

We're gonna change them, Don't worry.

But slowly entering into the world of generative models, um, let's make sure this actually starts training.

And once it does, I'll just posit, um and then all of a NPAs.

Okay, so there goes.

If yours is not training this fast.

Um, you're probably a CPU or maybe a lesser GPU.

But I'm pretty sure that, uh, this doesn't even come close to fully utilizing my gp.

Like I'm recording on it.

And my desktop is totally fine.

I could see my mouth's moving pretty pretty smoothly.

Anyways, um, what I'm gonna do is just pause it recording while this is running.

Ah, and then I'll pick back up when we've done.

I don't think I'm gonna finish all the pox, but I'd like to get us do at least a few.

So anyways, I'm gonna pause it while this is going and I'll pick back up, um, in a few minutes, but I'm not gonna stand here in total silence with you all Okay?

Actually, ah, found that it trained pretty quickly, so not a problem.

Now, Um, one thing that we can do is we can always check to see how training is going and kind of make a determination whether we need to continue training or not.

So, uh, you can always run.

Uh, oops.

Um, off screen.

It's just tensor board dash, dash log.

It's log Underscored her.

I can never remember.

No, it's one word.

Lager equals, and then we'll just say logs because that's where the logs are.

Um Then what we can do is we can navigate to this.

You are all here.

Years will be a little difference.

Pregnant HPC.

And here is our training.

So actually, to me, it looks like you probably could even kept going and primate a little bit more progress.

But that's okay.

Um so So it looks like it trained.

Let's see what it learned.

So the first thing that we're gonna d'oh is will quit out of here.

And now we want to run sample dot pie.

So with sample up, I we actually can just run straight sample that pie.

But it also has command line arguments that we could pass.

So you would say, Where's the save directory?

Um, you can prime it in this case, it's prime ing it with a space.

Ah, and then we connect, and then here we have the actual sample here, but I'm not gonna touch that, but but the big one to it is end.

So this is how many characters do we want?

A sample.

So it'll just continue going in the in this case is gonna generate 500 some samples.

So let's go ahead and pull back up where we were.

And let's go ahead and Python sampled up high and we'll just go with the defaults cause it's already set up to work with this data set.

But later on, we'll change some things.

So, um so here's Here's the return that we got, um, and you'll see that it's got, you know, the be here.

And the new lines aren't actually making new lines.

So one thing that we could do this is encoded for utf eight.

So one thing we could do is just not in code for utf eight.

Um, or we could just decode.

So, for example, we could say like, later on down the road, I think probably the best way to do this is gonna be actually something more like data data equals model, and you can, you know, do whatever you want to do with that data.

But then, if you wanted, you could just print data dot decode from ut f it.

But keep in mind that, you know, in this case, we don't want it to be utf eight because it's really just asking characters and new lines.

There's really no fancy characters.

But if you're using fancy characters like if you're you've got like, um, you know Chinese or something like that with the characters you're probably gonna want to keep, you know e t f ate.

Anyways, let's run it one more time just with sample that pie.

Um and then hopefully we'll get to see the actual structure a little better than what we're seeing here, right?

So as you can see why we even got Romeo, that's the first half left.

That's actually kind of interesting.

I've never seen it.

Ah, maintain usually makes up new names.

Queens.

Also there, um, I think I've ever seen it put Romeo.

That's cool.

Anyway, Pam, let's do end equals, uh, 2000.

Just do a little more than what we have.

Cool, anyway, so we can see is, um, it's clearly learned the whole name Colon, New line, all that stuff, and then in repeat, One thing I find interesting, though, is it's capitalizing the names.

Um, let's go back into the data set real quick.

Interesting.

So the beginning of this data set the names are not necessarily catalyzed because they're not names like it's like first.

It's like no unknown into T, I guess.

But then, once we learned once we have a name, it is all caps.

I guess that's just part of the structure.

Interesting.

Yeah, but if you actually know the person's name, it'll be in all caps.

So that I guess that's why that was in all caps.

I wasn't I was wondering why was it doing that?

Um, yeah, but you know, so So not only did learn the structure, it also learned.

I mean, some of these things aren't quite right, but you draw the Lord bounty cursed.

These are words about isn't really But, you know, my throngs in home.

Ah, by is not a word.

Um, you know, I'll say, there's, like, some words, not here.

Also, it's not really totally coherent, but at a quick glance, you would say, Yeah, that's definitely what we trained on.

So now, um, the next thing I know, I'm curious about when when I first saw this was could we if it can learn this structure, right.

This is pretty basic structure in English.

Um, could we get it to learn to code.

Like, could we get it to learn to do this on Python?

So that's what we're gonna do in the next tutorial is we're going to see if we can get a generative model to actually learn python instead of place.