Hello, world.

This is CS 50 Live where we do a bunch of programming from scratch.

Talk about technology would do all sorts of fun stuff.

C Studios.

Harvard's intro to computer science.

If you're not familiar, you can find all of our videos and stuff on youtube dot com slash CS 50.

So go ahead and check that out.

But today, So my name is Colt in Ogden.

Host the situation here, But today we have see cities Nick Wong here, and, uh, I'll let Nick take it away.

So we're continuing our kind of Siris on neural networks.

I thought attack people were asking, you know, linear algebra needed, like, required for no networks.

And I think you can actually attest to the fact that in order to get them working Not really.

Right.

Uh, experimenting over the weekend did all sorts of cool things, but actually, I'll ask you to talk about that a little bit in a sec, But to get him working to, like, run neural networks to follow with tutorial online, I'd say no, you don't really need in your, uh, linear out right.

You do need never lets to use little sleeping today, but yeah, I would say that you don't necessarily need it in order to get them to work in a very kind of technical sense of the question.

But I think that understanding some kind of fundamentals of linear algebra Ives, as was suggested in the chat, is super useful for actually doing meaningful things with neural network.

So if you want to answer your questions about why you're your own, that has just noise showing up every time you ask it to display an image.

Or if you want to understand why certain layers don't fit together or you need some sort of adjustment or your training some gan a generative, adversarial network and you want to know, you know, why are your images looking?

One way versus another than linear algebra is pretty helpful.

So what?

What did you find over the weekend?

Cold.

I did find a lot of noise, so I don't know if you want to do like a switcheroo of the laptop or if you wanted to know if we want Thio before we actually cut to the full view.

Now, I don't know that much at all about general generative adversarial networks for neural networks, at least in terms of the math.

You know how they sort of work in principle.

Um, but I was injured.

I am into game development, and there's a sort of appeal to me about generative art, right?

Like Sprite art.

Does that mean general generative art would be instead of an artist?

Let's say you and I are working on a game programmer.

You're the artist, and I want 100 weapons in my game.

It's a lot.

It's a lot of weapons you're gonna have to create every single one of them can barely draw like 11 sword, maybe.

Yeah, and you have to, like, creatively.

Think of all of the sort of differences between the weapons.

So there's a creative There's a creative cost.

There's a labor cost, a time cost that takes you away from doing other things that I might need you on.

Maybe also programmer, Maybe you could help.

Very.

Uh, I'm not sure you're talking about drawing eyes like I could also draw the trees and maybe like the shields.

Yeah.

Yeah.

Thank you for pointing down trees.

I brought firewood if you weren't where you were, but tentative.

A spreader is really cool because it takes away the cost of doing art as a human, which is a very slow process.

Even if you're a good artist, it's time consuming to make really good artwork.

And so I've always been interested in the appeal of potentially finding a holy grail of generative artwork.

And, um, generative adversarial networks are pretty cool.

People have done a lot of interesting things with them.

Um, there was a article on medium that I read about generative artwork, and I might even be able to actually find the article.

So medium sprite, our neural, you're gonna that a generative networks of super useful because they generate these very sharp images.

They could be used to generate things that look maybe even realistic photorealistic, which is crazy because a lot of these, like variation all autumn and coders or kind of like random variant generators are they generate that kind of pixelated images sometimes, or like if you're given an actual image, there's just too much complexity and like a human face or something like that for you to build something that looks realistic, based even on something that's realistic.

But these Ganz.

They have the ability to kind of question each other as you go.

Yeah, no, it's super cool.

The article that inspired this and I don't know how I actually ended up reading this was it's called Machine Learning is fun.

Part seven Abusing generative adversarial networks to make eight pixel art was like, I'm somebody who's interested in a large I'm also interested in generative stuff.

So there, those are the kind of goes into detail about, you know, Jenna references a paper on this thing called a DC gan which, actually what was what a.

D.

C.

Stands for trying to remember what that actually stood for.

Uh oh, deep convolution.

But a the end goal of this article was actually create pixel art that looked like this, and the sprites themselves aren't actually generated.

But all the tiles in this, the everything but a Simon here, this monster and this bat is all generated from a neural network, and this is done with Basically, they went into detail on how they did it.

But down here we scratched a little bit, and they go into obviously the details on how a d, uh, general adversarial network works and all that stuff which will actually end up getting to, if not this one, but like, one of the more extreme on general.

But basically, they took, I think, 10,000 screenshots of any s games, like all of these ones here him into a ah a a neural network.

And you can actually find the get every boat.

Uh, you could find the report on get hub and actually used it myself.

But it end up creating things that look like this.

So pretty, pretty chaotic.

Yeah, a little wild.

And, you know, you've talked about over fitting data.

It does over fit in a lot of situations.

You can see, like a direct copy and paste of certain menus and things like that in some of the screen show.

Definitely grab patterns, which is nice, but also grabbed a lot of noise.

Oh, yeah, I feel like that's a, uh, inevitability.

But they ended up creating stuff that looked like this, which is pretty cool.

So getting pretty close.

And they basically just took a bunch of tiles here from some of that work that they dio and then made a made a level lot of it, and what I wanted to end up doing myself was something very similar.

So what I did was I grab some artwork, for example, from the dungeon crawler stone dungeon crawl stone soup game which, if you go to dungeon crawl stones, soup tile.

I looked.

I was looking for the sliced tile set.

Um, where is it?

Here?

Open game are dot org's has it.

They're 32 by 32 tiles, and I apologize for anybody who are eager to get into, like the tensorflow part of this.

Okay, it's all good guesses.

A.

Basically a massive tile set for this game called Dungeon Crawl Stone Soup.

It's a rogue like game.

There's a ton of like, you know, weapons and enemies and items and all sorts of things.

Perfect for this.

Use cases are 32 by 32 pixel tiles, right?

And what I did was I grabbed that artwork I took.

I did a bunch of experiments, but we'll just talk about my most recent one.

I took all of the, um where is it's in data.

It's in the weapons.

I took all the weapons.

Everything that looks like this basically easy access maces.

Sword sides also fun stuff, but there's a lot of patterns were talking about this in advance, right?

They all kind of angle up into the right.

There's some ways that still like what's at the top of that versus the bottom oven.

A lot of cases, a lot of color variants.

Um, and I ran the network on this for 20 epics, which is just like a training period, sort of like an interval of time.

And what ended up coming out of that at the very end, with stuff that looked kind of like this.

So you can see there's, like, kind of a sword like this.

Halo weapons, yeah, is only after 20 mics, right, which is very short, pretty small period.

And on a CPU on a laptop on dhe, you know, we're doing a lot of things that were kind of what people would be like.

Oh, there's no way I would work it all, but it very clearly gets a lot of the patterns that you're looking for, where it gets the main one.

Everything singled up into the right.

It also gets that there's all these different colors that belong, and there's I think there are a couple of minutes we're looking at it before the stream that had kind of like a clear colored radiant like a lot of weapons will have like a darker portion of the bottom and then later at the top because metals are often reflective and woods and like plastic.

Something's or not.

Andi, it's kind of amazing.

I think to me that it's even capturing any patterns in a very short period of time.

It's very cool.

Yeah, no, it was It was super awesome.

And, uh, this is motivating me.

And I think the ultimate way we ended up getting to this around about was that the math is super important because I do not know what I need to change about this, to get it to produce the things that I want.

So work it from the top down approach.

I know.

Have a motivation, actually, dig into the internals of this and figure out how all of these pieces end up working together.

Now somebody asked for the linked to the article and to the repo.

So I'm gonna go ahead and do that.

Let me just find this really quick in the meantime, if you want, I'll give you the cable here.

We could switch over to your computer and all that.

You take it away.

That was a big shell on my part so that it's fun.

It's interesting stuff, but, um, let's, uh, let's get to the actual, I guess talk of our heart.

I mean, this is still part of pretty much all under the same set of ideas were trying to take in images, images.

They're tricky there.

There, I think particularly interesting to humans because we can see them.

We look at them all the time, our entire world.

This assumes kind of privileged wise that we can see my policies.

That's not for you.

But generally speaking, there is a sound right?

And that was kind of the other point someone mentioned the chat was like, Have it sample rock songs and generate music is really these are all just different forms of data, right?

An image, a sound wave.

They're all just different kinds of data on dhe.

You can essentially represent them within these matrices in pretty much the same way.

Now, a lot of work has been done on developing kind of these generative models for images that kind of recognizes that in the way that, like sound and sequences of numbers, are kind of linearly dependent, right?

So in a lot of times that make this like Markoff assumption, where any pixel that I'm currently at only depends on the pixel before it.

But that's not necessarily true in an image where the things around it actually kind of shape what's going on.

And you can actually generalize that kind of Marco V an assumption to being like, well, at any pixel here.

Only the pixels that came kind of before it top down, left, right, kind of style mattered.

And so this is still a kind of that same assumption, but it works a little bit better for images where you're actually taking, like all three things before it, instead of just the one I kind of like, kind of like image filtering is like purple exactly where we're kind of just like filtering down diagonally rather than you sample around a pixel to anyone actually trying to figure out what's in there on.

That's kind of a little bit clearer idea.

It works a little bit better for what we know to be true with how images often are.

But even that has its own issues there some pretty intricate complexities with how images form.

And we're not entirely sure of how they work in terms of like, how do I want to model an image for a computer?

Eso image is a really interesting to humans.

S O R.

Sounds and sounds are a little bit less.

I kind of popular.

I guess my nature of people are really interested in.

How do you change images around?

Well, there's a really cool, generative music thing that I saw right, and they're still around like it's not like people haven't researched.

It still exists, actually, finally, that you only do all sorts of cool things.

There is.

You can go a simple as just kind of taking the fundamentals the primitives of music theory and saying, Well, we have X number of notes.

We have X number of rhythms, and how do I re combine those two get things to work?

There's kind of this, like learned, almost natural language processing style, way of doing things where you take a bunch of music that has existed, but you have it's primitive.

So you have, like what it looks like, maybe like a MIDI file or an XML file, and you then try and generate over that.

And then you have this kind of like raw sound wave given a set of amplitude.

Sze, can I sample from that and figure out what's going on from there?

And so there's many different ways of kind of approaching each kind of data, so we're gonna focus a lot of images.

But sound is also super interesting.

There's all different ways to represent different kinds of data, and there's all different kinds of data and like it's an enormous field.

For that reason, it's being used in all sorts of things can from like genetic see sorry, genetics and sequences of DNA and genetic material to like protein constructions.

That's all very bio heavy all the way through to light.

How do I predict how, like a city's traffic mapple look, what is the city heat map look like from the top down?