Hello.

And welcome to a coding challenge, Generating StyleGAN

Rainbows With Runway ML and passing those to P5 JS

to display them in the browser for our delight.

And so this is basically making a version of software

that I built for the I/O Festival.

The talk that I gave at the I/O Festival

will be out on the internet at some point soon.

And if it is, I will include a link

to it in this video's description

where I had people play a game on stage.

And when they finished the game, an AI rainbow

was generated and tweeted to the I/O Rainbow's on Twitter

account.

So I'm going to build a little mini version of this today.

And so the primary tool that's going

to do the generating of the rainbows

is something called Runway ML.

So this is the first coding challenge

that I'm using with Runway.

And I've talked in a couple other videos

in a livestream more about what Runway

is, how to download and install it,

how to sign up for an account, how

to get some free credits, and that sort of thing.

So I'll refer you to that video if you want to find out more.

But if you have Runway downloaded--

find a link to download it in the description.

If you've signed up for an account,

and you're on the Browse Models page,

you will find yourself here.

And where you want to go next is you want to look for StyleGAN.

Now, I see StyleGAN right here.

But just in case you don't, I could type it in up here

under Browse Models.

I could click here.

This is now giving me, "Generate photorealistic images

of faces, landscape, and more."

There's more information about the license for StyleGAN,

credits about who created and authored the original StyleGAN

paper that you can find out about.

But I just want to use StyleGAN.

So I'm going to click Add to Workspace.

And if you don't have a workspace already,

you can click New Workspace.

I have one called Coding Train Livestream.

And then here I am.

Now, I can generate a variety of types of things with StyleGAN.

And these are known as checkpoints.

So there are checkpoints, you can see,

for cars, and landscapes, and portrait.

Wouldn't it be nice if there was a checkpoint for rainbows?

And guess what.

There is.

So I'm going to click the rainbows checkpoint.

Then I'm going to choose the input source, which

is just going to be a vector.

And I'll talk about what that is in a little bit

as I get to writing some code to do this.

And then what I want to do is click Run Remotely.

So this is very important for me to be clear.

This requires cloud credits that you have to pay for.

If you sign up through the link in this video's description,

you'll get free $10 in credits.

And there's actually a coupon code, CODINGTRAIN,

which you can get an additional $10 in credits as well.

So that's certainly enough to run this example.

Some models in Runway you can run locally on your computer

without using cloud credits.

But this one in particular only runs remotely at the moment.

So I'm going to click Run Remotely.

It's running.

So I'm going to give it some time to start up.

So now we see it's starting to populate.

And this is what's known up here as the latent

space, this sort of space of imaginary rainbows

that this generative model is producing.

And this is one of the reasons why

I love using Runway is that I can actually just kind

of browse around this space, and kind of

have this 2D view of this multi-dimensional world of all

of these rainbows.

And I can do things like, oh, I really like this one.

And I could just change the output here to preview.

And so I can see it here.

And I could download this one.

And I now have my beautiful StyleGAN generated rainbows.

But what if I want to have these rainbows in my own software?

If I wanted to show them on a web page?

Or if I want to tweet them from my Twitter bot?

Or any other types of thing that you

might be making, whether it's JavaScript, processing,

open frameworks, or some other piece of software

that you want to connect to Runway.

The way that you do that is through talking to Runway

over the network.

So over here in the bottom--

I'm sorry.

In the top right, there is a Network tab.

If I click on this, it's showing me

a variety of different options.

I can communicate with Runway over OSC.

This is something I did in another video tutorial,

communicating with processing and Runway over OSC

to demonstrate the PoseNet model for detecting

human skeletal poses.

I could also use Socket I/O for real time web sockets.

But really what I want to do is just an HTTP connection.

I want to make an HTTP request.

I want to post some data to Runway.

And I want to receive the image back.

And, in fact, there's JavaScript code right here out of the box

that I could copy and paste.

So I encourage you to just actually

go and grab this JavaScript code,

and make your own example.

But I am going to do this using the built in P5

function, HTTP Post.

So I'm going to write my own code

for doing this, referencing everything that's

here under the HTTP option.

My next step then is to go to the browser.

And I am going to write this code in the P5 web editor.

So first thing that I might do is

let's make sure this is running, if this works.

OK.

Great.

Let's make a button.

Create Button.

And I'm going to call that button rainbow.

And I'm going to attach that button to an event called--

whoops.

Let me hit stop here for a second--

Generate Rainbow.

So when I run it, a button will appear.

Rainbow.

And then presumably when I click the button,

a function called Generate Rainbow will be executed.

So I need to write that function.

And in that function, this is where

I want to send my request to Runway itself.

So I need to make an HTTP post.

So I probably want to look at the HTTP post.

This is a P5 specific function.

You could use fetch.

I have some video tutorials about how

to use the fetch function to make a post request.

And you could do that here.

But I'm going to use P5 in this example.

So I'm going to go to the HTTP post reference page on P5.

And this is going to show me the stuff that I need

to include in the post request.

So here's the stuff that I need.

I need a path.

Where am I posting this to?

I need a data type, which is what kind of data

am I sending along with this post request?

And then any parameters, data that

needs to go along with the post request, as well

as the callback and the error callback for getting

information back.

So if I could just take this and bring this into my code,

I'm just going to put this in the comments

right here as a reminder.

So what are the things?

The path is--

Runway's telling me this.

It's actually this.

The server address.

Local host port 8000.

So I'm going to paste that in here.

The data type is JSON.

That's the kind of data.

The data I'm sending is what?

So this is the data I need to send to Runway.

And Runway's telling me about that here.

Input specification.

What does it expect?

It expects a field called Z, which

is an array of 512 floats.

What is that?

And then it requires some other value called truncation.

What is that?

So if you wanted to dive deeply into this input specification,

you would probably want to do some more research

on the StyleGAN model itself.

Look at the paper.

Look at the GitHub repo.

And kind of understand more about the neural network's

architecture, and its parameters,

its hyper parameters that control its behavior.

I think it's worth, though, for a moment taking

a minute in this video tutorial to talk about what

this z is, cause it's a very important concept in machine

learning.

So there is this machine learning model called StyleGAN.

And it needs some kind of input in order

to generate some kind of output.

Now, the output that it generates is an image, 512x512.

I mean, ultimately what it's outputting

is just a whole lot of numbers.

But those numbers can be interpreted as colors of pixels

and repackaged in the image.

So that's happening for you by Runway.

Right?

We're seeing the output of it right here,

packaged as an image.

But what's the input?

I mean, ultimately, in this particular example,

I don't care about the input.

I just want, give me a rainbow.

Give me a rainbow.

Give me a rainbow.

But in order for the model to generate a rainbow,

it's got to start from somewhere.

And in essence, I could start with something random.

But what that random thing that I want to start with

is is something called a vector, referred to as z.

And what it is is 512 numbers.

So I have this list of 512 numbers,

probably between 0 and 1.

So generally inputs to neural networks

are normalized with some range.

And in a way, this is like a unique signature

for a particular output.

So if I want to just get any so-called output,

I can just make up a list of random numbers.

And I would always get the same exact rainbow

with the same set of numbers.

So we could see that happen.

Right?

If I fix that set of numbers, I'll

always get the same output.

But what I could do is tweak these numbers a little bit,

dial some up, dial some down.

And that's going to change the output.

And that's what you're seeing here in this space.

What you're seeing here is rainbows that

are attached to given z inputs.

And Runway is being very clever about showing you

similar ones in a two dimensional flat space

on a computer screen.

But actually, all of those rainbows that are generated

live in 512 dimensional space.

So that's kind of crazy, and mind blowing,

and very confusing.

I think I have a video tutorial where I do something

with four dimensional space, and I can barely understand that.

But this is kind of the weirdness

of working in machine learning is

you could imagine a three dimensional space would just

be full of rainbows in 3D.

Like all over this room, there'd be rainbows everywhere.

2D.