Yeah.

So, um, my favorite type of programming is attempting things that I've never done before, which is an extra reason why I shouldn't be a fulltime software engineer.

Because I love the whole process of failing and learning from it, which I think is really important.

Especially admitting that in front of hundreds of people, Um, so as the artist residency, um, I did a series where, um, of artworks that were called New romance and I took Ah, play it.

Well, it was a play on Neuromancer by William Gibson, Obviously, but, um, the reason that I did this project was because I wanted to blend special effects makeup in technology together to better, um, visualize what modern cybernetic augmentation could look like.

And I did this in the sense that this is digitalized makeup.

And when I had the show to show people this, they were like, Well, that's really cool, but it's also weird, and I don't want to go near it, but they also didn't know how I did it.

So I got to talk to them about no bots, because this is just running on an Arduino, and it's a simple circuit that I started together.

And, um, the face is Angelina Fabbro if it's making you oddly like familiar with it, but you can't put a finger on it.

Um, yeah.

So weird Art and technology, stuff like that.

And so that leads me to what we're talking about today.

Um, I do visuals under the name M s paint.

Miss Page, Um, this was a show that I just played in January at a festival called Mag Fast.

It has 30,000 attendees and, like was, surprisingly, not as scary as top conferences.

Yeah, um, when I started doing visuals, I was using tools that other people had made already.

And I was like, Why not make this harder on myself and build my own tools?

So I do a lot of these trip tune shows, and I wanted something toe work for myself.

The thing that I had been working with before was called Lumen.

And excuse me, I was using this for a while.

It's only available for Mac.

What it is is it's an old school type of video synthesizer that has, like, patch board stuff that you packed around, and it was really cool, but, um It was very specific, and I'm gonna show you what it looks like.

So it has a bunch of these patches so that you can pick different kinds of emulation Sze of however they're altering the waves and you can change it in here and you could hook it up Thio a midi input and David And then you can get really wild back here.

I don't even know.

I read.

I read the instructions a little bit, but I gave up because I was like this.

That's just way too much.

And so I was like, I got it.

I got to do something myself great and then sear like, Why?

Why would you do that yourself?

Um, so I already was doing visuals, so it seemed like a valid time and effort output to, you know, do it myself.

Ana and I also love challenging myself.

Like I said, I love trying things that I've never done before.

So, um, my process going through when I make things is probably similar to like how a lot of other people put things through.

Um I think of it separately.

I'm like, Okay, cool.

So I have this.

I had an idea of how I wanted to control that input because I wanted to look cooler than just like standing up on stage with a laptop and pushing buttons.

I was like, I want to look even cooler This this is called a midi fighter.

It is, ah, midi input through us be.

There's a bunch of buttons.

There's a potential ometer inside There's buttons on the side and I was like, I was way cooler up onstage with one of these things in just a laptop.

So I have my buttons and I know I wanted to do something in the browser so that I could build that.

And then I was like, All right, so how am I gonna do this?

And, um, that I had figured out how to put them together the hard way?

Essentially, and that was, you know, learning a whole new A P I.

So maybe we all know about I'm to give you a little bit of a history of the protocol.

And so it's an acronym that stands for musical instrument digital interface, and it's a way to connect devices that make it, uh, making control sound.

So some people think that like when I talk about Mickey, I'm gonna be talking about the sound that gets out.

But we're just talking about the messages that communicate with it.

I'm so it's a protocol that allows Elektronik instruments and synthesizers to communicate, meaning a user control multiple electronic instruments from one synthesizer or a computer.

There used to be no coherent way for manufacturers to do this.

But then in 1981 at the Audio Engineering Society Convention in New York, Dave Smith proposed a rough version of a universal digital interface.

But he called the universal synthesizer into interface or us.

I, um, and he was like gonna go out and meet all these people and not that many people came.

So, you know, he was basically like, Here's my project and I want other people to contribute to it, And everybody was like, I'm not gonna do that.

It's always been a problem.

Apparently, Um, except somebody that did show up was Qatar Okaka Hashi, who was the founder of Roland at the time.

And so the following year they displayed the fruits of their labor, and it was a demonstration on how many worked on how it was born, and then they want a technical Grammy for it in 2013 which is cool.

And yeah, So how does his MIDI work specifically?

And the most important thing to know about MIDI is that it is based on the idea of message passing between devices.

So, like pieces of equipment or software.

And so you have a keyboard and a synthesizer, and you would like to record a sequence using the sounds that are in the synthesizer.

And so you connect them so that they could communicate using that MIDI protocol and start record.

And when that happens, you play notes on the synthesizer, and all of the physical actions are transmitted as many messages to the computer sequencing software, which records the messages and so many messages or brief numeric descriptions of the action, which I'm gonna show you with the Web.

Nitti AP I When I push the buttons on the so that could be key presses, joystick wiggles, knob turns, they're all encoded and know you are interpreting what those messages mean.

So when you hear the sound that you're making, it's not because of theme idiots because of the messaging that the Midi is telling the synthesizer together.

Basically, the computer isn't making the earth.

Isn't recording the sound itself?

Um, yeah.

Sorry.

Why is my slide Dacula?

Sorry about that.

Oh, So the concept of channels essential to how most meeting messaging works.

And that's an independent path over which messages travel to their destination.

There's 16 channels.

Permit e device on a track on your sequence of program plays one instrument over a single channel when dealing with the musical application.

Maybe messages in their track find their way to the instrument that chill.

Like I said, we're not dealing with music today.

We're dealing with dignity, and the middIe tipple typically enable synthesizers, drum machines, you know, like the music stuff that we talked about.

But since it doesn't transmit audio, if you wanted to make a drum machine using the web giddy AP, I would just use audio attack simple so you could build your own drum machine if you like.

The other kind of thing that you can use it for is live event controls for lighting and special effects.

Really, it's like you can do whatever you want.

So the weatherman e a p I specs supports maybe protocol by allowing Web APS to enumerated select input and output devices on the client system and suddenly received many messages on.

But you're able to utilize non music applications as well as music ones by providing low level access to the media devices on the user systems and the women.

The AP I give you an output that tells you what kind of device is connected.

Ah, and you could do so much with it.

It's really, really wild.

Um, I've said this a lot.

I'm gonna skip over it, use the audio tag if you're wanting to make music.

Um, basically, Web midi is really enabling a new class of application that can respond to Midian.

Put using external hardware.

So, um, use it along with other AP eyes and from 43 up and you're ready to go.

Okay, Well, so I'm gonna show you some demos now of how it works.

All right?

No.

Also ignore the deprecation messages in the console.

All right, so where is my screen?

All right.

So the way that you use media's, you request many access its built in chrome.

So that's why I'm able to not have anything else in here?

Um, yeah, It request access to the middIe device, and this is how you can see if the browser supports it.

If it doesn't, it'll tell you it does not.

And then this is gonna out.

Put the the object that I was telling you about before that talks to you about biddy inputs in the middie outputs.

Because not only is a button press, it's the press.

The hold on the release, depending on the middie device.

So you have that outputs all that messaging and then this one also the one bad thing about it is since it's a potentially ometer like every single movement is just gonna keep on giving me messages to you.

So let me clear this out and let's open one of these up.

All right, so in here, you can see that my MIDI device is a mini fighter, but the Ellis one of them Wait.

Here we go.

So this is a midi fighter, Freedy.

It is made by D j Tech tools.

Uh, this is not a sponsor And talk.

I just shows their device because I like it on DDE.

Then when you go in a little bit further.

There's a data Doug right here, and it gives you an array that identifies a lot of different things.

It's based off of the speck that you get for the MIDI controller, but for for me at least, I know that the index zero item in all of the data is contributing to a specific button that is on the device.

So that's how I'm able to control the work that I'm going to show you, Um, in a second.

Then it also allows you to do fun things like, Hello, So that's what I mean.

Like, you don't have to do music.

You don't have to dio like games.

You can even just use it to control simple elements on a Web page.

All right, when you get back to keynote.

So what I built, I decided to call a video.

Um, I originally was going to do like Shader is.

I was like, that would have been really cool, But shooters are hard.

So there's this really awesome library called pixel since made by Donald Hanson.

And it's, um, it essentially did exactly what I was looking for, so I asked him if I could use it and adapt it to working with the MIDI controller.

And he said yes, which is a nice thing to ask.

Um, yeah.

So I'm gonna show you how that works.

Um, here is another example of what the middIe fighter is.

So it's full color configurable, arcade buttons, total motion control on three accesses.

It's it's really cool.

People use it for drumming like this, but I used it for a video.

I do want to give a warning in case anybody instead of the flashing light.

Um, if you have that issue, let me know, and I will give you a second toe leave.

Does anyone have that two?

Don't.

Okay, cool.

So now I'm gonna give you a demo of how that works.

All right, So this is it.

It is all in the browser and see that when you use the potentially ometer it effects on one access the rotation and on the other one, it's scale.

So it's really great for live music because even if I'm not familiar with the artist that I'm performing for, I can do it on the fly.

If I'm on beaten, at least and then the other other functions that are programmed into this is ugly color set.

Let me switch it to something cuter.

I have a randomizer on the side that allows me to keep on going through until I can get a setup that I like.

And then from there, I'm also able to change the width of the waves so that it's very it's kind of hard to see you, but it allows for quick changes on the fly, which is really cool.

And I'll show you how the code looks for that.

It is mess.

But, um, you know, at the beginning, we're requesting that media access.

Like I was saying, I'm gonna make this bigger.

Also, it don't get hub If you wanna dr it for me or, you know, perform yourself command what p be Thank you.

See, it's been so long.

I don't even write code on my job anymore.

VP E Yeah.

Thank you.

Think.

All right.

So I'm not gonna go over all of the synthesizer code because it's essentially just a lot of our g b A waves.

Um, really.

Midi is the star here, and everything in here is commented for Ah, clarity and yeah, so I have a lot of that output values for the different buttons.

I have, um, an output for controlling the town.

She ometer and the accelerometer are just potential ometer because we need it around that too.

A more median number.

Otherwise it would have been all over the place and very sensitive.

So we are controlling it through these helper functions, which is just like averaging out the frequency of the, uh, Betty, please.

This is all of the synthesizer stuff, which is cool.

Mm.

Here is what we did with the changes.

So the scale on the rotation is getting that input from geometry.

And then I'm handling the messaging and honestly, like just random izing every it works, it allows you to, like check for the specific buttons for changing the waves that allows us to track for the potentially ometer and then, um, defining through the scale in rotation.

So it's almost all the way there.

The only issue is like, since it is a canvas animation.