字幕列表 影片播放 列印英文字幕 MALE SPEAKER: Welcome. It's my pleasure to introduce Gil Weinberg. He's a professor at Georgia Tech for the Center of Music and Technology. And I went to Georgia Tech a long, long time ago and we didn't have a music department back then. So when I visited recently, the folks I was meeting with in ECE sent me over to meet Gil and he's doing some really cool stuff with interactive computing and musicianship. So building machines that you can interact with, not just from a dialogue perspective, like the kind of thing I'm thinking about, but more from a music perspective where you want to jam and do some jazz improvisation with a robot. And he's going to talk about that in many other things today. So welcome. GIL WEINBERG: Thank you. [APPLAUSE] Thank you. Thank you for coming. So I'm going to talk about three or four main projects. Only three of them I showed in these slides. The last one is a surprise. We did the project which we just finished and I have some very, very fresh new slides. The reason I started to be interested in robotic musicianship is that I'm a musician before I became interested in computation or in robotics, I was a musician and still am. And I was always fascinated by playing in a group. By being constantly ready to change what I'm doing. By trying to build on what other people are playing in real time. Improvising in a group, a visual cues, auditory cues, of course. So when I started to be interested in robotics, I wanted to capture this experience. Before I show you some of the efforts that I did, maybe I'll show a short clip of me playing with a trumpet player and trying to see the kind of experiences that I was trying to recreate. [VIDEO PLAYBACK] [JAZZ MUSIC] [END PLAYBACK] So I think you've seen a lot of eye contact, trying to build a motif that I'm not sure what is going to be and trying to create something interesting with them back and forth. And the first robots that I tried to develop it was building on these ideas. But what I wanted to do is to have the robot understand music like humans do. The big idea that I started with, was to create robots that listen like humans, but improvise like machines. Because I felt that if I want to push what music is about through new, novel improvisation algorithms and new acoustic playing, I first have to have connection between humans and robots, and that's why there is the listening like human part. Only then would I be able to start to make the robot play like a machine in order to create this kind of connection and relationship. I'll start with something very simple that probably many of you are familiar with. If I want a robot to understand me, maybe the first simple thing that I can make him do is understand the beat of the music that I play. And here we use auto-correlation and self-similarity algorithms. This is a piece from Bach. You see the time is both on the x and on the y. And you see that it's symmetric and by comparing the [INAUDIBLE] to the algorithm you can try to find sections that are similar and detect the beat from that. But what we try to do is to actually have it in real time. And you see here my student Scott Driscoll used this algorithm based on Davies and Plumbley from Queen Mary. And you see that's it becomes much more sophisticated because it's not just analyzing the beat in Bach or in the Beatles-- Scott is playing in real-time so he is trying to get the beat, but then with Haile, start to play with it. Scott is trying to fit what he is doing to what Haile-- Haile is a robot-- and back and forth. So it's a little more complicated you see. Sometimes they escape from the beat and get back to the beat. And here's a short example. Got the beat. Now Scott will start faster and slower. Faster , got it. So as you can see, it loses it, it gets it back. I think in a second, it will play slower which shows how the system fails. The next thing that I was trying to get is to have the robot understand other things at a high level musically. Not just the beat but concepts that we humans understand such as stability and similarity. And basically we had a huge database of rhythm generated almost randomly-- with some rules, some stochastic rules. And then we had a coefficient for stability and similarity. Whenever Haile listened to a particular beat there's some settings and coefficient for stability and similarities. At some point the robot actually decided by itself. First the human on the side can change the similarity and stability and create some output to bring rhythm back. And this similarity is based on Tanguiane from 1993, basically looking at the overlapping onset between beats. I can get more into this if you want, maybe later. And this is as similarity algorithm based on Desain and Honing. [INAUDIBLE] between adjunct intervals is what set how stable the rhythm is. This is based on music perception studies that I've been doing. And basically, there is a mathematical procedure where you compare each one of the notes to the note that comes after it. And at some point, after giving preference to one and two, which are stable, you can get for every particular rhythem-- for example, this one, the quarter quarter, two-eighths quarter-- a particular onset stability by combining all of the ratios and getting something stable. And here's a short example of Scott playing with Haile and Haile trying to understand the stability of Scott's rhythms. And then based on a curve of similarity, starting most similar then going less similar. And basically a human put a curve. But I can see a scenario where Haile could come up with a curve by itself, trying to start with something that Scott understands. Scott is playing seven quarters, and then slowly introduce new ideas. And you can see how Scott actually listens to the robot and at some point, building on what the robot is doing. So Haile is building on what Scott is doing, obviously, by looking at the stability and similarity. But at some point, Scott is almost inspired. Maybe inspired is too big of a word, but that's a goal that Scott will come up with an idea that he would come up with if he played with humans. [VIDEO PLAYBACK] [DRUMMING] That's almost a new idea. Scott is building on it. [END PLAYBACK] And this is a darbuka drum player concert. You will see how the professional darbuka player, actually from Israel, is kind of surprised. But I think his facial gestures were interesting for me because I think he was surprised for the better. And at some point you'll see how all of us are playing and Haile tries to get to beat. So we combine the stability similarity and beat detection into a drum circle. [VIDEO PLAYBACK] [DRUMMING] This is call and response. Later it will be simultaneously. [DRUMMING] And now, what it does, it listened to these two drummers, and tricked the pitch from one drummer, and the rhythm from another. And the other arm, most from pitch and the timbre of the two players. [END PLAYBACK] So we played the rhythm that one player played, and the pitch-- well, it's not really pitch, it's in the drum, but this is lower and this is higher next to the rim, and tried to create something that is really morphing between these two. Again, things that humans cannot do and maybe shouldn't do, but here something interesting can come up. Another thing that I was interested in is polyrhythm. It's very easy for a robot to do things that humans cannot. Sometimes I'll ask my student to clap. I will not ask you to clap. I'll give you an example. I think there is two main reasons here. [SPEAKING RHYTHM] This is nine. [SPEAKING RHYTHM] I don't ask you to clap but sometimes I would. It was [SPEAKING RHYTHM] seven. And then I asked my students to do the nine in one hand and the seven in another hand which I will definitely not ask you. But see how Haile here captured-- decided to record the rhythm. So it records the nine. He choose the nine and the seven, and at some point he introduced them in polyrhythmic, interesting rhythms. [VIDEO PLAYBACK] [DRUMMING] And we add more and more rhythms over it. [END PLAYBACK] And I don't know if know, but Pat Metheny had a project that he used robots in. He came to our lab and I explained it to him, I showed it to him. I said, this is something no humans can do. And he said, sure, my drummers can do it. And he can also do the four with his leg, and another three with another leg. So I think everyone can do it except maybe Pat Metheny's drummer. And here is something that's at the end of concert, it's obviously, you know, fishing for cheers. We just have a little short MIDI file where we playing with it together. Unison always works, so I'll just play this. We grabbed the nine and seven. [VIDEO PLAYBACK] [DRUMMING] And that's from a performance in Odense in Denmark. They had a robot festival. [END PLAYBACK] So the next project was Shimon. And actually I have an story about this because I put a video of Haile and Scott playing, the first one, on my website. It was before YouTube, or before I knew about YouTube. And someone grabbed it and put it on YouTube, and then CNN saw it and they asked to come and do a piece. And when they put a piece, the next day I got an email from the NSF, from the NSF director who said we saw your piece, please submit a proposal and continue to develop that. Rarely happens. Never happened since. I tried. I put so many videos-- And this is the next robot that we came up which adds multiple things. The main thing that it adds is the concept of pitch. It plays a marimba. And the second aspect it adds-- we're talking about the personal connection, gestures, visual cues-- is the head. And many people ask me, why the head? It doesn't play any music. In a second I'll show some of the utilization of what the head does. And of course it has a camera. That's the first question you get about any head, robotic head. So the name is Shimon, and here is the first example of my post-doc, Guy Hoffman. He's now a professor in Israel. [VIDEO PLAYBACK] [PLAYING PIANO] It shows how a headless Shimon responds to dynamics, to expression, and to improvise. Look at Guy, how he looks at him as if his kid did something nice. [END PLAYBACK] [LAUGHTER] So when Guy played fast, the robot responded. Simple, just tempo detection. And then loud and louder, and tried to get some understanding of tonality and of giving the right codes. And we had some studies done about it. If you remember the first code was [SPEAKING CODE] Three codes. And we tried to see if, by looking at the robot, at the gestures, with the arms before the head, if the synchronization between the human and the robot is better. And you see that the first code is maybe easier. The second code, very difficult to synchronize in terms of milliseconds. The third code, you kind of adjust yourself. So it's a little better. And we have multiple conditions, one only visual, one audio, and one with synthesized sound. And you can see that the visual is better because it's less milliseconds of delay between the human and the robot in both of the conditions, precise and imprecise. Especially in imprecise, you'll see it's better. For imprecise, the robot could play 50 milliseconds, before or after. And just looking at the robot helped it become a much more synchronized, tight performance. And it works much better at slower tempos, less than 100 BPM as in more than that. Then we added some other aspects. We said, OK, we have a head, let's do what the head can do. Actually we didn't think of this way. We thought of what the head can do and then built the head, but for the story. And what it can do is multi-modal detection. We had a camera there and we tried to detect the beats of the hip hop artist to coordinate with the beat detection of the music. And you see how the multi-modal detection work. And also just to help them feel together-- and this is Guy Hoffman's work-- and develop gestures that will make everyone feel that they are synchronized with each other. [VIDEO PLAYBACK] [MUSIC PLAYING] So that's just the audio. [MUSIC PLAYING] Now double-checking with the beat of the head. And of course blinking never hurts. The blinking is random. [END PLAYBACK] All right. OK. So that was a little bit about listening like a human and maybe grooving like a human. And then I started to think more about creativity, or machine creativity and what it means. And actually, this is the first time I've showed it. I hope it's not streamed, I think it's not streamed. This is a TED Ed video that I just finished doing. I didn't put in online yet, so it's a part of it. I didn't show it yet. But it shows one of the ideas I had for creating machine creativity using genetic algorithms. And the idea was that if have a population of motifs, maybe random motifs, and then I have a set of mutations and cross-breeding between these motifs, and have a fitness function that is based on human's melody, then I can create new, interesting music that has some aesthetics for humans in them. Because it's easy to create melodies that were never created before. Most of them will not be listenable. The question is, how do you make meaningful and valuable melodies? And in the video, you'll see a little reference to the Lovelace Test, which probably some of you know about maybe I'll just say that the Lovelace Test is based on Ad Lovelace, the first programmer, or is considered to be the first programmer, trying to-- instead of intelligence like the Turing test, try to determine when the robot is creative. And she said something like, a robot will never be creative before it generates ideas that its own programmers cannot understand how they came to be. And then in 2001 there was a group of scientists, some of them continued to develop [INAUDIBLE] that came up with this Lovelace test, which is mostly a theoretical thought experiment of creating something like that. But even if we do something like that, I didn't think that the music that would come be valuable. And I'll show this little TED Ed clip and see how I went about it. [VIDEO PLAYBACK] -So how can evolution make a machine musically creative? Well, instead of organisms, can start with an initial population of musical phrases. [MUSIC PLAYING] GIL WEINBERG: They put the drum beat. It's not mine. -And a basic algorithm that mimics reproduction and random mutations by switching some parts, combining others, and replacing random notes. Now that we have a new generation of phrases, we can apply selection using an operation called a fitness function. Just as biological fitness is determined by external, environmental pressures, our fitness function can be determined by an external melody, chosen by human musicians or music fans to represent the ultimate beautiful melody. [MUSIC PLAYING] GIL WEINBERG: Chopin. -The algorithm can then compare between our musical phrases and that beautiful melody, and select only the phrases that are most similar to it. Once the least similar sequences are weeded out, the algorithm can re-apply mutation and recombination to what's left, select the most similar or fitted ones, again from the new generation and repeat for many generations. [MUSIC PLAYING] The process that got us there has so much randomness and complexity built in, that the result might pass the Lovelace Test. More importantly, thanks to the presence of human aesthetic in the process, we'll theoretically generate melodies we would consider beautiful. GIL WEINBERG: All right. [END PLAYBACK] So here's an example of how this is used in real-time. Chopin is not [INAUDIBLE] with fitness function as you see here. But actually, the saxophone player and myself playing piano-- I programmed Haile with multiple motifs and every time it listened to one of our musical motifs, it used dynamic time warping to see what's most similar and come up with a variation. And it stopped after something like 50 generations because, of course, if it ran more than 50 it would become exactly what we played. So we just stopped one time to create something that is similar to what we did, but a little different and based on the, let's call it, original DNA of the population that we started with. And [INAUDIBLE] Haile will never use it with Shimon. This is Haile when we first tried to have Haile with pitch. So the whole idea of creating rich, acoustic sound is not there. We put the little toy vibraphone below Haile. So excuse the quality of the sound. [VIDEO PLAYBACK] [MUSIC PLAYING] And listen to how all three of us are building on each other's ideas. Saxophone, piano, and robot. [MUSIC PLAYING] [END PLAYBACK] Yeah. It's difficult to evaluate it because of quality of sound. But that was the idea of trying to put it in a real-time scenario. Another idea where I tried to have it play more like a machine was style morphing. And as you can understand by now, I play jazz, I like jazz, I like Monk on the left and Coltrane on the right. And I thought, wouldn't it be awesome if we tried to morph between Coltrane and Monk and create something that they themselves will never be able to create. They cannot morph into each other. And maybe something interesting will come up that is just a combination of both their styles. And we used HMMs here, basically analyzing a lot of improvisation by both of these musicians and trying to see what the likelihood is that at any given note, what the next note would be, what the next rhythm would be, just a second order. And then with an iPhone I can play a little melody. [VIDEO PLAYBACK] And this is a little app that we built regardless. And at some point when the melodies that I play, of course it can be piano too, is ready, I will send it to Shimon, and then I will be able to morph between my own melody, Monk's style, Coltrane's style and with two styles with saw a different kind of balances between these two improvisations and myself. [MUSIC PLAYING] If you know Monk's syncopation in clusters. If you know Coltrane, you know the crazy, fast licks. And this is somewhere in between. And of course, one doesn't play saxophone over piano, so it doesn't sound like any of them. [END PLAYBACK] But at least the notes are based on the improvisation. Some other quick examples of things that we tried. Here we had the connect because we wanted to get more than just what this simple camera in the head had. And we tried to have motion-based improvisation. You see the Atlanta symphony percussionist, Thomas Sherwood, doing some tai chi-like movement and trying to have Shimon improvise. And actually Tom is playing an electronic marimba and Shimon has a system in which sometimes it will repeat what is listened to. Sometimes it's [INAUDIBLE], sometimes it will introduce new ideas, a stochastic system that's supposed to inspire Tom to play differently than he would otherwise. [VIDEO PLAYBACK] So that's the tai chi part. [MUSIC PLAYING] And it wasn't easy for it to repeat, exactly this pattern. It's very limited with the four arms and where they can be. I'll show some path planning strategies later. So not every motif was captured perfectly. And now it starts to introduce new motifs. It will repeat and then create some kind of a musical interaction. [END PLAYBACK] This is my student, Ryan Nikolaidis, the entire motif. [VIDEO PLAYBACK] And Shimon is developing this motif. This is part of the US Science and Technology Fair in Washington. [MUSIC PLAYING] [END PLAYBACK] And for something else, some of them just, you know, they love dubsteps. And the machine learning maybe interesting or not, but dubsteps are there. So it's just a dubstep piece that one of our students developed for Shimon. [VIDEO PLAYBACK] [MUSIC PLAYING] So it can do all kinds of stuff. [END PLAYBACK] Here we used vision-- and you see this green ball on the conductor's baton-- and tried to combine the input from conductor and from the audio analysis from the orchestra to improvise, based on the chords that the orchestra is playing. [VIDEO PLAYBACK] That's the Georgia Tech student orchestra. [MUSIC PLAYING] [END PLAYBACK] All right. And I think this is my favorite clip. This actually was a PSA of Georgia Tech for three or four years. You know that we come from Georgia where dueling banjos are a big thing, at least in some movies. So we tried to do dueling marine buzz if you like, and present kind of the dueling aspect and Shimon with the gestures-- and this is again, Guy Hoffman's work-- project some personality. Hopefully better than the movie. [VIDEO PLAYBACK] [CALL AND RESPONSE MUSIC PLAYING] [END PLAYBACK] I think I have two more clips before we go to the next project. I just got the request, how about having a robot band? They listen to each other and create something. I'm sure robotics people here are being asked about, will robots replace humans often? And in my case it's really painful sometimes because they come to me and tell me, how dare you? Music, really? Like, OK cars, OK cleaning carpets, but you're going to take human from us and let the robots play it. So I never tried, I'm not interested in having robots playing together by themselves. It's only and always have been about inspiring human and creating something interesting for humans. And here you can see a quick clip where you see bit detection and [INAUDIBLE] improvisation between two students and two robots. [VIDEO PLAYBACK] [MUSIC PLAYING] He's not so good with the beat, by the way, so-- strictly for Haile to detect it. But it got it. And now it start to improvise with the other arm. What you hear here is a student that we don't see, playing the synthesizer. [MUSIC PLAYING] And now Shimon, before we had a head, actually we used someone else's CGI head. Andrea Tomaz from Georgia Tech. I'm dropping back and forth in time here. [END PLAYBACK] All right. So that's a robot-robot interaction for you. I promised to talk a little bit about path playing. There are multiple ways in which these forums can be next to the piano or next to the marimba. In this case I would just put a piano. And we have state machines that represent each one off these states. Given any melody, what would be the most reasonable path to take? And the inputs that we have is a curve that humans play for tension-- melodic tension or harmonic tension, or rhythmic stability. And I'll show it here. This is some the pitch or note density. And using the Viterbi algorithm, actually [INAUDIBLE] Viterbi, we're trying reduce the space and try to come up a reasonable path to create improvisation, following these curves. So it's basically developing, as I say, an embodied musical mind, where not only of the music it has to play but also it's own body. The whole idea of embodiment is a bit part that robot musicians have and that software musicians or machine musicians don't have. Only robotic musicianship has. So we are trying to come up with these algorithms, and actually, my student, Mason Bretan programmed really extensive jazz theory into Shimon. So now it can listen to chords that Mason, in a second will play, and then try to improvise, using this both music and physical limitations that he has using Viterbi. He tried some other, but the Viterbi worked pretty well and this is his improvisation. This clip actually went viral kind of for our fill. There were around 50k viewers last month. [VIDEO PLAYBACK] [MUSIC PLAYING] Mason fed all the chords offline. All the calculation of what's going to be play was done offline. [MUSIC PLAYING] [END PLAYBACK] So I don't know if it's got [INAUDIBLE] quality but it's not bad. I was happy. All right, so we all think it's very important, the path planning and creativity and so on. But it's important to realize that this can be taken in a different light. And I think if you're being made fun of for what you do, this guy is probably the best guy to make fun of you. Some people tell me, don't show it, but I think-- I'm proud of being made fun of by Stephen Colbert. [VIDEO PLAYBACK] -Jazz robots, which combines two of the biggest threats to our nation-- jazz and robots. He's been programmed to dig it too. [LAUGHTER] Bobbing his head to the beat and checking in with this fellow players. Meaning that long after the robot uprising kills all humans on earth, there will still be someone on the planet pretending to enjoy jazz. [LAUGHTER] [END PLAYBACK] OK. Shimi was an effort to take Shimon and make it a little smaller. Actually it almost became a product. There was a failed Kickstarter there. And it focused on the gestures, not so much from the acoustic reproduction of sound, which is part of the reason that I went into robotics because I kind of grew tired by sound from speakers, no matter how great your hifi system is. I was missing the acoustic sound which is one of the reasons I went to robotics. So here we don't have acoustic sound, but we have a bunch of other applications. We call it a robot companion that can understand music. It can connect to your Spotify. It can dance, it can do the bit detection. It has a set of algorithms to get data from the audio and respond accordingly. And here's a clip. [VIDEO PLAYBACK] [MUSIC COLDPLAY, "VIVA LA VIDA"] GIL WEINBERG: In this case it takes the music from the phone, but we also have an application that takes it from Spotify. And the camera and the phone can see if you don't like it. And better to say no, because the music is loud. [TAPPING "DROP IT LIKE IT'S HOT"] [MUSIC SNOOP DOGG, "DROP IT LIKE IT'S HOT"] GIL WEINBERG: [INAUDIBLE] speaker dock. You can see the dancing speaker dock. [MUSIC SNOOP DOGG, "DROP IT LIKE IT'S HOT"] [END PLAYBACK] Here is a couple of other applications, some using speech, and some using audio analysis. That stuff is audio analysis. Looking for a beat in what Mason is playing. [VIDEO PLAYING] [PLAYING GUITAR] GIL WEINBERG: No, there wasn't anything there. [PLAYING GUITAR] You get it. It has a system of gestures. [END PLAYBACK] GIL WEINBERG: And here, using Google Voice. Mason is connecting to Spotify. [VIDEO PLAYBACK] -Look at me. Can you play something happy? GIL WIENBERG: And we have some algorithm to determine happy, disco, rock. [HAPPY MUSIC PLAYING] -Can you play disco? [MUSIC BEE GEES, "SATURDAY NIGHT LIVE"] GIL WIENBERG: Of course this. So a different set gestures for different genres. All right. [END PLAYBACK] Annie Zhang, another student, worked on the Shimi band, trying to create collaborations between three Shimis. And she looked at a lot of [INAUDIBLE], music [INAUDIBLE] elements you would get from the audio. Anything for the beat detection to the beat to the arm is for the energy. MFCC for vocalist to see if there's vocals there or melody. And then this is her interface. And she then came up from the low level features, detecting some high level features, beats, energetics, melodics, vocalness, percussiveness of the sound. And then she actually went and studied [INAUDIBLE] system and tried to create a dancing grammar for all of these gestures that will connect between how Shimi is dancing. And here are some of her ideas for that. And a whole system of different ways in which three Shimis can dance together-- solo, having the wave, two in synchronicity from both sides, and the main one is different. So a full system that later allowed her to have this. [VIDEO PLAYBACK] [MUSIC PLAYING] And every time it moved different, the gestures. [END PLAYBACK] And even in a very noisy environment in a demo-- [VIDEO PLAYBACK] [MUSIC PLAYING] [END PLAYBACK] Here too, we try to have an embodied musical mind and have an understanding of what are the important music notes in a piece. Tonally, maybe they're more stable, maybe it's the end of a phrase that goes up and so on. And then combine it with some understanding of what are the positions in which what can move to particular places, being aware of its own limitation in terms of the time of the gestures. So here you see the red dots that represents a particular end point in gestures that try to be based on the important notes melodically. Again, the combination of being aware of the embodiment and of musical theory and creating a system that, I think Mason here put some titles that you can read as he plays. [VIDEO PLAYBACK] [PLAYING GUITAR] [END PLAYBACK] And as you can see, Mason is extremely talented, and he is looking for an internship this summer. [LAUGHTER] Just to end the Shimi part-- maybe I'll just play this little clip. I couldn't make a Shimi without letting it dance to this song. [VIDEO PLAYBACK] [MUSIC STAR WARS, "CANTINA SONG"] [END PLAYBACK] And that's the last project that we just finished last year. I actually took a break. I took a sabbatical year off. And just when I was about to go to the sabbatical I got an email from Jason Barnes. He's an amputee drummer. He lost his arm, now it would be like three years ago. When we met it was two. And he saw some of the videos online and he was really devastated. There's a clip online. You can see him talking about how he thought he had nothing to live for if he cannot drum when he lost his arm when he was electrocuted in some weird accidents on a roof. And then he saw these video and he emailed me and asked me, can you use some of the technology to build me a robotic arm? And, of course, that was very intriguing for me. What he really wanted was to be able to control the grip. Because he had the arm up all the way up until here so he can still hit. But for a drummer, it's very important for them to have control of the grip, tight or loose to create different kinds of drum strokes. And that was very interesting for me. And the people I usually collaborate with couldn't come through, so I-- I don't know if anyone here is from Meka, but I did contact Meka who built this arm. And that was great but because I'm interested in improvisation and creativity, I really wanted to have this aspect of my research in this arm. So I asked Jason if he minded that we had two different sticks, not only one. One was all about grip, so you can have your expression and play it as he wants. The other one actually would listen and improvise and will listen to the music and listen to other musicians and try to maybe inspire you to create something else and become better or try to create music that he would never create otherwise. Let me show you a couple of clips first of the basic interaction of just controlling the group. And what we discovered after Phillip from Meka built the arm is that we don't need just to hold the stick, we can actually use it to generate hits. And after some work on noise reduction, we were able to some pretty quick responses, very low latency so Jason could just flex his muscle, in the muscle and actually create a beat, in addition to using his own arm. So there's some aspect of a cyborg there because part of the gesture is his own body and part of it is electronically based on his muscle and maybe other muscles, not only the forearm muscle. So actually, this took eight months. It was amazingly fast, again, thanks to Meka. And this is the first time Jason tried it. MALE SPEAKER: Next. Next. Next. GIL WIENBERG: And that's where we really got-- I didn't expect that, that we can create 20 hertz. And you know, my favorite polyrhythm games. I'll show more of the musical context later. But let's first-- see, here we didn't have the EMG working yet. It's a little knob that changes between tight and loose, but just to see the effect. This was a tight hit. [VIDEO PLAYBACK] [DRUMMING] Assuming the holding of the stick tight, this is a loose hit. [DRUMMING] And this is a role tight. [DRUMMING] [END PLAYBACK] Many more expression than he had before. He couldn't do all of this. And actually, he was in tears the first time. He said, I didn't play this well since I lost my arm. Which was the first time I actually worked with people. And it was pretty powerful. But, here, you can see the second stick. The kick is very short. Look here, we have an accelerometer and when he wants-- only when he wants, he can lift the arm quickly and the second stick will pop up. It will be really fast. OK. And then we can start with the improvisation and the creativity, the machine creativity. Before we do that we can just have a computer generate strikes. So nothing to do with him. Even though it's part of his body, suddenly someone else is generating hits for him. And even Pat Metheny's drummer cannot do. Not with one arm. And this is, again, playing with roll, with tumble, because it's so fast it becomes actually color and not even rhythm. And this is what I call cyborg drumming where you combine some of the computer generated hits with his own hits. He decides where to put the arm, where not, what to choose. Create a human machine interaction and musical collaboration. And at some point-- actually I wrote an NSF proposal about it-- we want to create some human brain machine interaction. But now some of the most sophisticated stuff we're doing with pedals. With his leg, it controls dynamics and expressive expression elements. And when this first clip went online, it also kind of became viral. And I got a lot of emails and one of them was from a guy called Tom Leighton who actually played with Richard Feynman. I didn't know that Richard Feynman was a drummer, playing hand drums. And he said, I have a bunch of recordings from Richard Feynman and I would love for you to play with it and see if you could do something with it. And of course I jumped on this opportunity. And what we did here is we put some of the rhythm that Richard Feynman played in his arm, so he had a chance to play with Richard Feynman in a weird way. And first you'll see him hovering over the drum, just listening to the rhythm, and then he starts to decide what drum, when he whats to lift the symbol, and so on so on. So it created all kinds of interaction with someone who is not with us anymore. And I think it also was a powerful experience for him. [VIDEO PLAYBACK] [DRUMMING] That's the rhythm by itself, just hovering. And now he adds to it with the other arm. [DRUMMING] Now he starts to decide where to put it. He's hovering but now we drew this as part of his gestures and movements. [END PLAYBACK] Here we have analysis of the chords and we try to create similar connections, a very intimate connection between the guitar player, Andrew, and Jason. Different chord that he plays. We analyze the chord and change the rhythms that Jason is playing. So suddenly he has to respond, not only to the arm that is generated by the computer or generated by his muscle, but generated or controlled by a different musician. And you see the whole eye contact and the whole, trying to create something together, happening between them two. [VIDEO PLAYBACK] [DRUMMING AND GUITAR] It was one of the first times that he did it and it's not tight, but it becomes tighter with every rehearsal and every performance. [DRUMMING AND GUITAR] [END PLAYBACK] And here he is, in context, trading four for those of you who play jazz. You play four bars and then someone else improvises four bars and back and forth. Here, the arm listens to the improvisation for four bars and then Jason gets to improvise. And the arm is responding, improvising, [INAUDIBLE] classic manipulation of what it heard in the four bars before. [VIDEO PLAYBACK] [JAZZ MUSIC] [END PLAYBACK] And I have longer clips for all of this online. This is the latest project. I just received this videos from Deepak, my student. Just yesterday he just finished doing that. And if you may have noticed, a lot of the expressions that if you are a drummer you probably know that you can hit in much more versatile ways than just a hit or not a hit. There's a lot of multiple bounces and a lot of expression in it. Here is Deepak who is a drummer. My students need to be musicians, and in this case roboticist. Actually he was just accepted to a robotics program at Northwestern. The first master student from music technologies that actually got into a robotics program. It shows the different kind of drumming techniques that he has. [VIDEO PLAYBACK] So all of this richness and expression, we couldn't have. So Deepak had to go and learn PID control, which again, is not one of the classes that we offered in music technology. [END PLAYBACK] And he tried to come up with a model for the talk, taking into account gravity. And something he had to know, it's a very complex system. The bounciness of the drum, for example, is not being taken into account. But he tried to put everything in simulation. And with some coefficient of restitution here is 0.85 just as a constant. And he tried to create all kind of bounce behaviors. Some of them are just based on humans and bouncing balls kind of techniques. But some of them are actually bounces that humans will not create or are not likely to. You see this is much faster. At its end, it's lifting for the next hit. And here you have a high to low force profile you see. This hit is softer than the next one. So you really have to change-- it's time varying talk changes. And he put it into a model and created a PID control. And this is the first clip. He just did it yesterday morning and sent it to me just before I left. [VIDEO PLAYBACK] So hopefully this will be embedded into some of the new performances. And we're going to France to perform in May. Hopefully Jason can play a much more wide range of expression. This is some of the future works that the NSF proposal that I have wrote is going to address. Am I'm working with a neuroscientist from Northwestern, [INAUDIBLE]. And we are trying to get some understanding about anticipation. Because there are latencies in the system, and no one isn't focusing on volition detection, which is basically-- your brain is shooting, sometimes seconds before your motor cortex actually knows that you're about to hit. In a way, you know you're going to do something before you know you're going to do something, or a part of your brain knows about it. I tried to get this data. We did some experiments with 36 EEG electrodes, and there maybe some way to get information before the hit is going to be made and then better anticipation algorithms. So if he hits with both hands, or hits with both times-- this hand hits by itself, it will be on time because we have some information. We did some machinery to detect different kind of strokes and we saw that the sum-- it's promising. It might be tricky and we have some outliers, but we can pretty much detect single strokes versus double strokes, which is something else that we can try to implement. And this is one way it will probably never look like. But this picture is in the proposal. Justin is an example. The trick here, when you create such cyborgs is how not to interfere with what you're doing anyhow. But as you can see, I'm jumping into third arm scenarios. So half of the proposal would be working with Jason on his own arm, creating this. But the second part is, let's try, available to anyone, whether you are disabled or abled, and have a third arm. And the promise, the spiel is that if we have anticipation algorithms and timing and synchronization, that works in music-- and music is a very, very time demanding and medium. You know, you hear five milliseconds that what you heard is two. So if we can pass synchronization in music, we can anticipate and help in all kind of scenarios-- on operation rooms or fixing a space station with a third arm. So how to put it on the body in a way that you can be completely-- first, the cognitive load, how to control it. So we thought about the brain, the organ of muscles that we can still try to hit. But the muscles are used for legs for the drums and for arms for the drums. This is a different kind of design for the future brain machine control. And I'll be happy to go for questions right now. Thank you very much. [APPLAUSE] AUDIENCE: I think you alluded to this, but can you say more about why you think the physical manifestation of a striking thing is important. Are there other artists actually getting information from the motion of the robot or is it just for looks. GIL WIENBERG: So one study showed that it helped with synchronization. That's the graphs that I showed when you see the arm moving, when you see the beat. You'll better synchronize your gestures with it. I'm a big believer in embodiment. I believe that that's why we go to concerts as opposed to listening to music at home. We want to see how people interact, not only by music but by gestures. A drummer and a guitar will end the song together. And there's a lot about interacting with gestures and being aware of your body as a musician that I think you cannot do without it. Of course, all of it can two iPhones that play this music. But I think we're going to miss the whole musical experience. It's something really deep and strong in embodiment in the musical experience. AUDIENCE: So two sort of related questions, if you don't mind. The first is, have you tried sort of persisting some information across several sessions for any of these to try to have the robot develop some kind of style more incrementally, and sort of lock into certain patterns that it prefers systematically over longer periods of time? GIL WIENBERG: I thought about it. We didn't get to it. It's a great idea because musicians-- that's what they do. There's length, there's lifelong learning. And we're doing just small episodes. There was a student that was interested in starting to do but then he decided that a master's is enough. He doesn't need a PhD. But it's a great suggestion and it's definitely something important for music. AUDIENCE: And then related to that, some composers or musicians have-- you talked about embodiment-- certain sort of different bodily features like, I don't know, someone like Django Reinhardt in jazz had only two fingers to use on the guitar. Rachmaninoff had very big hands. So have you thought about sort of creating robots that have different variations on what kind of motion is available to them and seeing how that guides things, or sort of, how more constraint can change? GIL WIENBERG: So that's the latest work by Mason with the path planning. And he tried to come up with a system. His PhD thesis will come up with a model of other musicians. That's one of his main goals to continue to develop the field of robotic musicianship. Other people who build robots will be able to use this kind of path planning ideas and put as a coefficient the different limitations that they have. AUDIENCE: OK. Thank you. GIL WIENBERG: Yeah. AUDIENCE: There was only one instance where you mentioned a robot knowing anything context-specific or genre-specific. For most of the examples, it was instead, everything that the robot should know about how to generate the content comes from the content it's receiving. But then there was one where one of your students programmed all of these jazz theories into it. Do you have an aversion to telling a robot genre-specific or context-specific information? Or do you want it to be as blank of a slate as possible? GIL WIENBERG: I don't have aversion for anything. I come from jazz. Jazz is about improvisation. A lot of what I'm trying to do is [INAUDIBLE] improvisation. So it doesn't necessarily fit maybe other genres. Like with classical, yes, Bach can be improvised and you can have all kind of counterpoints based on improvisation. But I can see a scenario when you're trying to put classical theory in it. I less see how it's going to be used for improvisation. I think that it should be very interesting, yes. AUDIENCE: Do you think you would ever do it negatively and say, OK, well, these things, if you ever come across this, this is not idiomatic? You should avoid it? I'm thinking, for example, in jazz of course, but also for the hand drums, like, I don't know anything about that style of music but would you ever want to say like, in whatever mutation or combination that comes up, this particular pattern is seen as ugly? GIL WIENBERG: I don't know if ugly, but-- AUDIENCE: Or not-- GIL WIENBERG: Not part of the grammar. Yes, we have it in the jazz system. AUDIENCE: OK. Cool. GIL WIENBERG: And I think it will be interesting to try to see what happens when we go to other genres, with maybe a different kind of tonal theory. AUDIENCE: I have a question back to the pathing analysis. I'm curious if you have a measure of how much less you are able to do when you do have these physical constraints, versus if you could have the machine improvise it and watch other people, and it would just have to produce tones. Is there some sort of measure as to the limitations, qualitatively versus what it could do, if it didn't have the physical limitations of forearms that can't cross over one another? GIL WIENBERG: I think we can deliver-- I don't have a measure. But I think a measure can be developed and I think a measure can also be used for future robot designers to build robots that have less of these limitations. And when we came up with the four arms, that this arm can never play this octave, we knew that. And we knew that, in a way, it might play faster than human and create 16 chords simultaneously which humans cannot do. But it will also be more limited than humans. And I think when I play piano, I do path planning in a way. If I don't have the fingers, if I just to look at notes with the fingers, I constantly, probably, do something similar to Viterbi's beam search job algorithm or something. Creating a measure in terms of coming up with different kind of chords that you can or cannot play-- different kind of speed of phrases that you can or cannot, you know. They're what, if you noticed, one for the black keys, for the accidentals and one line for-- so the limitation just there and see if you can play clusters. Can it play a cluster of C C sharp D E flat E? It can't. We don't have a measure to really come up with a coefficient, but I think it's a good idea. I think all of you have a lot of good ideas. AUDIENCE: Thank you. AUDIENCE: First, awesome talk. Thank you very much. The question I have has to do with, kind of, wanting to play with these robots myself. There's a bit of a bottleneck in that your team is doing the hardware and also all the software. Have you thought about moving towards a more platform-based approach where other labs could write code against your robot and be able to try it out? GIL WIENBERG: That's a good one. And by the way I'd like to give credit when credit is due. My students are mostly computer science students. The platform itself-- I have colleagues come from the MIT Media Lab. Guy Hoffman designed and Rob [INAUDIBLE] designed the robots and Meka designed the latest one. So this is not our research. Probably the Meka robot is the best candidate to become a platform. The others are too flaky, and too specific. But part of what I'm interested in is also the art and the design and the coolness of creating a new robot. So it's just because I love it. But I think it can be a good idea to try to, for the arm, for example, to try to make a platform, create code, create, I don't know, an API for other people to try to play with. Definitely. I think this field is pretty new and there's all kinds of [INAUDIBLE]. At some point, if enough people think it's important, maybe we will use a robotic operating system. But for now, we just focus on creating cool musical one-offs. AUDIENCE: Thanks. AUDIENCE: In the physical gesture robot, you showed a picture of a score where you had analyzed important points in the melody where you might end a gesture. But it was taking as input, raw audio. Is that right? As opposed to a MIDI file or a digital encoded score. GIL WIENBERG: The input was a chord that Mason and played from a MIDI piano. So it got the symbolic notes from the MIDI. AUDIENCE: Oh, OK. And he was playing a guitar I thought at one point as well? GIL WIENBERG: Yes, that was different. The guitar was only played with shimmies. Piano plays with Shimon. The guitar for Shimi to depict the right-- and yes, it did repeat-- actually, as I mentioned, this is offline. Mason played this melody, it did it offline and then it recreated it. AUDIENCE: Oh, OK. OK. Thank you. And one other thing. About the auto-correlation beat detection. Does that infer any time signature information? Or how good is that at detecting strange time signatures? If at all? GIL WIENBERG: It's not as good. Four quarters is much easier. A big estimation, so it pretty much focused on regular time, three, four. Others would be more difficult. AUDIENCE: Thank you. MALE SPEAKER: I had one question. I'm a classical musician and this is about online learning. I mean, yeah, online real-time learning from the instructor. So as a classical musician, you sit down, you have a lesson and you have a instructor that teaches you, based on what they're hearing, giving you feedback and then they correct or improve your playing right then. I don't know anything about instruction for jazz improvisation. I guess I've seen a bit of that. Wynton Marsalis has a program right now that you can watch where he's working with some young jazz musicians and he's giving them feedback. A lot of it's about communication and listening. Have you thought about that sort of thing? Not so much-- what you're showing here is the actual improvisation, but if you wanted to give feedback to the robot and say, you know, hey, why don't you try this. Or that's a little bit too busy. You maybe should be kind of simpler at this time and then add some more color later or something. I don't know. GIL WIENBERG: The same students that decided that MS is fine were thinking about learning. Starting from nothing and learning getting some input and reinforced. And then develop a style that will develop over time. We were thinking about. We didn't get into it. But yes, in jazz too, I think similar to classical music, that's how it works. First production of the sound. You know, when I first played piano, it was a lot of hits on my fingers and holding turtles. If I have some more students using learning-- not in the machinery applications of it, but actually your robot that knows nothing and starting to teach it slowly. It can be very interesting. Maybe very interesting new music perhaps that wouldn't come this way. MALE SPEAKER: OK, thank you very much. That was awesome. GIL WIENBERG: All right. Thank you. [APPLAUSE]
B1 中級 Gil Weinberg教授:"佐治亞理工學院的機器人音樂家"|谷歌講座 (Professor Gil Weinberg: "Robotic Musicianship at Georgia Tech" | Talks at Google) 129 7 richardwang 發佈於 2021 年 01 月 14 日 更多分享 分享 收藏 回報 影片單字