Placeholder Image

字幕列表 影片播放

  • ♪ (intro music) ♪

  • It's really just in all of -- what machine learning is capable of,

  • in how we can extend human capabilities.

  • And we want to think more than just about discovering new approaches

  • and new ways of using technology; we want to see how it's being used

  • and how it impacts the human creative process.

  • So imagine, you need to find or compose a drum pattern,

  • and you have some idea of a drum beat that you would like to compose,

  • and all you need to do now is go to a website

  • where there's a pre-trained model of drum patterns

  • sitting on the online-- you just need a web browser.

  • You give it some human input and you can generate a space of expressive variations.

  • You can tune and control the type of outputs

  • that you're getting from this generative model.

  • And if you don't like it, you can continue going through it

  • exploring this generative space.

  • So this is the type of work that Project Magenta focuses on.

  • To give you a bird's eye view of what Project Magenta is about,

  • it basically is a group of researchers, and developers and creative technologists

  • that engage in generative models research.

  • So you'll see this work published in machine learning conferences,

  • you'll see the workshops,

  • you'll see a lot of research contributions

  • from Magenta.

  • You'll also see the code that after it's been published

  • put into open source repository on GitHub in the Magenta repo.

  • And then from there we'll see ways of thinking and designing creative tools

  • that can enhance and extend the human expressive creative process.

  • And eventually, ending up into the hands of artists and musicians

  • inventing new ways we can create and inventing new types of artists.

  • So, I'm going to give three brief overviews of the highlights

  • of some of our recent work.

  • So this is PerformanceRNN.

  • How many people have seen this? This is one of the demos earlier today.

  • A lot of people have seen and heard of this kind of work,

  • and this is what people typically think of when they're thinking

  • of a generative model, they're thinking, "How can we build a computer

  • that has the kind of intuition to know the qualities

  • of things like melody and harmony, but also expressive timing and dynamics?"

  • And what's even more-- it's even more interesting now

  • to be able to explore this for yourself in the browser enabled by TensorFlow.js.

  • So,this is a demo we have running online.

  • We have the ability to tune and control some of the output that we're getting.

  • So in a second, I'm going to show you this video of what that looks like,

  • you would have seen it out on the demo floor

  • but we will show you and all of you watching online,

  • and we were also able to bring it even more alive by connecting

  • a baby grand piano Disklavier that is also a midi controller

  • and we have the ability to perform alongside the generative model

  • reading in the inputs from the human playing the piano.

  • So, let's take a look.

  • ♪ (piano) ♪

  • So this is trained on classical music data from actual live performers.

  • This is from a data set that we got, from a piano competition.

  • ♪ (piano) ♪

  • I don't know if you noticed, this is Nikhil from earlier today.

  • He's actually quite a talented young man.

  • He helped build out the browser version of PerformanceRNN.

  • ♪ (piano) ♪

  • And so we're thinking of ways that we take bodies of work,

  • we train a model off of the data, then we create these open source tools

  • that enable new forms of interaction of creativity and of expression.

  • And this is all these points of engagement are enabled by TensorFlow.

  • The next tool I want to talk about that we've been working on

  • is Variational Autoencoders.

  • How many people are familiar Layton's space interpolation?

  • Okay, quite a few of you.

  • And if you're not, it's quite simple-- you take human inputs

  • and you train it through on your own network,

  • compressing it down to an embedding space.

  • So you compress it down to some dimensionality

  • and then you reconstruct it.

  • So you're comparing the reconstruction with the original and trying to train,

  • build a space around that, and what that does is that creates

  • the ability to interpolate from one point to another

  • touching on the intermediate points where a human may have not given input.

  • So the machine learning model may have never seen an example

  • that it's able to generate, because it's building an intuition

  • off of these examples.

  • So, you can imagine if you're an animator,

  • there's so many ways of going from cat to pig.

  • How would you animate that?

  • So, we're train--there's an intuition that the artist would have

  • in creating that sort of morphing from one to the other.

  • So we're able to have the machine learning model now also do this.

  • We can also do this with sound, right?

  • This technology actually carries over to multiple domains.

  • So, this is NSynth, and we've released this,

  • I think some time last year.

  • And what it does is it takes that same idea

  • of moving one input to another.

  • So, let's take a look. You'll get a sense of it.

  • Piccolo to electric guitar.

  • (electric guitar sound to piccolo)

  • (piccolo sound to electric guitar)

  • (piccolo and electric guitar sound together)

  • So, rather than recomposing or fading from one sound to the other,

  • what we're actually able to do is we're able to find these intermediary,

  • recomposed sound samples and produce that.

  • So, it looks, you know, there's a lot of components to that.

  • There's a wave and a decoder, but really it's the same technology

  • underlying the encoder-decoder variational autoencoder.

  • But when we think about the types of tools that musicians use,

  • we think less about training machine learning models.

  • We see drum pedals right? -- I mean not drum pedals.

  • Guitar pedals, these knobs and these pedals

  • that are used to tune and refine sound to cultivate the kind of art

  • and flavor a musician is looking for.

  • We don't think so much about setting parameter flags

  • or trying to write lines of python code

  • to create this sort of art in general.

  • So what we've done--

  • Not just are we interested in finding and discovering new things.

  • We're also interested in how those things get used in general--

  • used by practitioners, used by specialists.

  • And so we've created hardware, we've taken a piece of hardware

  • where we've taken the machine learning model, we've put it into a box

  • where a musician can just plug in

  • and explore this latent space in performance.

  • So take a look on how musicians feel, what they think in this process.

  • ♪ (music) ♪

  • (woman) I just feel like we're turning a corner

  • of what could be new possibility.

  • It could generate a sound that might inspire us.

  • (man) The fun part is even though you think you know what you're doing,

  • there's some weird interaction happening

  • that can give you something totally unexpected.

  • I mean, it's great research, and it's really fun,

  • and it's amazing to discover new things, but it's even more amazing to see

  • how it gets used and what people think to create with alongside it.

  • And so, what's even better is that it's just released, NSynth Super,

  • in collaboration with the Creative Lab London.

  • It's an open source hardware project.

  • All the information and the specs are on GitHub.

  • We talk about everything from potentiometers,

  • to the touch panel, to the code and what hardware it's running on.

  • And this is all available to everyone here today.

  • You just go online and you can check it out yourself.

  • Now music is more than just sound right?

  • It's actually a sequence of things that goes on.

  • So when we think about this idea of what it means

  • to have a generative music space, we think also about melodies,

  • and so just like we have cat to pig,

  • what is it like to go from one melody to the next?

  • And moreover, once we have that technology, how does it --

  • what does it look like to create with that?

  • You have this expressive space of variations--

  • how do we design an expressive tool that takes advantage of that?

  • And what will we get out of it?

  • So this is also another tool

  • that's developed by another team at Google,

  • to make use of melodies in a latent space,

  • so how interpolation works,

  • and then building a song or some sort of composition with it.

  • So let's take a listen. Say you have two melodies....

  • ♪ ("Twinkle Twinkle Little Star") ♪

  • And in the middle....

  • ♪ (piano playing variation) ♪

  • You can extend it....

  • ♪ (piano playing variation) ♪

  • And we really are just scratching the surface of what's possible.

  • How do we continue to have the machine learn

  • and have a better intuition for what melodies are about.

  • So again to bring it back full circle, using different compositions