Placeholder Image

字幕列表 影片播放

  • Just to get started.

  • Anyone who hasn't heard of WebAssembly?

  • Great.

  • Anyone here who has had a reason to use WebAssembly in a production app?

  • All right.

  • So we are all about on the same page.

  • I would like to invite you back into journey of time to the year 2017.

  • Like most of you, I heard about WebAssembly but beyond surface level stuff I didn't know

  • much about it.

  • I read this headline and thought huh, I guess if WebAssembly really does kill JavaScript

  • I will figure that stuff out but it seems like people have wanted JavaScript to die

  • for a while and we are still here.

  • What does a typical web dev need Wasm for?

  • I work for a consulting firm based in Seattle called GEN/UI.

  • I think our dev manager described it best when he said our web expertise is in doing

  • something weird.

  • We built a screen recording and editing app for Chrome OS that runs entirely client-side.

  • Our requirements were as follows: The app was meant to look and feel like it was native

  • to Chrome OS and act as a competitor to others out there.

  • The client requested the video provided video editing capability and needed to run locally

  • offline.

  • There are a couple ways to develop for Chrome OS and we decided to use Chrome package OS

  • and they are essentially web apps given additional hardware APIs.

  • As we started to investigate how we wanted to implement the project, it was a surprise

  • to discover how much it is possible to do with media in native JavaScript.

  • For instance, if the user consents, you can capture their whole screen or Webcam and use

  • the native recorder API to record the screen, create a video and allow the user to download

  • it to their machine.

  • That is possible in any modern web browser.

  • There is nothing special about Chromebook that lets you do it.

  • If all we needed was a screen recording app we would be set with those tools but the client

  • had requested some video editing capabilities.

  • As a result, we started to get into a realm of things that are technically possible in

  • JavaScript that is just not a great idea for a variety of reasons.

  • For instance, if you wanted to create a water mark you can create a media stream that collects

  • data from a canvas element.

  • You can then draw your watermark on to every frame of that canvas element and merge it

  • into the stream from your screen recording but unsurprisingly it is too memory intensive.

  • The resulting videos were kind of like watching a very badly dubbed foreign film.

  • You have got all the pieces there but they don't line up.

  • Another video editing feature we wanted to implement was trimming or shortening the video.

  • It turns out you can actually do that in plain JavaScript so long as you only want to trim

  • the end of the video.

  • If you want to trim the beginning, you are out of luck, at least in Chrome.

  • That brings us to functionality that simply isn't possible in native JavaScript.

  • Once you create that WebM file you are done.

  • There is no way to add a watermark or filter after the fact and it isn't possible to re-encode

  • the video.

  • The media recorder API gives you a WebM so if you want a gif or MP4 you can't do that

  • with native functionality.

  • At this point, we went back to the client and recommended a server-side only solution.

  • There are two big reasons for this.

  • Video editing is a memory intensive process and Chromebooks are meant to do most work

  • from the Cloud by design.

  • And client side video editing pushes the limits on what is possible today.

  • We had two people working on this without an infinite budget.

  • The client asked us to move ahead with a client-side only implementation eventually.

  • This set us off on the journey of determining what is possibly limiting.

  • We determined video editing isn't possible in native JavaScript we needed to find a way

  • to run a code that is not native JavaScript.

  • Running non-JS code in a browser has a long history.

  • Anyone remember this thing called Flash?

  • Anyone?

  • It is kind of obscure.

  • Any ActionScript devs out there?

  • Any OUBT about Silver Light?

  • You may have heard WebAssembly isn't just another way to write plugins and that is true

  • but without understanding more about plugins in general it is hard to know why the comparison

  • is made.

  • The '90s are in again.

  • Let's go back to 1995.

  • We are all wearing JNCOs and the Slater guy is this guy.

  • Your browser of choice is Netscape and they just released the Netscape plugin API or NP

  • API which was the first way to develop plugins or browser extension and handle media from

  • within the browser.

  • What happened was the browser would come across content it couldn't handle and detect the

  • file type and load the appropriate plugins that will run in place on your web page.

  • As of today, NP API is deprecated in every browser.

  • That is because the Advent of HTML5 means the browser can handle a lot of things natively

  • that would otherwise have been handled by plugins.

  • Plugins didn't actually fully die with NP API.

  • Google released the pepper API which is a build chain that produces binary file types

  • native client or pepper native.

  • It was meant to provide a more secure, less complex alternative to NP API.

  • You can recompile libraries written in C or C++ and that could be run within the browser

  • and accessed in JavaScript.

  • For the WebAssembly fans in the crowd this may sound familiar.

  • Why aren't we talking about pepper and how it harkens the death of the most widely used

  • program language?

  • An open standard for pepper was not created and they were only supported on Chrome, Chromium

  • and blink-base browsers.

  • Portable native client is deprecated for every platform except Chromebook but we were building

  • for Chromebooks so let's put a pen in that.

  • WebAssembly, like portable native client, takes advantage of common hardware capabilities

  • to deliver native or near native performance and allows libraries written in C, C++ and

  • rest into a Wasm format.

  • Developers can write directly in WebAssembly for the super nerds out there.

  • Unlike pepper WebAssembly is an open standard and has been adopted by every modern browser.

  • It is really something that you can use and deliver to the vast majority of the web.

  • That doesn't actually say so much about what is great about WebAssembly in the first place,

  • or why we would want to use C or C++.

  • There are a couple reasons why you might use WebAssembly in your application.

  • The first of these is performance.

  • Performance on WebAssembly is faster than native JavaScript and that is mostly due to

  • a lack of dynamic typing in WebAssembly.

  • You might look into WebAssembly if you have a particularly heavy operation you could perform

  • in JavaScript but prefer not to.

  • Every single page web application framework has a DOM diffing operation.

  • It is a heavy thing to do and it can have negative performance implications for updating

  • your page particularly in component-heavy scenario.

  • Glimmer, which is the DOM rendering engine by the Ember team has an existing build that

  • leverages WebAssembly for this purpose and that is a methodology that should result in

  • a much faster rendering engine.

  • I am definitely not saying that because of the primary architecture of Glimmer is speaking

  • later today.

  • There are folks looking into this as well for React.

  • Performance aside, the second reason you may want to use Wasm speaks to video editing.

  • You cannot edit video in JavaScript but you can edit video in C++.

  • You can do several things in C++ or Rest like video editing, platform immolation and games.

  • Wasm was built for our use case and we needed to choose a video editing library we can use.

  • It was important to find a library because there were only two of us and we had neither

  • the budget or expertise to build something from scratch.

  • It turns out there are not tons of video editing options in C++ or Rust and that may be due

  • to FFmpeg which is a command line tool that supports adding watermark or tripping or cropping

  • video or anything you might want to do with it.

  • FFmpeg could be used through portable native client or WebAssembly and we were unable to

  • find another good candidate.

  • The first thing we did was go to the mpm registry and there is an FFmpeg.

  • I am grateful to the author of this package and the person who forked it and added extra

  • documentation.

  • Without these folks, we would have been lost but these were enough bread crumbs for us

  • to figure out how to do this.

  • Getting back to my MPM install.

  • I saw a couple concerning things off the bat.

  • First is the size.

  • The entirety of FFmpeg is about 15 bytes and that is a lot to load and a lot to ask of

  • users to download.

  • The default build includes tons of functionality we didn't plan on using.

  • It also includes the H2-64 kodak which was still pat patented.

  • This is an old version, also.

  • We came to the conclusion there is an existing MPM install for this library, but we needed

  • to compile our own Wasm file.

  • This is the point at which we started to get into technologies that JS devs don't tend

  • to use on a day to day basis and the first is called Emscripten.

  • For those that haven't done a lot of this thing it is vaguely analogous to a tool like

  • Babel.

  • It takes uncompiled C++ library and builds it in a way that a Wasm is produced.

  • You can pass in various build options.

  • If you want to build your Wasm binary in a way it is compatible with web workers you

  • will do that as part of the overall Emscripten and build command.

  • That is a lot of high level information.

  • Let's bring it down.

  • The following example is a little hello, world written in C. Like most of you I don't program

  • in C# but the main method is what runs when the app loads.

  • We imported the standard input and output library.

  • In a C# application this prints to the console and in our app prints to the JavaScript console

  • but we will display it in the DOM also.

  • Now we need to compile our hello world to a Wasm file.

  • You can do that in Emscripten with the following command.

  • You can see we do need to specify Wasm and that is because Emscripten can compile to

  • other formats too.

  • And dash 0 output to HTML lets us know I want Emscripten to build the page rather than just

  • compile a binary.

  • Let's look at the web page.

  • There is a lot more going on than just LEL hello world.

  • The default gives you a page with a console you can use to see the Wasm output and it

  • has JavaScript glue code.

  • If this was a Thanksgiving turkey drawing tutorial we are at this strep drawing an outline

  • of the hand.

  • You know what comes next?

  • We don't need hello, world.

  • We need FFmpeg and there are a few steps in between.

  • It is worth just trying here to convert FFmpeg and see what happens next.

  • Make is a Unix tool.

  • And clang is a front end compilation for interface and gcc is the GNU/Unix.

  • This is the root folder of the FFmpeg library.

  • We need to use the project's make file and substitute cling and gcc with the own Emscripten

  • build tool.

  • Piece of cake.

  • In case you thought it was cake, let me assure you it isn't.

  • Emscripten built their own tool to help you substitute for clang and gcc.

  • It is called M con emconfigure.

  • We will run emmake make and that should build the library into the format and bit code.

  • Assuming everything is well, we can run a command similar to the hello, world command

  • from earlier but of course, everything hasn't gone well.

  • After about 20 minutes of compilation, emmake make has an error generating deep from within

  • the heart of this massive library.

  • Anyone who has struggled to get builds to translate knows the pain that can accompany

  • production builds particularly as they become more complex.

  • Building a project in C# isn't different and compiling tat project into a format it was

  • never intended for adds a greater degree of complexity.

  • Looking at the hello, world.

  • C from earlier it seems reasonable to debug the process even though we are not C# developers.

  • It didn't see reasonable to debug this giant make file and dive in the heart of this legacy

  • framework.

  • Emconfigure makes assumptions about your C# base build environment that may not be true.

  • It was necessary to go in and replace every reference to the existing build change.

  • Emscripten can't use anything using ASM and that is used in any C# libraries to provide

  • compiler instructions to gcc.

  • Since we are not building with gcc we need to pass the option to build without this.

  • And finally, Emscripten will not convert every possible application.

  • In the end we prevailed.

  • Here we are looking at a video that has been created in native JavaScript through a screen

  • recording of a YouTube clip.

  • Our media recorder returned a blob we can convert to a file and save to the user's local

  • machine.

  • You will notice we cannot seek through the video and this is due to a Chrome-specific

  • bug where their mediator recorder doesn't create key frames as it reports therefore

  • the find of file created doesn't know important things.

  • The client wanted to add trim functionality to the app and that makes total sense.

  • We have started screen recordings for getting to set-up our screen and needing to trim out

  • the first bit.

  • This is where FFmpeg comes in handy.

  • If we run our FFmpeg we will ask it to trim to the desired length.

  • I compiled my FFmpeg build, create a new web worker and listen to make sure that worker

  • is ready, once everything is loaded I will post a message containing all the information

  • that FFmpeg needs to kick off a trim operation.

  • We can see the type of thing I want to run is called run and that is the default or main

  • method in FFmpeg and I have passed in data which is actually an array buffer.

  • Finally, I pass in the same arguments I would pass in if I were using FFmpeg from the command

  • line.

  • We have got a dash-SS where you start the video.

  • Dash T is how long the video is.

  • Dash c copy means to use the existing kodak.

  • Let's see what that looks like.

  • I kicked off the FFmpeg by default in the background and we can see before too long

  • we get a little video.

  • It is different than it was before.

  • It is only two seconds long.

  • That being said, there are some important caveats for using this technology.

  • The last video editing feature was to provide crop and that is in case the user wants to

  • only display a portion of what they have created in their video.

  • You can see we have kicked off our operation, we are getting lots of logging output from

  • FFmpeg and that is great.

  • We know something is happening but that video is only a minute long and we are still watching

  • it convert so we can't put that in front of the user, obviously.

  • The other