BIOGRAPHY. >> WELL, THE CHAT IS VERY SMALL,

I WANTED TO SEE YOU AND HEAR YOU, HALF THE TIME, I SET UP MY

MIC INCORRECTLY. WELCOME TO A SPECIAL FRIDAY TRAINING EPISODE,

I'M DAN SHIFFMAN, THERE'S A LOT OF THINGS THAT ARE EXCITING

ABOUT THIS EPISODE. SO SOMEBODY SAID HI, SO I'M

GOING TO TAKE THAT AS THINGS ARE WORKING. AND SO TODAY IS A

SPECIAL EPISODE. NUMBER ONE, WE ARE DOING A MACHINE LEARNING

PROJECT FROM START TO FINISH, TRAINING A MODEL ENTIRELY IN THE

CLOUD, GETTING THAT TRAINING MODEL BACK, AND THEN

IMPLEMENTING THAT MODEL IN THE BROWSER USING JAVASCRIPT. SO

ALL THOSE PIECES, THAT IS GOING TO HAPPEN, AND THE WHOLE THING

IS GOING TO TAKE AN HOUR AND A HALF. TO PRESENT ALL OF THIS TO

YOU, WE HAVE A GUEST. YINING SHI, YOU MIGHT REMEMBER HER FROM

THE CODING TRAIN TUTORIAL THAT SHE MADE, I WILL LINK HER, SHE

IS AN ARTIST AND RESEARCHER, A CORE CONTRIBUTOR FOR THE MACHINE

LEARNING 5 LIBRARY, THE ML5.JS LIBRARY, PART OF THIS TUTORIAL,

SHE WROTE THE STYLE TRANSFER MODULE OF ML5.JS, AND THAT IS

WHAT SHE IS GOING TO DO AND PRESENT. SO YINING WILL BE HERE

IN A MINUTE, AFTER MY LONG INTRODUCTION. AND THIS VIDEO IS

SPONSORED BY SPELL, SPELL IS A CLOUD COMPUTING FOR MACHINE

LEARNING SERVICE. I DID AN INTRODUCTION TO SPELL, HOW TO

SET IT UP, WHAT IT DOES, WHAT ARE THE BASIC COMMANDS. IF YOU

ARE WATCHING THIS AS AN ARCHIVE, YOU MIGHT WANT TO WATCH IT FIRST

AND RETURN. IF YOU ARE WATCHING THIS LIVE, HAVE NOT SEEN THAT,

WE WILL HELP YOU GET SET UP WITH THAT. IF YOU WANT TO SIGN UP

FOR AN CCOUNT AND FOLLOW ALONG, YOU CAN GET $100 IN FREEBIES.

YOU CAN GO TO SPELL.RUN/CODINGTRAIN.

OKAY, AND ALSO, THANK YOU TO

SPELL, I'M -- SO WE HAVE CLOSED CAPTIONING, FOR THE FIRST TIME,

I'M USING

REALP TIME HUMAN WRITTEN CAPTIONED GENERATED BY WHITE

COAT CAPTIONING. YOUTUBE HAS AUTO CAPTIONS, BUT THIS IS TYPED

BY A PROFESSIONAL CAPTIONER IN REALTIME AS I'M SPEAKING, I

THINK. THIS REMINDS ME OF THE ELEPHANT

AND PIGGY BOOK, WHICH IS -- YOU ARE IN A BOOK, AND THE

CHARACTERS -- THEY CAN MAKE THE CHARACTERS SAY WHATEVER THEY

WANT. I CAN MAKE THE CAPTIONER TYPE BLUEBERRY, MANGO,

WATERMELON, THOSE WORDS SHOULD BE APPEARING. SO THANK YOU TO

SPELL.RUN FOR THE SPONSORSHIP, THANK YOU TO YINING FOR BEING

HERE, AND THANK YOU TO WHITE COAT CAPTIONING FOR THE

CAPTIONING SERVICES AND TO SPELL FOR PROVIDING THE FUNDS FOR

THOSE. AND I WILL BE TO THE SIDE LOOKING FOR THE YOUTUBE

CHAT, I WILL TRY TO ANSWER THEM, AND MOSTLY WE WILL SAVE

QUESTIONS UNTIL THE END. IF THERE IS AN IMPORTANT KEY

QUESTION, I MIGHT INTERRUPT AND ASK THAT. AND ONE OTHER THING:

YINING WILL TELL YOU ABOUT THIS, BUT I CANNOT RESIST. TO TRAIN A

STYLE TENSOR MODEL ON THE CLOUD, WITH A GPU, IT TAKES A LONG

TIME. SO WE ARE LIKE A COOKING SHOW MECHANIC, WE'RE GOING TO

START THE TRAINING PROCESS AND THEN HAVE THE PRE-TRAINED MODEL

IN THE OVEN, FULLY BAKED, TO SHOW YOU HOW IT WORKS. IF YOU

WATCH THIS TUTORIAL, YOU WILL BE ABLE TO TRAIN YOUR OWN STILL

TRANSFER MACHINE LEARNING MODEL USING SPELL.RUN, AND IMPLEMENT

THAT MODEL IN THE

BROWSER. OKAY, SO I'M LOOKING IN THE

CHAT. THAT IS ALL OF MY INTRODUCTORY STUFF, YES. SO I

AM JUST GOING TO TRANSFER IT OVER TO YINING, I'M GOING TO

MUTE MY MICROPHONE, I WILL UNMUTE IT ONCE IN A WHILE ONCE I

HAVE SOMETHING IMPORTANT TO SAY. AND WE WILL GET STARTED, OKAY?

SPEAKER: THANK YOU SO MUCH. HI, I'M YINING, AND I'M EXCITED TO

BE HERE TODAY TO TALK ABOUT

STYLE TRANSFER. HERE.

AND I WANT TO THANK EVERYONE FOR WATCHING THIS VIDEO.

I HOPE YOU ENJOY THIS VIDEO.

LET'S GET STARTED! TODAY, WE ARE GOING TO TALK

ABOUT STYLE TRANSFER. WE ARE GOING TO DO FOUR THINGS

TODAY. WE WILL TALK ABOUT WHAT IS STYLE

TRANSFER, HOW DOES IT WORK, AND WE ARE GOING TO A PLATFORM

CALLED SPELL TO TRAIN A NEW STYLE TRANSFER MODEL, AND PORT

THE MODEL INTO ML5.JS TO CREATE A AN INTERACTIVE DEMO.

SPELL AND ML5JS ARE BOTH TOOLS

THAT MAKE ML MORE APPROACHABLE FOR A BROAD RANGE OF AUDIENCE.

FOR OUR PROJECT TODAY, ML5JS ALLOWS US TO RUN OUR MODEL IN

THE BROWSER. BY THE WAY, ML5.JS IS A

JAVASCRIPT LIBRARY BASED ON TENSORFLOW.JS. SO OUR MODEL

THAT WE HAVE TODAY WOULD ALSO WORK IN THE

TENSORFLOW.JS. AND SPELL PROVIDES COMPUTING

POWERS FOR US TO TRAIN A MODEL FASTER.

IF I TRAIN THE MODEL ON MY OWN

LAPTOP, IT MIGHT TAKE SEVERAL

DAYS, BUT WITH THE REMOTE GPU PROVIDED SPELL, IT WILL ONLY

TAKE A FEW HOURS. LET ME SHOW YOU WHAT ARE WE

GOING TO BUILD AT THE END OF THIS VIDEO.

THIS IS A DEMO: HTTPS://YINING1023.GITHUB.IO/

STYLETRANSFER_SPELL/. THIS DEMO READS THE IMAGES IT

GETS FROM OUR WEBCAM, AND TRANSFER THE STYLE OF THE IMAGE

INTO THE STYLE OF THIS ART WORK. THE STYLE IMAGE IS AN ANCIENT

CHINESE PAINTING CALLED FUCHUN SHANJU TU.

THE STYLE IMAGE DOESN'T HAVE TOO

MANY COLORS, BUT IF YOU TRAIN THE MODEL WITH OBVIOUS STYLE, IF

YOU USE THOSE STYLE IMAGES, YOU WILL GET A MORE OBVIOUS

RESULT. THIS IS THE DEMO THAT WE ARE

GOING TO BUILD

TODAY. BEFORE WE BUILD ANYTHING,

WHAT IS STYLE TRANSFER? STYLE TRANSFER IS THE TECHNIQUE

OF RECAST THE CONTENT OF ONE IMAGE IN THE STYLE OF ANOTHER

IMAGE.

FOR EXAMPLE, HERE IS A

PHOTOGRAPH, THIS TECHNIQUE CAN EXTRACT THE CONTENT OF THE

PHOTO, AND THE STYLE OF THIS ART

WORK, AND COMBINE THE TWO TO CREATE A NEW IMAGE.

HERE ARE MORE EXAMPLES.

SO, HOW DOES IT WORK? STYLE TRANSFER WAS FIRST

INTRODUCED IN THE PAPER A NEURAL ALGORITHM OF ARTISTIC STYLE IN

2015 BY GATYS.

IN THE PAPER, THEY PROPOSED A SYSTEM THAT USES CONVOLUTIONAL

NEURAL NETWORKS TO SEPARATE AND

RECOMBINE CONTENT AND STYLE OF ARBITRARY IMAGES.

BY THE WHY, AN

COLUSIONAL NEURAL NETWORK IS A DEEP NEURAL NETWORK USED TO

ANALYZE IMAGES. THE IDEA IS THAT IF WE TAKE A

CONVOLUTIONAL NEURAL NETWORK THAT IS TRAINED TO RECOGNIZE

OBJECTS WITHIN IMAGES THEN THAT

NETWORK HAS DEVELOPED SOME

INTERNAL REPRESENTATIONS OF THE CONTENT AND STYLE OF AN IMAGE.

MORE IMPORTANTLY, THE PAPER

FINDS THAT THE REPRESENTATIONS

IN THE IMAGE CAN BE

SEPARATED. WE CAN TAKE THE CONTENT AND

STYLE IN ONE

IMAGE AND ARE SEPARABLE, WHICH MEANS WE CAN TAKE THE CONTENT

REPRESENTATION FROM ONE IMAGE AND STYLE REPRESENTATION FROM

ANOTHER TO GENERATE A BRAND NEW IMAGE.

THE CNN THAT GATYS USED IS CALLED VGG. IT'S A NETWORK

CREATED BY THE VISUAL GEOMETRY GROUP AT OXFORD UNIVERSITY.

THIS CNN IS THE WINNER OF IMAGENET, AN OBJECT RECOGNITION

CHALLENGE IN 2014.

WE WILL SEE THE NAME VGG AGAIN WHEN WE TRAIN THE MODEL.

THAT'S BECAUSE WE ARE TRYING TO

GET REPRESENTATIONS OF AN IMAGE

FROM THIS VGG CONVOLUTIONAL

NEURAL NETWORK. NEXT, CONVOLUTIONAL NEURAL

NETWORKS LOOK LIKE

FILTERS, DIFFERENT LAYER HAS DIFFERENT REPRESENTATIONS OF AN

IMAGE. AN INPUT IMAGE CAN BE

REPRESENTED AS A SET OF FILTERED IMAGES AT EACH LAYER IN THE CNN.

WE CAN VISUALISE THE INFORMATION AT DIFFERENT LAYERS IN THE CNN

BY RECREATE THE INPUT IMAGE FROM ONE OF THE FILTERED IMAGE.

WE CAN SEE THAT IMJ

A, B, C, D, E ARE THE RECREATED IMAGES.

THEY ARE ALMOST

PERFECT. AS THE LEVEL GETS HIGHER AND

HIGHER, ALL OF THOSE DETAILED PIXEL INFORMATION IS LOST, BUT

THE HIGH LEVEL CONTENT OF THIS IMAGE IS STILL HERE.

FOR EXAMPLE, FOR THIS IMAGE E HERE, GIVEN THAT WE CANNOT SEE

IT CLEARLY, BUT WE CAN SEE THAT -- HERE'S A HOUSE, THIS IMAGE.

SO THIS IS HOW CONTENT REPRESENTATION LOOKS LIKE IN

THIS NETWORK. NEXT, WE WILL TALK ABOUT STYLE

REPRESENTATION.

ON TOP OF THE ORIGINAL CNN CONVOLUTIONAL NEURAL NETWORK,

REPRESENTATIONS THEY BUILT A NEW FEATURE SPACE THAT CAPTURES THE

STYLE OF AN INPUT IMAGE. THE STYLE REPRESENTATION

COMPUTES CORRELATIONS BETWEEN THE DIFFERENT FEATURES IN

DIFFERENT LAYERS OF THE CNN.

FOR DETAILED IMPLEMENTATION, WE CAN CHECK THE PAPER.

BUT AS THE LEVEL GETS HIGHER AND HIGHER, WE FIND THAT THEY

RECREATE THE STYLE OF THE INPUT IMAGE FROM STYLE

MATCHES THIS ARTWORK BETTER AND BETTER, BUT THE INFORMATION OF

THE GLOBAL ARRANGEMENT OF THE SCENE IS LOST.

FOR EXAMPLE, FOR IMAGE D AND E, THE STYLE IS VERY CLEAR TO US

NOW. BUT WE CANNOT SEE IF THERE'S A HOUSE ON THIS PHOTO

ANYMORE, BECAUSE THE CONTENT REPRESENTATION IS

LOST. AND THEN AFTER WE HAVE THE

CONTENT REPRESENTATION OF THE PHOTO, AND THE STYLE

REPRESENTATION OF THIS ART WORK, AND WE'RE GOING TO SYNTHESIZE

A NEW IMAGE THAT CAN MATCH THOSE TWO AT THE SAME TIME. THIS IS

HOW STYLE TRANSFER

WORKS. AND JANE COGEN, THE CREATOR OF

MACHINE LEARNING FOR ARTISTS, HE MAKES THIS AMAZING DEMO VIDEO

THAT TALKS ABOUT WHAT'S A CONVOLUTIONAL NEURAL NETWORK,

AND HOW IT SEES EACH LAYER,O YOU SO YOU WILL HAVE A BETTER

UNDERSTANDING OF HOW THIS CONVOLUTIONAL NEURAL NETWORK

SEES IMAGES AND HOW IT FILTERS OUT THE IMAGE AND GETS THE

REPRESENTATION OUT OF ONE IMAGE AFTER WATCHING HIS

VIDEO. O I HIGHLY RECOMMEND THAT YOU

WATCH HIS

VIDEO. AND GATYS'S PAPER OPENED UP A

NEW AREA OF RESEARCH, AND DIFFERENT KINDS OF TRANSFER

APPEARED IN THE LAST THREE YEARS.

WE ARE GOING TO TAKE A LOOK AT A FEW OF THEM HERE.

AND THEN WE ARE GOING

TO DIVE INTO TRAINING YOUR STYLE TRANSFER MODEL WITH

SPELL. IN 2016, THIS

PAPER CAME OUT. IT IS CALLED A FAST STYLE

TRANSFER, IT SHOWS THAT A NEURAL NETWORK CAN APPLY A FIXED STYLE

TO ANY INPUT IMAGE IN

REALTIME. IT BUILDS ON THE

GATYS STYLE TRANSFER MODEL, BUT IT IS A LOT FASTER. THIS FAST

STYLE

TRANSFER HAS AN IMAGE TRANSFORMATION NETWORK AND A

LOSS CALCULATION NETWORK TO TRAIN THIS NETWORK. WE NEED TO

PICK A FIXED STYLE IMAGE AND USE A LARGE BATCH OF DIFFERENT

CONTENT IMAGE AS TRAINING EXAMPLES. SO, IN THEIR PAPER,

THEY TRAINED THEIR NETWORK, THIS MICROSOFT COCO DATASET, WHICH IS

AN OBJECT RECOGNITION DATASET OF 18,000

IMAGES. TODAY, WE WILL USE A TENSORFLOW IMPLEMENTATION OF

THIS STYLE TRANSFER, SO WE ARE ALSO GOING TO USE THIS COCO

DATASET. WE ARE GOING TO DOWNLOAD THIS DATASTYLE LATER.

AND HERE IS AN IMAGE FROM

THEIR PAPER. THIS IS THE ORIGINAL PHOTO, THIS

GATYS RESULT AND THIS IS THE STYLE TRANSFER RESULT AND IT

WORKS A LOT FASTER. AND THE NEXT STYLE TRANSFER IS

FOR VIDEOS. THIS MODEL CAME OUT IN 2016,

TOO. WE MAY THINK WE KNOW HOW TO

TRANSFER IMAGES, FOR VIDEOS, WE CAN JUST TRANSFER THE FRAME --

EACH FRAME OF THE VIDEO ONE BY ONE AND THEN STITCH THOSE IMAGES

TOGETHER TO MAKE A

TRANSFER

VIDEO. BUT IF WE DO THAT, WE CAN SEE

THE RESULT IS NOT GOOD BECAUSE

THE VIDEO WILL FLICKER A LOT, BECAUSE MACHINE DOESN'T KNOW ANY

INFORMATION ABOUT THE PREVIOUS IMAGE.

SO YOU CAN SEE, IF WE JUST DO THAT, THE VIDEO WILL FLICKER

A LOT. THE PAPER IMPROVED

FRAME-TO-FRAME STABILITY BY ADDING AN OPTICAL-FLOW ALGORITHM

THAT TELLS THE MACHINE THE

POSSIBLE MOTIONS FROM FRAME TO FRAME.

IT'S ALSO CALLED TEMPORALLY

COHERENT, SO THE TRANSFERRED

VIDEO WOULDN'T BE FLICKERING TOO MUCH.

SO WE CAN SEE SOME RESULTS HERE. THIS VIDEO IS NOT FLICKERING AT

ALL. AND THEY GOT AMAZING RESULTS FROM THEIR

MODEL. THIS IS THE TRANSFERRED VIDEO,

THE RESULT LOOKS

GREAT.

LET'S GO TO THE NEXT

MODEL. THIS IS A VERY COOL MODEL

APPEARED IN 2017, IT'S CALLED

DEEP PHOTO TRANSFER: THE STYLE

TRANSFER WE SAW SO FAR WORK REALLY WELL IF WE ARE LOOKING

FOR SOME ARTISTIC PAINTING RESULTS, BUT THEY ADD SOME

DISTORTION TO THE INPUT IMAGE. THEY DON'T LOOK REALISTIC.

BUT THIS DEEP PHOTO TRANSFER CAN

PRODUCE REALISTIC PHOTOS.

THIS INPUT IMAGE ON THE LEFT, AND IN THE MIDDLE, THIS IS THE

STYLE IMAGE, AND THEN ON THE RIGHT, THIS IS THE OUTPUT IMAGE.

THE OUTPUT IMAGE LOOKS LIKE A REGULAR PHOTO TO ME, SO THE