SARA ROBINSON: Hi, everyone.

Thank you for coming.

Today we're going to talk about machine learning APIs

by example.

I'm going to teach you how you can

access pre-trained machine learning models

with a single API call.

My name is Sara Robinson.

I'm a developer advocate on the Google Cloud Platform

team, which basically means I get to help build demos, give

talks about them, and bring product feedback back

to the engineering teams.

You can find me on Twitter, @SRobTweets.

And I live in New York.

So before we get started, let's talk

about what machine learning is at a high level.

So at a high level, machine learning

is teaching computers to recognize patterns

in the same way that our brains do.

So it's really easy for a child to recognize

the difference between a picture of a cat and a dog,

but it's much more difficult to teach computers

to do the same thing, right?

So we could write rules to look for specific things,

but we can almost always find a condition that's

going to break those rules.

So instead, what we want to do is

write code that finds these rules for us and improves

over time through examples and experience.

Here we have a neural network that's

identifying a picture as either a picture of a cat or a dog.

And we can think of the input to this network

as pixels in the image.

And then each neuron is looking for

a specific identifying feature.

Maybe it's the shape of the ear or the hair.

And then the output is a prediction--

in this case, that is a dog.

Let's take a step back from this for a moment

and let's try some human-powered image detection,

if we were to do this on our own.

We'll take this picture of an apple and an orange.

And let's say we were going to start

writing an algorithm that would identify

the difference between these two.

What are some features that you might look for you?

You can shout it out.

AUDIENCE: Color.

SARA ROBINSON: Color.

I heard a bunch of "color."

AUDIENCE: Shape.

SARA ROBINSON: Shape.

AUDIENCE: Texture.

SARA ROBINSON: Texture-- lots of good ones.

So color's a good one, but then what

would happen if we had black and white images?

Then we might have to start all over again.

So in that case, we could for a stem.

Texture would be good.

But then what happens if we add a third fruit?

If we add a mango, we have to start all over again as well.

But these pictures are all pretty similar, right?

So what would happen if we had pictures of two things that

were very different?

This should be really easy, right?

A dog and a mop have pretty much nothing in common

from these pictures that I can see.

But it's actually a little tricky.

So what we have here is pictures of sheep dogs and mops.

And it's actually kind of hard to tell the difference, right?

If we were going to write code that identified these two,

it would be pretty tricky to do.

And then what happens if we have photos of everything?

We don't want to write specific rules

to identify each little thing that we're trying to look for.

And in addition to photos, we could have many other types

of unstructured data.

We could have video, audio, text,

lots of different types of data we'd be dealing with.

And really, we want some tools to help

us make sense of all of this unstructured data.

And Google Cloud Platform has several products

to help you benefit from machine learning.

On the left-hand side here, you can use your own data

to build and train your own machine learning model.

So we have TensorFlow and Cloud Machine Learning Engine for you

to do that.

And on the right-hand side, this is

where I'm going to focus today.

This is what I like to call friendly machine learning.

These are machine learning APIs that give you

access to a pre-trained machine learning model with one

single REST API request.

So you make a request.

You send it some data.

You get some data back from this pre-trained model.

And you don't have to worry about building and training

your own model or anything that's going on under the hood.

I'm going to give you an introduction

to each of these five APIs.

I'm going to start with Vision, and I'm

going to end with Video Intelligence, which

is our newest API you may have seen in the keynote yesterday.

So let's get started with Vision.

The Vision API lets you do complex image detection

with a simple REST request.

And I'm going to start each section

by talking about a customer or customers

that are using each API.

So for the Vision API, the first example on the left

is Disney used the Vision API for a game

to promote the movie "Pete's Dragon."

And the way the game worked is that users were given a quest.

So they had a clue, and they had to take

a picture of that word--

maybe couch, computer, everyday objects.

And if they took the picture correctly,

they would superimpose an image of the dragon on that object.

So the problem was they needed a way

to verify that the user took a picture of the correct object

that they were prompted for.

And the Vision API was a perfect fit to do that.

They used the label detection feature, which basically tells

you, what is this a picture of?

And they were able to verify images in the game that way.

Realtor.com uses the Vision API for their mobile application.

So its a real estate listing service, and people

can go around as they are looking for houses

and take pictures of a "for sale" sign.

And they use the Vision API's OCR, Optical Character

Recognition, to read the text in the image

and then pull up the relevant listing for that house.

So those are two examples of the Vision API in production.

Let's talk a little bit more about the specific features

of the Vision API.

So as I mentioned, we have label detection,

which is kind of the core feature which

tells-- you send it an image.

In this case, it's a cheetah.

And it'll tell you what this is a picture of.

It'll give you a bunch of different labels back.

Face detection will identify faces in an image.

It'll tell you where those faces are in the image.

And it'll even tell you if they're happy, sad, surprised,

or angry.

OCR is what I mentioned with realtor.com's use case.

This can identify text in an image.

It will tell you where the text is, what the text says,

and what language it's in.

Explicit content detection will tell you,

is this image appropriate or not--

really useful if you've got a site

with a lot of user-generated content

and you don't want to manually filter images.

You can use this API method to do that really easily.

Landmark detection will tell you, is this a common landmark?

If so, what is the latitude and longitude?

And then logo detection, pretty self-explanatory,

will identify logos in an image.

So a quick look at some of the JSON response

you might get back for these different features.

This is face detection.

This is actually a selfie that I took with two teammates

on a trip to Jordan last year.

And the response you're looking at in the slide is on my face.

So it'll return an object for each face

you find in the image.

And we can see it says, headwear likelihood, very unlikely,

which is true.

I'm not wearing a hat.

But for both of my teammates, it did return headwear likelihood,

very likely.

And then we can see it's highlighted below.

It says, joy likelihood is very likely, which is true.

I am smiling in the picture.

The next feature I want to show you the response for

is landmark detection.

So we have a picture here of what

looks like the Eiffel Tower.

It's actually the Paris Hotel and Casino in Las Vegas.

I wanted to see if the Vision API was fooled.

And it was not.

It correctly identified this as the Paris Hotel and Casino.

You can see that MID in the JSON response.

That's an ID that maps to Google's Knowledge Graph

API, which will just give you a little more data

about the entity.

And it also tells us the latitude and longitude of where

this Paris Hotel and Casino is.

In addition to these features, we launched some new features

this week, which you may have heard

about in yesterday's keynote.

I'm going to quickly talk about what they are.

And then I'll show you some examples.

The first one is crop hints, which

will give you suggested crop dimensions for your photos.

Web annotations-- I'm super excited about this one.

This will give you some granular data

on entities, web entities that are found in your image.

It'll also tell you all the other pages where the image

exists on the internet.

So if you need to do copyright detection,

it'll give you the URL of the image and the URL

of the page where the image is.

And then finally, we announced document text annotations.

So in addition to the OCR we had before, this is improved

OCR for large blocks of text.

If you have an image of a receipt or something

with a lot of text, it'll give you

very granular data on the paragraphs and the words

and the symbols in that text.

Some examples of the new features,

I want to highlight web annotations.

So here we have a picture of a car.

It's in a museum.

I'm actually a big "Harry Potter" fan,

so this is a car from one of the "Harry Potter" movies.

And I wanted to see what the Vision

API was able to find from the web entities in this image.

So it was able to identify it correctly

as a Ford Anglia, which is correct.

This is from the second "Harry Potter" movie when they

tried to fly a car to school.

Second entity it returned is art science museum.

And this is a museum in Singapore

where this car is on display.

And it is finally able to tell me

that this car is from "Harry Potter," which

is a literary series.

So lots of great metadata you can get back from web

annotations.

Even more data that it returns--

it tells you the full matching image URLs.

Where else does this image exists on the internet?

Partial matching images.

And then finally, all the pages that point to this image.

It's really useful information you get back

with web annotations.

And you can try this in the browser with your own images

before writing any code.

Just go to cloud.google.com/vision.

You can upload your images, play around,

see all the responses you get back

from the different features.