I thought we talk about something called optic flow.

So this is something that you might know I've heard about before, but it actually turns up in quite a lot of different things.

So very broadly optic flow is about understanding how things are moving in an image.

So the pixel level, how motion is happening.

So how things are moving around across the whole image, and this is useful for a number of reasons.

If you've got a wobbly, shaky camera, you can use it for image stabilization.

I wasn't insinuating anything on dhe.

Yes, so you can use optic flow to see what kind of global motion is happening and sort of take that out drones.

Things like that.

Some of them have got optical sensors that will kind of look a sort of how the world's moving around them.

So this isn't about tracking sort of individual objects or things.

This is about seeing how the global emotional the surfaces in the scene are moving around.

If you can kind of break up the scene by motion, it's another way of segmenting an image or labeling an image.

If you've seen some videos about that before, So in addition to things like color or texture, you can use how things are moving around to help sort of figure out how to divide up a scene.

So I always like three dear.

Suppose you get parallax is exactly what you can see, that you're moving differently to the window.

Exactly, and you can either.

You can pull out some other things there as well, so things closer to the camera tend to move faster.

So you get kind of an idea of depth.

And there's certain techniques, such as something called Structure for Emotion, which does exactly that.

It figures out how things appear in three D by seeing how the camera moves around.

Let's have a think about sort of what's going on in an image as you move a camera.

So obviously an image is composed of lows of pixels, potentially millions of pixels on.

When you move your camera around, there's some kind of change happening in those pixels on.

What we want to figure out is how that motion at an individual pixel level is happening.

Okay, so that means pretty much for every pixel in the image we want to try and apply some kind of motion vector to it to understand where it's moving too.

Okay, now, this is a pretty hard problem.

There's many, many pixels in the image on to understand exactly how each of them are moving.

We're gonna have to kind of make some assumptions and some simplifications, and that's where theon tic flow techniques kind of come in.

So this is a really simple image here where we just got sort of one shape in the middle of it like this, perhaps a ball or something like that on.

If we're calculating optic flow across this image and the ball is moving, what we're going to end up with is remember, this is sort of every single pixel.

We're gonna get these flow vectors when we're talking about optic flow.

Really?

What we're considering is sort of the motion, this level within the object or surface that's moving.

What's going on?

Okay, so here I've drawn Cem had optic flow vectors, showing that maybe this ball is sort of moving down.

And if the background was moving as well, you know, practice, you'd have lots of vectors over the background showing that the background was moving a certain way.

It's a different surfaces in the image could move in different directions.

But it's important to remember that we're just working at the pixel level.

So we're trying pick apart what's happening on individual pixels within the image.

So this is quite complicated to do because the real world is complicated.

So we have to make a lot of assumptions in order to try and pull out the motion of essentially these individual pixels.

That represents something called motion flow.

And that's something sort of moving through the real world.

So if I move something around that's moving in the real world, but how does it move on the image?

We've just got this to de plane of pixels on.

So how do we understand from that house?

Something's moving sort of in three day.

So there's a number of assumptions that we have to make.

One of the assumptions that we have to make is that the things like the lighting doesn't change, Okay, so if you if you think, is this piece of paper, if I move my hand across it and we get a shadow, then optic flow techniques might pick up on that shadow is being something actually moving in the image rather than just the lighting changing.

So one thing we have to assume is that lighting isn't isn't changing too much.

Um, and obviously in the real world, that's quite challenging, because we get lots of shadows and stuff happening that we it's gonna confuse these kind of approaches.

So another sort of simplification that we make is that when we're looking at this flow, we only do it over very small changes in in time.

So unlike tracking where we might be following something through a whole video sequence here, we're just looking at two frames.

So we'll take two neighboring frame of the video and see how the pixels have moved between them, as well as having things like lighting changing, which can break optic flow.

Another problem we can get some of the other way round is that maybe there's not enough features on the surface to be ableto detect the difference.

Anyway, I was gonna show you an example with a ball, but the only really showing people like find was this tiny one.

So if you spin it, it's spinning, of course, but you can't see it spinning because you can't see the texture turning round on the vault, especially because it's reflecting lighting, which isn't moving.

So one of the problems we have with optic flow is that we're making all these assumptions, you know, is such that we can actually see things moving.

And that's not always true.

One of the key constraints we make when we're writing down how we're gonna calculate optic flow is this idea of a brightness constraint on what that really means is if we've got our image here like this, we've got the intensity of a particular point.

So this point here are brightness in our image.

So our grayscale values, the higher is in the new year, is toe white lower isn't here, is too black.

Is that our position X comma?

Why?

So that's all pixel at a particular time t on.

What we're thinking about is essentially where has the thing that pixel here moved to in the next frame.

So it's gonna have moved only a little bit even much more than that, actually, but less draw it like that.

So we've got our new position here that we think this information has moved to whatever's in the real world has moved here in the image.

And so this is gonna be essentially the brightness are X plus a little bit.

Why, plus a little bit AT T plus electric.

So we've got some changes happening in Space X, plus a little bit.

Wipe us a little bit and also a change happening in time as well.

So because we got these little changes you can use, actually derivative, you can use a neat tricks where we're just looking at radiance in the image to figure out this equation.

We could rewrite it as something called the optic flow equation was just basically uses image derivatives toe look at changes in gray scale over very tiny patches.

Of course, if you want to calculate derivatives in in an image, you can use things like kernels like That's a Bell Colonel, which I think Mike's done a video before.

So the maths boils down to essentially being able to do pixel wise calculations to do with gray scale.

So looking in neighboring reason regions to see changes in space on looking at changes over time as well, say differences between frames.

Looking at that now, from my videographer head and thinking four K 60 frames per second.

Yeah, this is a lot calculator.

So there's a few tricks.

So perhaps at least nicely onto how a few people have solved this because this is the problem.

And actually, now you've got a load of different optimization techniques that will try and give you these vectors where we've moved to here.

This is kind of known as you remove.

The Saudis are our motion vectors that we've got everywhere in the image.

And you're right.

It could take a little while to calculate that.

So we thought about how you can talk about optic flow with little changes in space on little changes in time, that means derivatives.

And so things like sub l on which we have had a camera with a fairly high so settings.