Name: 梯度下降，神經網絡如何學習 | 深度學習第 2 章 (Gradient descent, how neural networks learn | Deep Learning Chapter 2)
Uploaded: 2025-08-01T10:04:53.000Z
Duration: 20 min 33 s

Last video I laid out the structure of a neural network.

上一段視頻中，我介紹了神經網絡的結構。

程度副詞

I'll give a quick recap here just so that it's fresh in our minds and then I have two main goals for this video.

我在這裡簡單回顧一下，讓大家記憶猶新，然後我為這段視頻設定了兩個主要目標。

情態副詞

The first is to introduce the idea of gradient descent which underlies not only how neural networks learn but how a lot of other machine learning works as well.

首先是介紹梯度下降的概念，它不僅是神經網絡學習的基礎，也是許多其他機器學習的基礎。

Then after that we're gonna dig in a little more to how this particular network performs and what those hidden layers of neurons end up actually looking for.

之後，我們將深入研究這個特定網絡的性能，以及這些隱藏層神經元最終實際尋找的是什麼。

As a reminder our goal here is the classic example of handwritten digit recognition, the hello world of neural networks.

在此提醒大家，我們的目標是手寫數字識別這一經典案例，這是神經網絡的 "你好世界"。

These digits are rendered on a 28 by 28 pixel grid, each pixel with some grayscale value between 0 and 1.

這些數字呈現在一個 28 x 28 像素的網格上，每個像素的灰度值介於 0 和 1 之間。

Those are what determine the activations of 784 neurons in the input layer of the network.

這些決定了網絡輸入層中 784 個神經元的激活。

And then the activation for each neuron in the layers is based on a weighted sum of all the activations in the previous layer plus some special number called a bias.

然後，各層神經元的激活基於前一層所有激活的加權總和，再加上一些稱為偏置的特殊數字。

Then you compose that sum with some other function like the sigmoid squishification or a ReLU the way that I walked through last video.

然後，用其他函數（比如上一段視頻中提到的 sigmoid squishification 或 ReLU）對總和進行運算。

In total given the somewhat arbitrary choice of two hidden layers here with 16 neurons each the network has about 13,000 weights and biases that we can adjust and it's these values that determine what exactly the network you know actually does.

在這裡，我們任意選擇了兩個隱藏層，每個隱藏層有 16 個神經元，是以網絡總共有大約 13,000 個權重和偏置可以調整，正是這些值決定了網絡的實際運行效果。

Then what we mean when we say that this network classifies a given digit is that the brightest of those 10 neurons in the final layer corresponds to that digit.

那麼，當我們說這個網絡對給定的數字進行分類時，我們的意思是，在最後一層的 10 個神經元中，最亮的神經元與該數字相對應。

And remember the motivation that we had in mind here for the layered structure was that maybe the second layer could pick up on the edges and the third layer might pick up on patterns like loops and lines and the last one could just piece together those patterns to recognize digits.

記得我們設計分層結構的動機是，第二層可以識別邊緣，第三層可以識別循環和線條等模式，最後一層可以將這些模式拼湊起來識別數字。

在這裡，我們瞭解到網絡是如何學習的。

What we want is an algorithm where you can show this network a whole bunch of training data which comes in the form of a bunch of different images of handwritten digits along with labels for what they're supposed to be and it'll adjust those 13,000 weights and biases so as to improve its performance on the training data.

我們需要的是一種算法，在這種算法中，你可以向這個網絡展示一大堆訓練數據，這些數據以手寫數字的不同影像形式出現，並標註了這些數字應該是什麼，然後它就會調整這 13,000 個權重和偏差，從而提高它在訓練數據上的性能。

Hopefully this layered structure will mean that what it learns generalizes to images beyond that training data.

希望這種分層結構意味著它所學到的知識能推廣到訓練數據以外的影像中。

And the way we test that is that after you train the network you show it more labeled data that it's never seen before and you see how accurately it classifies those new images.

我們的測試方法是，在訓練網絡後，向它展示更多從未見過的標註數據，看看它對這些新影像的分類準確度如何。

Fortunately for us and what makes this such a common example to start with is that the good people behind the MNIST database have put together a collection of tens of thousands of handwritten digit images each one labeled with the numbers that they're supposed to be.

幸運的是，MNIST 數據庫背後的好心人收集了數以萬計的手寫數字圖像，每張影像都標註了數字。

And as provocative as it is to describe a machine as learning, once you actually see how it works it feels a lot less like some crazy sci-fi premise and a lot more like, well, a calculus exercise.

儘管將機器描述為學習機器很有煽動性，但一旦你真正看到它是如何工作的，你就會覺得它不像是一個瘋狂的科幻前提，而更像是一個微積分練習。

I mean basically it comes down to finding the minimum of a certain function.

我的意思是，基本上可以歸結為找到某個函數的最小值。

Remember conceptually we're thinking of each neuron as being connected to all of the neurons in the previous layer and the weights in the weighted sum defining its activation are kind of like the strengths of those connections.

請記住，從概念上講，我們認為每個神經元都與上一層的所有神經元相連，而定義其激活的加權和中的權重就像是這些連接的強度。

And the bias is some indication of whether that neuron tends to be active or inactive.

而偏差則是該神經元趨於活躍還是不活躍的某種跡象。

And to start things off we're just going to initialize all of those weights and biases totally randomly.

一開始，我們將完全隨機地初始化所有權重和偏置。

Needless to say this network is going to perform pretty horribly on a given training example since it's just doing something random.

不用說，這個網絡在給定的訓練示例中會表現得非常糟糕，因為它只是在做一些隨機的事情。

For example you feed in this image of a 3 and the output layer just looks like a mess.

例如，你輸入這張 3 的影像，輸出層看起來就像一團糟。

So what you do is you define a cost function, a way of telling the computer, no, bad computer, that output should have activations which are 0 for most neurons but 1 for this neuron.

是以，你要做的就是定義一個成本函數，用這種方式告訴計算機，不，壞計算機，輸出的激活值對大多數神經元來說應該是 0，但對這個神經元來說應該是 1。

To say that a little more mathematically what you do is add up the squares of the differences between each of those trash output activations and the value that you want them to have.

說得更數學化一點，你要做的就是把每個垃圾輸出激活值與你希望它們具有的值之間的差值平方相加。

And this is what we'll call the cost of a single training example.

這就是我們所說的單個培訓案例的成本。

Notice this sum is small when the network confidently classifies the image correctly but it's large when the network seems like it doesn't really know what it's doing.

請注意，當網絡有信心正確地對影像進行分類時，這個總和很小，但當網絡似乎不知道自己在做什麼時，這個總和就很大。

So then what you do is consider the average cost over all of the tens of thousands of training examples at your disposal.

是以，你要做的就是考慮你所掌握的數以萬計的培訓案例的平均成本。

This average cost is our measure for how lousy the network is and how bad the computer should feel.

這個平均成本就是我們衡量網絡有多糟糕、電腦應該感覺有多糟糕的標準。

Remember how the network itself was basically a function, one that takes in 784 numbers as inputs, the pixel values, and spits out 10 numbers as its output.

還記得網絡本身基本上是一個函數，它的輸入是 784 個數字（像素值），輸出是 10 個數字。

And in a sense it's parameterized by all these weights and biases.

從某種意義上說，所有這些權重和偏差都是它的參數。

Well the cost function is a layer of complexity on top of that.

成本函數則是在此基礎上增加了一層複雜性。

It takes as its input those 13,000 or so weights and biases and it spits out a single number describing how bad those weights and biases are.

它將 13 000 個左右的權重和偏差作為輸入，然後吐出一個數字，描述這些權重和偏差有多糟糕。

And the way it's defined depends on the network's behavior over all the tens of thousands of pieces of training data.

而定義的方式取決於網絡在數以萬計的訓練數據中的表現。

But just telling the computer what a crappy job it's doing isn't very helpful.

但是，僅僅告訴電腦它做得有多糟糕並沒有什麼用。

You want to tell it how to change those weights and biases so that it gets better.

你要告訴它如何改變這些權重和偏差，使它變得更好。

To make it easier rather than struggling to imagine a function with 13,000 inputs just imagine a simple function that has one number as an input and one number as an output.

與其費力地想象一個有 13 000 個輸入的函數，不如想象一個只有一個數字作為輸入和一個數字作為輸出的簡單函數。

How do you find an input that minimizes the value of this function?

如何找到使該函數值最小的輸入值？

Calculus students will know that you can sometimes figure out that minimum explicitly, but that's not always feasible for really complicated functions.

學過微積分的學生都知道，有時可以明確地算出最小值，但對於真正複雜的函數來說，這並不總是可行的。

Certainly not in the 13,000 input version of this situation for our crazy complicated neural network cost function.

當然，在這種情況下，我們瘋狂複雜的神經網絡成本函數的 13,000 個輸入版本是不會出現這種情況的。

A more flexible tactic is to start at any old input and figure out which direction you should step to make that output lower.

一個更靈活的策略是，從任何舊的輸入開始，找出你應該朝哪個方向邁步，使輸出降低。

Specifically, if you can figure out the slope of the function where you are, then shift to the left if that slope is positive, and shift the input to the right if that slope is negative.

具體來說，如果你能找出函數的斜率，那麼如果斜率為正，就向左移動輸入，如果斜率為負，就向右移動輸入。

If you do this repeatedly, at each point checking the new slope and taking the appropriate step, you're going to approach some local minimum of the function.

如果反覆這樣做，在每個點上檢查新的斜率並採取適當的步驟，就會接近函數的某個局部最小值。

And the image you might have in mind here is a ball rolling down a hill.

在這裡，你可能會想到一個從山上滾下來的球。

And notice, even for this really simplified single input function, there are many possible valleys that you might land in, depending on which random input you start at.

請注意，即使是這個非常簡化的單一輸入函數，也有很多可能的谷底，這取決於你從哪個隨機輸入開始。

And there's no guarantee that the local minimum you land in is going to be the smallest possible value of the cost function.

而且也不能保證你所處的局部最小值就是成本函數的可能最小值。

That's going to carry over to our neural network case as well.

這也將延續到我們的神經網絡案例中。

And I also want you to notice how if you make your step sizes proportional to the slope, then when the slope is flattening out towards the minimum, your steps get smaller and smaller.

我還想讓你們注意到，如果你的步長與坡度成正比，那麼當坡度向最小值逐漸變平時，你的步長就會越來越小。

And that kind of helps you from overshooting.

Bumping up the complexity a bit, imagine instead a function with two inputs and one output.

如果把複雜程度再提高一些，想象一個有兩個輸入和一個輸出的函數。

You might think of the input space as the xy-plane, and the cost function as being graphed as a surface above it.

你可以把輸入空間想象成 xy 平面，把成本函數想象成其上方的一個曲面。

Now instead of asking about the slope of the function, you have to ask which direction should you step in this input space so as to decrease the output of the function most quickly.

現在你要問的不是函數的斜率，而是在這個輸入空間中應該向哪個方向移動，才能最快速地減少函數的輸出。

In other words, what's the downhill direction?

換句話說，下坡的方向是什麼？

And again, it's helpful to think of a ball rolling down that hill.

同樣，把它想象成一個從山上滾下來的球，也會有所幫助。

Those of you familiar with multivariable calculus will know that the gradient of a function gives you the direction of steepest ascent.

熟悉多元微積分的人都知道，函數的梯度給出了最陡的上升方向。

Basically, which direction should you step to increase the function most quickly?

從根本上說，你應該向哪個方向邁進才能最快速地增加功能？

Naturally enough, taking the negative of that gradient gives you the direction to step that decreases the function most quickly.

很自然地，取梯度的負值就能得出函數下降最快的方向。

And even more than that, the length of this gradient vector is actually an indication for just how that steepest slope is.

不僅如此，這個梯度矢量的長度實際上還表明了最陡峭的坡度有多大。

Now if you're unfamiliar with multivariable calculus and you want to learn more, check out some of the work that I did for Khan Academy on the topic.

現在，如果你對多元微積分不熟悉，想了解更多，可以看看我為可汗學院做的一些關於這個主題的工作。

Honestly though, all that matters for you and me right now is that in principle there exists a way to compute this vector.

老實說，現在對你我來說最重要的是，原則上存在著計算這個向量的方法。

This vector that tells you what the downhill direction is and how steep it is.

這個矢量可以告訴你下坡的方向和坡度。

You'll be okay if that's all you know and you're not rock-solid on the details.

如果你只知道這些，而對細節又不是很清楚，那也沒關係。

Because if you can get that, the algorithm for minimizing the function is to compute this gradient direction, then take a small step downhill, and just repeat that over and over.

因為如果你能做到這一點，那麼最小化函數的算法就是計算出這個梯度方向，然後向下走一小步，就這樣不斷重複。

It's the same basic idea for a function that has 13,000 inputs instead of two inputs.

對於一個有 13 000 個輸入而不是兩個輸入的函數來說，基本思路是一樣的。

Imagine organizing all 13,000 weights and biases of our network into a giant column vector.

想象一下，將我們網絡中所有 13,000 個權重和偏差組織成一個巨大的列向量。

The negative gradient of the cost function is just a vector.

成本函數的負梯度只是一個向量。

It's some direction inside this insanely huge input space that tells you which nudges to all of those numbers is going to cause the most rapid decrease to the cost function.

在這個巨大的輸入空間中，有一個方向會告訴你，對所有這些數字進行哪些調整會導致成本函數最快速地下降。

And of course with our specially designed cost function, changing the weights and biases to decrease it means making the output of the network on each piece of training data look less like a random array of 10 values, and more like an actual decision that we want it to make.

當然，有了我們專門設計的成本函數，改變權重和偏置來減少成本函數，就意味著讓網絡在每個訓練數據上的輸出看起來不像由 10 個值組成的隨機數組，而更像我們希望它做出的實際決策。

It's important to remember this cost function involves an average over all of the training data, so if you minimize it, it means it's a better performance on all of those samples.

重要的是要記住，這個成本函數涉及到所有訓練數據的平均值，是以如果將其最小化，就意味著在所有這些樣本上都有更好的表現。

The algorithm for computing this gradient efficiently, which is effectively the heart of how a neural network learns, is called backpropagation, and it's what I'm going to be talking about next video.

有效計算這種梯度的算法，實際上就是神經網絡學習的核心，叫做反向傳播，這也是我下一個視頻要講的內容。

There, I really want to take the time to walk through what exactly happens to each weight and each bias for a given piece of training data, trying to give an intuitive feel for what's happening beyond the pile of relevant calculus and formulas.

在這裡，我真的想花點時間來說明，在給定的訓練數據中，每個權重和每個偏置究竟會發生什麼變化，除了一堆相關的微積分和公式之外，我還想給大家一個直觀的感受。

Right here, right now, the main thing I want you to know, independent of implementation details, is that what we mean when we talk about a network learning is that it's just minimizing a cost function.

在這裡，現在，我想讓你們知道的最主要的事情，就是當我們談論網絡學習時，我們的意思是，它只是在最小化一個成本函數。

And notice, one consequence of that is that it's important for this cost function to have a nice smooth output, so that we can find a local minimum by taking little steps downhill.

注意，這樣做的一個後果是，成本函數必須有一個平滑的輸出，這樣我們就可以通過小步下坡找到局部最小值。

This is why, by the way, artificial neurons have continuously ranging activations, rather than simply being active or inactive in a binary way, the way that biological neurons are.

順便提一下，這就是為什麼人工神經元具有連續的激活範圍，而不是像生物神經元那樣簡單地以二元方式激活或不激活。

This process of repeatedly nudging an input of a function by some multiple of the negative gradient is called gradient descent.

這種以負梯度的某個倍數反覆推動函數輸入的過程被稱為梯度下降。

It's a way to converge towards some local minimum of a cost function, basically a valley in this graph.

這是一種向成本函數的局部最小值收斂的方法，基本上就是這個圖中的一個山谷。

I'm still showing the picture of a function with two inputs, of course, because nudges in a 13,000 dimensional input space are a little hard to wrap your mind around, but there is actually a nice non-spatial way to think about this.

當然，我仍然展示的是一個有兩個輸入的函數的圖片，因為在一個 13000 維的輸入空間中進行推導有點難以理解，但實際上有一種非空間的好方法來思考這個問題。

Each component of the negative gradient tells us two things.

負梯度的每個分量都告訴我們兩件事。

The sign, of course, tells us whether the corresponding component of the input vector should be nudged up or down.

當然，符號告訴我們輸入矢量的相應分量應該向上還是向下移動。

But importantly, the relative magnitudes of all these components kind of tells you which changes matter more.

但重要的是，所有這些組成部分的相對大小可以告訴你哪些變化更重要。

You see, in our network, an adjustment to one of the weights might have a much greater impact on the cost function than the adjustment to some other weight.

你看，在我們的網絡中，調整其中一個權重對成本函數的影響可能比調整其他權重大得多。

Some of these connections just matter more for our training data.

其中一些連接對我們的訓練數據更為重要。

So a way that you can think about this gradient vector of our mind-warpingly massive cost function is that it encodes the relative importance of each weight and bias.

是以，你可以把這個梯度向量看作是我們令人難以置信的龐大成本函數，它編碼了每個權重和偏置的相對重要性。

That is, which of these changes is going to carry the most bang for your buck?

也就是說，在這些變化中，哪種變化能讓您獲得最大的收益？

This really is just another way of thinking about direction.

這其實只是另一種思考方向的方式。

To take a simpler example, if you have some function with two variables as an input, and you compute that its gradient at some particular point comes out as 3, 1, then on the one hand you can interpret that as saying that when you're standing at that input, moving along this direction increases the function most quickly.

舉個更簡單的例子，如果你有一個以兩個變量為輸入的函數，你計算出它在某個特定點的梯度為 3、1，那麼一方面你可以把它理解為，當你站在這個輸入點時，沿著這個方向移動能最快地增加函數。

That when you graph the function above the plane of input points, that vector is what's giving you the straight uphill direction.

當你在輸入點的平面上方繪製函數圖時，該矢量會給你帶來筆直的上坡方向。

But another way to read that is to say that changes to this first variable have three times the importance as changes to the second variable.

但另一種理解是，第一個變量的變化的重要性是第二個變量變化的三倍。

That at least in the neighborhood of the relevant input, nudging the x-value carries a lot more bang for your buck.

至少在相關輸入的鄰近地區，對 x 值的推移會帶來更多的效益。

Alright, let's zoom out and sum up where we are so far.

好了，讓我們放大並總結一下目前的情況。

The network itself is this function with 784 inputs and 10 outputs, defined in terms of all of these weighted sums.

網絡本身就是一個有 784 個輸入和 10 個輸出的函數，由所有這些加權和定義。

The cost function is a layer of complexity on top of that.

成本函數在此基礎上又增加了一層複雜性。

It takes the 13,000 weights and biases as inputs and spits out a single measure of lousiness based on the training examples.

它將 13,000 個權重和偏差作為輸入，並根據訓練示例輸出單一的 "糟糕度 "測量值。

And the gradient of the cost function is one more layer of complexity still.

而成本函數的梯度又增加了一層複雜性。

It tells us what nudges to all of these weights and biases cause the fastest change to the value of the cost function, which you might interpret as saying which changes to which weights matter the most.

它告訴我們，對所有這些權重和偏差進行哪些調整會導致成本函數值發生最快的變化，你可以將其理解為對哪些權重進行哪些調整最重要。

So, when you initialize the network with random weights and biases and adjust them many times based on this gradient descent process, how well does it actually perform on images that it's never seen before?

那麼，當你用隨機權重和偏置初始化網絡，並根據梯度下降過程多次調整它們時，它在從未見過的影像上的實際表現如何呢？

字幕列表影片播放

梯度下降，神經網絡如何學習 | 深度學習第 2 章 (Gradient descent, how neural networks learn | Deep Learning Chapter 2)

relevant

struggle

recognize

figure