Name: 人工智能如何學會看東西 [DINOv3] (How AI Taught Itself to See [DINOv3])
Uploaded: 2025-10-01T01:29:34.000Z
Duration: 15 min 57 s

情態副詞

When we look at images, we can easily recognize objects, semantics, and understand the 3D geometry of the world.

當我們觀察影像時，可以很容易地識別物體、語義，並理解世界的三維幾何。

In this video, we will explore how AI learned to see without label data.

在本視頻中，我們將探討人工智能如何在沒有標籤數據的情況下學會看東西。

For a computer, an image is just a grid of numbers.

對於計算機來說，影像只是一個數字網格。

To make sense of these numbers, AI needs to extract meaningful features from an image.

要理解這些數字，人工智能需要從影像中提取有意義的特徵。

Good features help us identify which regions of the image are similar to the red cross, revealing semantic structures even in a complex cluttered scene.

好的特徵可以幫助我們識別影像中哪些區域與紅十字相似，即使在複雜雜亂的場景中也能揭示語義結構。

Let's look at a simple example to see why features are important.

讓我們看一個簡單的例子，瞭解功能為何重要。

Here we have data points from two groups, blue and orange.

這裡有來自藍色和橙色兩組的數據點。

In this space, building a linear classifier to separate the two groups is very hard.

在這個空間中，建立一個線性分類器來區分兩組是非常困難的。

No matter how we try, we can't find a good way to separate them.

無論我們怎麼努力，都找不到把它們分開的好辦法。

Now let's transform the data into radial coordinates.

現在，讓我們將數據轉換成徑向座標。

By changing our perspective on the data, separating the two groups with a straight line becomes much easier.

換個角度看數據，用直線將兩組數據分開就變得容易多了。

This example highlights the importance of having good feature representations.

這個例子凸顯了擁有良好特徵表徵的重要性。

But how do we design an effective feature representation?

但是，我們該如何設計有效的特徵表示呢？

One way is to learn features by training a new network to solve tasks like classification.

一種方法是通過訓練一個新網絡來學習特徵，以解決分類等任務。

On top of the extracted features, we place an output projection head, typically a linear layer or a multilayer perceptron, that predicts class scores.

在提取的特徵之上，我們放置一個輸出投影頭，通常是線性層或多層感知器，用於預測類別得分。

We then use a softmax layer to turn the scores into probabilities for each class.

然後，我們使用 softmax 層將分數轉化為每個類別的概率。

We train the model to make its predicted probabilities match this label by minimizing cross-entropy loss.

我們通過最小化交叉熵損失來訓練模型，使其預測概率與該標籤相匹配。

This loss measures the difference between the predicted probability distributions from the softmax layer and the actual one-hot encoded label.

這種損耗測量的是軟最大層預測的概率分佈與實際單擊編碼標籤之間的差異。

After training the model on a large label dataset, we can use its learned weights as a great starting point for new tasks.

在大型標籤數據集上對模型進行訓練後，我們就可以利用其學習到的權重作為新任務的起點。

This process is called transfer learning.

It allows us to fine-tune the model for a wide range of applications, such as fine-grained classification, object detection, or semantic segmentation.

這使我們能夠對模型進行微調，以適應各種應用，如細粒度分類、物體檢測或語義分割。

However, class labels are often too simple.

然而，類別標籤往往過於簡單。

They only tell us the main object in the image, like bear or cat.

它們只能告訴我們影像中的主要物體，如熊或貓。

But they do not describe all the rich, fine-grained details inside the image.

但它們並不能描述影像內部所有豐富、精細的細節。

Natural language, like captions or descriptions, can give us much more information.

自然語言，如標題或描述，可以為我們提供更多的資訊。

For example, a caption might say, a friendly bear waves hello, or a cat reclines on the pavement.

例如，標題可以是 "一隻友好的小熊向你揮手問好 "或 "一隻貓躺在人行道上"。

This tells us about objects, their actions, and scene.

這告訴了我們關於物體、其動作和場景的資訊。

To use this extra information, we add a text encoder.

為了使用這些額外資訊，我們添加了一個文本編碼器。

The text encoder turns the caption or description into a feature vector, just like the image encoder does for the image.

文本編碼器會將標題或說明轉化為特徵向量，就像影像編碼器處理影像一樣。

Intuitively, we want the feature vectors for the image and the feature vector for its matching caption to be as similar as possible.

直觀地說，我們希望影像的特徵向量和與其匹配的標題的特徵向量儘可能相似。

This means the model learns that the image and its captions are related.

這意味著該模型可以學習到影像及其標題是相關的。

At the same time, we want the image and text features for pairs that do not match to be far apart.

與此同時，我們希望不匹配的影像和文本特徵對之間相距甚遠。

Let's visualize the image and text features as 2D vectors.

讓我們將影像和文字特徵可視化為二維向量。

We normalize them so that they have unit lengths.

我們將它們歸一化，使它們具有組織、部門長度。

We measure similarity by looking at the angle between the two vectors.

我們通過觀察兩個向量之間的角度來衡量相似度。

Since the vectors are normalized, the dot product directly reflects how similar they are.

由於矢量已歸一化，點積直接反映了它們的相似程度。

The closer the dot product is to 1, the smaller the angle and the more similar the features.

點積越接近 1，角度越小，特徵越相似。

This is what we want to see after training.

這就是我們希望在訓練後看到的結果。

Along the diagonal, the features from matching image-text pairs are well aligned.

沿著對角線，匹配影像和文本的特徵對齊。

Off the diagonal, the features from non-matching pairs become dissimilar.

在對角線外，非匹配對的特徵變得不相似。

This is known as contrastive language image pre-training, or CLIP.

這就是所謂的對比語言影像預訓練，簡稱 CLIP。

By training on 400 million image-text pairs, the image encoder learns to capture and generalize a wide range of visual concepts.

通過對 4 億對影像和文本進行訓練，影像編碼器學會了捕捉和概括各種視覺概念。

But this type of training is expensive because it requires human annotation for every image.

但這種訓練方式成本高昂，因為每張影像都需要人工標註。

Can we train our image encoder without any labels, using only images?

我們能否在沒有任何標籤的情況下，僅使用影像來訓練我們的影像編碼器？

If we can achieve this, we can leverage the enormous amount of unlabeled images already available to train our models at scale.

如果能做到這一點，我們就能利用已有的大量未標記影像來大規模訓練我們的模型。

This is the goal of self-supervised learning.

Just like supervised learning, we extract image features, add an output head, and use a loss function.

就像監督學習一樣，我們提取影像特徵，添加輸出頭，並使用損失函數。

The challenge here is to create a supervision signal for training, but without relying on any human-provided labels.

這裡的挑戰是創建一個用於訓練的監督信號，但不依賴於任何人類提供的標籤。

Over the years, researchers have developed creative strategies for self-supervised learning.

多年來，研究人員為自我監督學習開發了創造性的策略。

For example, we can train the model to colorize an image, predict the rotation angle of an image, impan the center patch from the surrounding context, or more generally, predict missing local patches from randomly masked input images.

例如，我們可以訓練模型對影像進行著色、預測影像的旋轉角度、從周圍環境中提取中心斑塊，或者更廣泛地說，從隨機屏蔽的輸入影像中預測缺失的局部斑塊。

However, training on these manually designed tasks does not reinforce learning representations that are semantically rich or transferable.

然而，對這些人工設計的任務進行訓練並不能強化語義豐富或可遷移的學習表徵。

Let's revisit the contrastive learning idea.

讓我們重溫一下對比學習的理念。

We extract image features from two different augmented views of input images.

我們從輸入影像的兩種不同增強視圖中提取影像特徵。

The objective is to bring the embedding vectors from matching pairs closer, while pushing embeddings of different images far apart.

這樣做的目的是拉近匹配影像對的嵌入向量，同時將不同影像的嵌入向量推遠。

More specifically, we compute the pairwise similarity of all pairs using vector dot product.

更具體地說，我們使用向量點積計算所有配對的相似性。

Here, Sij denotes the similarity score between i's embedding vector and j's embedding vector.

這裡，Sij 表示 i 的嵌入向量與 j 的嵌入向量之間的相似度得分。

We highlight positive pairs with yellow boxes, and negative pairs with blue boxes.

我們用黃色方框標出積極配對，用藍色方框標出消極配對。

During training, the model learns to increase the similarity for positive pairs.

在訓練過程中，模型會學習提高正向配對的相似度。

Intuitively, we train the model to extract feature representations that are stable under different augmented views.

直觀地說，我們對模型進行訓練，以提取在不同增強視圖下穩定的特徵表徵。

When contrastive learning has been successful, making it work is challenging.

當對比學習取得成功時，要使其發揮作用則具有挑戰性。

The main issue is that it needs a large number of negative samples to avoid learning trivial representations.

主要問題在於，它需要大量的負樣本來避免學習瑣碎的表徵。

As a result, training requires very large batch size and is computationally expensive.

是以，訓練需要很大的批量規模，計算成本很高。

To tackle this challenge, let's look at how it could work with a single image.

為了應對這一挑戰，讓我們來看看如何使用單張影像。

Given an input image, we create two augmented views like different crops.

給定一張輸入影像，我們會創建兩個不同作物的增強視圖。

We call the image encoders a student and teacher models.

我們將影像編碼器分別稱為學生模型和教師模型。

We extract features, feed them into a projection head to obtain logics, and then apply a softmax function to produce probability distributions.

我們提取特徵，將其輸入投影頭以獲得邏輯，然後應用軟最大函數生成概率分佈。

With this setup, we train the student network to match the teacher's predictions by minimizing the cross entropy between them.

在這種設置下，我們通過最小化兩者之間的交叉熵來訓練學生網絡，使其與教師的預測相匹配。

A key detail is that we stop the gradient from flowing into the teacher network during training.

一個關鍵的細節是，我們在訓練過程中阻止梯度流入教師網絡。

In this setup, only the student network's parameters are updated to match the teacher's predictions, while teachers provide a stable target.

在這種設置中，只更新學生網絡的參數以匹配教師的預測，而教師則提供一個穩定的目標。

This approach is known as knowledge distillation.

Typically, we use it to distill knowledge from a powerful large teacher model to a smaller student model, making inference more efficient.

通常情況下，我們利用它將知識從強大的大型教師模型提煉到較小的學生模型，從而提高推理效率。

In our case, both student and teacher models have the same architectures and model size.

在我們的案例中，學生模型和教師模型具有相同的架構和模型大小。

First, if we don't have labels from people, how do we know what a class is?

首先，如果沒有人的標籤，我們怎麼知道什麼是 "類"？

Fortunately, we can define the output dimension to be a large number like 65,000.

幸運的是，我們可以將輸出維度定義為 65,000 這樣的大數字。

This enables the model to discover and represent a wide range of visual concepts without requiring us to define them in advance.

這樣，模型就能發現和表現各種視覺概念，而不需要我們事先對其進行定義。

Second, in knowledge distillation, the teacher model is usually already trained.

其次，在知識提煉方面，教師模型通常已經過培訓。

But without label data, how do we get this teacher model?

但是，沒有標籤數據，我們如何獲得這種教師模型呢？

One effective method is to update the teacher model's weight as a moving average of the student's weight over time.

一種有效的方法是將教師模型的權重更新為學生權重隨時間變化的移動平均值。

This allows the teacher model to change gradually, providing a stable and consistent target for the student to learn from.

這樣，教師的教學模式就可以逐步改變，為學生提供一個穩定而一致的學習目標。

Specifically, after each training step, we update the teacher's parameters using the following formula.

具體來說，在每個訓練步驟之後，我們都會使用以下公式更新教師參數。

The value lambda controls the update rate for the teacher model.

lambda 值控制著教師模型的更新率。

In practice, we set the initial value of lambda as 0.996 and gradually increase it over the course of training.

在實踐中，我們將 lambda 的初始值設定為 0.996，並在訓練過程中逐步提高。

However, this approach alone is not enough to prevent the model from collapsing to trivial solutions.

然而，僅靠這種方法還不足以防止模型坍縮為瑣碎的解。

For example, the model could end up producing outputs where a single dimension dominates the distributions, resulting in uninformative or collapsed representations.

例如，模型最終可能會產生單一維度主導分佈的輸出結果，從而導致信息量不足或表述混亂。

To address this, we use a trick called centering.

為了解決這個問題，我們使用了一種叫做居中的技巧。

Let's illustrate how it works with an example.

Suppose we have a batch of four samples, and our model predicts probabilities over three classes for each sample.

假設我們有一批四個樣本，我們的模型預測每個樣本三個類別的概率。

After applying softmax, most of the probability mass is concentrated in the first dimension.

應用 softmax 後，大部分概率品質都集中在第一維度。

Next, we calculate the average logic, the center, across the batch, and subtract the center from each of the teacher's logics.

接下來，我們計算出整批學生的平均邏輯思維，即中心邏輯思維，然後從每位教師的邏輯思維中減去中心邏輯思維。

The centering trick encourages the model to spread its predictions more evenly, rather than focusing on a single output.

居中訣竅鼓勵模型更均勻地分散其預測，而不是專注於單一輸出。

Centering helps prevent collapse by ensuring the average output of the teacher network remains balanced across all output dimensions.

通過確保教師網絡的平均輸出在所有輸出維度上保持平衡，中心定位有助於防止崩潰。

Instead of computing the center vector for each batch independently, we maintain a moving average of the center over time.

我們不再獨立計算每個批次的中心向量，而是保持中心隨時間變化的移動平均值。