Name: 但人工智能影像和視頻究竟是如何工作的？| 韋爾奇實驗室特邀視頻 (But how do AI images and videos actually work? | Guest video by Welch Labs)
Uploaded: 2025-09-27T00:44:38.000Z
Duration: 37 min 20 s

Over the last few years, AI systems have become astonishingly good at turning text prompts into videos.

在過去的幾年裡，人工智能系統在將文字提示轉化為視頻方面已經取得了驚人的成就。

At the core of how these models operate is a deep connection to physics.

這些模型運作的核心是與物理學的緊密聯繫。

關係子句

This generation of image and video models works using a process known as diffusion, which is remarkably equivalent to the Brownian motion we see as particles diffuse, but with time run backwards, and in high-dimensional space.

這種生成影像和視頻模型的方法採用了一種稱為 "擴散 "的過程，它與我們所看到的粒子擴散的布朗運動極為相似，但時間是倒流的，而且是在高維空間中。

As we'll see, this connection to physics is much more than a curiosity.

我們將看到，這種與物理學的聯繫遠不止是一種好奇。

We get real algorithms out of the physics that we can use to generate images and videos.

我們可以從物理學中獲得真正的算法，用來生成影像和視頻。

And this perspective will also give us some really nice intuitions for how these models work in practice.

這種視角還能為我們提供一些非常好的直覺，讓我們瞭解這些模型在實踐中是如何運作的。

But before we dive into this connection, let's get hands-on with a real diffusion model.

不過，在深入探討這種聯繫之前，讓我們先來親身體驗一下真正的擴散模型。

While the best models are closed-source, there are some compelling open-source models.

雖然最好的模式是閉源模式，但也有一些引人注目的開源模式。

This video of an astronaut was generated by an open-source model called WAN 2.1.

這段太空人的視頻是由一個名為 WAN 2.1 的開源模型生成的。

We can add to our prompt and have our astronaut hold a flag, hold a laptop, or hold a meeting.

我們可以根據提示，讓太空人手持國旗、筆記本電腦或召開會議。

If we cut down our prompt to just an astronaut, we get this.

如果我們把提示減少到只有一名太空人，就會得到這個結果。

And if we cut down our prompt to nothing, we interestingly still get this video of a  If we dig into our WAN model's source code, we'll find that the video generation process begins with this call to a random number generator.

有趣的是，如果我們把提示符減到零，我們仍然可以看到一段視頻。如果我們深入研究廣域網模型的源代碼，我們會發現視頻生成過程是從調用隨機數生成器開始的。

Creating a video where the pixel intensity values are chosen randomly.

創建一個隨機選擇像素強度值的視頻。

From here, this pure noise video is passed into a transformer.

從這裡開始，這段純噪聲視頻被送入變壓器。

This is the same type of AI model used by large language models like ChatGPT.

這與 ChatGPT 等大型語言模型所使用的人工智能模型類型相同。

But instead of outputting text, this transformer outputs another video that now looks like this.

但這個轉換器輸出的不是文字，而是另一段視頻，現在看起來是這樣的。

Still mostly noise, but with some hints of structure.

大部分還是噪音，但也有一些結構的影子。

This new video is added to our pure noise video, and then passed back into the model again, producing a third video that looks like this.

這段新視頻會被添加到我們的純噪聲視頻中，然後再次傳回模型中，產生如下所示的第三段視頻。

This process is repeated again and again.

Here's what the video looks like after 5 iterations, 10, 20, 30, 40, and finally 50.

以下是視頻迭代 5 次、10 次、20 次、30 次、40 次和 50 次後的效果。

Step by step, our transformer shapes pure noise into incredibly realistic video.

我們的變壓器一步一步地將純噪音轉化為無比逼真的視頻。

But what exactly is the connection to Brownian motion here?

但這裡與布朗運動到底有什麼聯繫呢？

And how is our model able to use text input so expressively to shape noise into what our prompt describes?

我們的模型又是如何如此有表現力地使用文本輸入，將噪音塑造成我們的提示所描述的內容呢？

In this video, we'll unpack diffusion models in three parts.

在本視頻中，我們將分三個部分解讀擴散模型。

First we'll look at a 2021 OpenAI paper and model called CLIP.

首先，我們來看看 2021 年 OpenAI 的一篇論文和一個名為 CLIP 的模型。

As we'll see, CLIP is really two models, a language model and a vision model, that are trained using a clever learning objective that allows them to learn this really powerful shared space between words and pictures.

正如我們將看到的，CLIP 實際上是兩個模型，一個語言模型和一個視覺模型，這兩個模型通過一個巧妙的學習目標進行訓練，使它們能夠學習文字和圖片之間這個非常強大的共享空間。

Experimenting with this space will help us get a feel for the high dimensional spaces that diffusion models operate in.

對這一空間進行試驗將有助於我們瞭解擴散模型運行的高維空間。

But learning a shared representation is not enough to generate images.

但是，僅僅學習共享表徵還不足以生成影像。

From here we'll look at the diffusion process itself.

下面我們來看看擴散過程本身。

At a high level, diffusion models are trained to remove noise from images or videos.

在高層次上，擴散模型經過訓練後可以去除影像或視頻中的噪音。

However, if you dig into the landmark papers in the field, you'll find that this naive understanding of diffusion really doesn't hold up in practice.

然而，如果你深入研究該領域具有里程碑意義的論文，就會發現這種對擴散的天真理解在實踐中根本站不住腳。

In this section, we'll dig into the connection between diffusion models and diffusion processes in physics.

在本節中，我們將深入探討物理學中的擴散模型與擴散過程之間的聯繫。

This connection will help us understand how these models really work in practice and give us some powerful theory for dramatically speeding up image and video generation.

這種聯繫將幫助我們瞭解這些模型在實踐中的真正工作原理，併為我們提供一些強大的理論，以顯著加快影像和視頻的生成速度。

Finally, we'll bring these worlds together and see how approaches like CLIP are combined with diffusion models to condition and guide the generation process towards the videos we asked for in our props. 2020 was a landmark year for language modeling.

最後，我們將把這些世界融合在一起，看看 CLIP 等方法是如何與擴散模型相結合，以調節和引導生成過程，實現我們在道具中要求的視頻。2020 年是語言建模具有里程碑意義的一年。

New results in neural scaling laws and OpenAI's GPT-3 showed that bigger really was better.

神經縮放定律和 OpenAI 的 GPT-3 的新成果表明，越大越好。

Massive models trained on massive datasets had capabilities that simply didn't exist in smaller models.

在海量數據集上訓練出來的大型模型具有小型模型所不具備的功能。

It didn't take long for researchers to apply similar ideas to images.

沒過多久，研究人員就把類似的想法應用到了影像上。

In February 2021, a team at OpenAI released a new model architecture called CLIP, trained on a dataset of 400 million image and caption pairs scraped from the internet.

2021 年 2 月，OpenAI 的一個團隊發佈了一個名為 CLIP 的新模型架構，該架構是在一個由從互聯網上搜羅的 4 億張圖片和標題對組成的數據集上訓練出來的。

CLIP is composed of two models, one that processes text and one that processes images.

CLIP 由兩個模型組成，一個處理文本，另一個處理影像。

The output of each of these models is a vector of length 512, and the central idea is that the vectors for a given image and its caption should be similar.

每個模型的輸出都是一個長度為 512 的向量，中心思想是給定影像及其標題的向量應該相似。

To achieve this, the OpenAI team developed a clever training approach.

為此，OpenAI 團隊開發了一種巧妙的培訓方法。

Given a batch of image-caption pairs, for example our batch could contain a picture of a cat, a dog, and me, with the captions a photo of a cat, a photo of a dog, and a photo of a man, we then pass our three images into our image model, and our three captions into our text model.

給定一批影像-標題對，例如我們的批次可能包含一張貓、一條狗和我的照片，標題是貓的照片、狗的照片和人的照片，然後我們將三張影像傳入影像模型，將三個標題傳入文本模型。

We now have three image vectors and three text vectors, and we would like the vectors for the matching image-caption pairs to be similar.

現在我們有三個影像向量和三個文本向量，我們希望影像和標題配對的向量是相似的。

The clever idea from here is to make use of the similarity not just between the corresponding images and captions, but between all image-caption pairs in the batch when training our models.

這裡的巧妙之處在於，在訓練我們的模型時，不僅要利用相應影像和標題之間的相似性，還要利用批次中所有影像-標題對之間的相似性。

If we arrange our image vectors as the columns of a matrix, and our text vectors as the rows, the pairs of vectors along the diagonal of our matrix correspond to matching images and captions, and all the pairs off-diagonal are non-matching images and captions.

如果我們將影像向量排列成矩陣的列，將文本向量排列成矩陣的行，那麼矩陣對角線上的向量對就對應於匹配的影像和標題，而對角線外的所有向量對則對應於不匹配的影像和標題。

The CLIP training objective seeks to maximize the similarity between corresponding image-caption pairs, while simultaneously minimizing the similarity between non-corresponding image-caption pairs.

CLIP 的訓練目標是最大化對應影像標題對之間的相似性，同時最小化非對應影像標題對之間的相似性。

The C in CLIP stands for contrastive, because the model learns to contrast matching and non-matching image-caption pairs.

CLIP 中的 "C "代表對比（contrastive），因為該模型學會對比匹配和不匹配的影像-標題對。

The CLIP algorithm measures similarity between vectors using a metric called cosine similarity.

CLIP 算法使用一種叫做餘弦相似度的指標來衡量向量之間的相似性。

Geometrically, we can think of each of these vectors as pointing in some direction in high-dimensional space.

從幾何學角度看，我們可以把這些向量看作是指向高維空間中的某個方向。

Cosine similarity measures the cosine of the angle between our vectors in this space.

餘弦相似度測量的是我們的向量在這個空間中的夾角餘弦。

So if our text and image vector point in the same direction, the angle between our vectors will be zero, resulting in a maximum value for our cosine similarity score of one.

是以，如果我們的文字和影像向量指向同一個方向，那麼向量之間的夾角就是零，餘弦相似性得分的最大值就是 1。

So the image and text models that make up CLIP are trained to maximize the alignment of related images and captions in this shared high-dimensional space, while minimizing the alignment between unrelated images and captions.

是以，組成 CLIP 的影像和文本模型經過訓練後，可以在這個共享的高維空間中最大限度地對齊相關影像和標題，同時最大限度地減少不相關影像和標題之間的對齊。

The learned geometry of this shared vector space, known as a latent or embedding space, has some really interesting properties.

這種共享向量空間被稱為潛空間或嵌入空間，它的學習幾何具有一些非常有趣的特性。

If I take two pictures of myself, one not wearing a hat and one wearing a hat, and pass both of these into our CLIP image model, we get two vectors in our embedding space.

如果我拍了兩張自己的照片，一張沒戴帽子，一張戴了帽子，把這兩張照片都傳入 CLIP 影像模型，我們就會在嵌入空間中得到兩個向量。

Now if I take the vector corresponding to me wearing a hat, and subtract the vector of me not wearing a hat, we get a new vector in our embedding space.

現在，如果我取下與我戴帽子相對應的向量，再減去我不戴帽子的向量，我們就會在嵌入空間中得到一個新的向量。

Now what text might this new vector correspond to?

現在，這個新矢量可能對應什麼文本？

Mathematically, we took the difference of me wearing a hat and me not wearing a hat.

我們用數學方法算出了我戴帽子和不戴帽子的差值。

We can search for corresponding text by passing a bunch of different words into our text encoder, and for each computing the cosine similarity between our newly computed difference vector and the text vector.

我們可以將一系列不同的單詞輸入文本編碼器，然後計算新計算出的差異向量與文本向量之間的餘弦相似度，從而搜索相應的文本。

Testing a set of a few hundred common words, the top ranked match with a similarity of 0.165 is the word hat, followed by cap and helmet.

在對一組幾百個常用詞進行測試後，相似度為 0.165 的匹配詞中，排名第一的是 hat，其次是 cap 和 helmet。

The learned geometry of CLIP's embedding space allows us to operate mathematically on the pure ideas or concepts in our images and text, translating the differences in the content of our images, like if there's a hat or not, into a literal distance between vectors in our embedding space.

通過 CLIP 嵌入空間的幾何知識，我們可以對影像和文本中的純粹想法或概念進行數學運算，將影像內容的差異（如是否有帽子）轉化為嵌入空間中向量之間的實際距離。

The OpenAI team showed that CLIP could produce very impressive image classification results by simply passing an image into our image encoder and then comparing the resulting vector to a set of possible captions, one for each label that could be assigned to the image, and classifying the image with whatever label resulted in the highest cosine similarity.

OpenAI 團隊證明，CLIP 可以生成非常令人印象深刻的影像分類結果，只需將影像傳入我們的影像編碼器，然後將生成的向量與一組可能的標題進行比較，每一個標籤都可以分配給影像，並將影像分類為餘弦相似度最高的標籤。

So techniques like CLIP give us a powerful shared representation of image and text, a kind of vector space of pure ideas.

是以，像 CLIP 這樣的技術為我們提供了一種強大的影像和文本共享表徵，一種純粹想法的矢量空間。

However, our CLIP models only go one direction.

然而，我們的 CLIP 模型只能朝一個方向發展。

We can only map image and text to our shared embedding space.

我們只能將影像和文本映射到我們的共享嵌入空間。

We have no way of generating images and text from our embedding vectors. 2020 turned out not only to be a transformative year for language modeling.

我們無法根據嵌入向量生成影像和文本。2020 年不僅是語言建模的變革之年。

A few weeks after the GPT-3 paper came out, a team at Berkeley published a paper called Denoising Diffusion Probabilistic Models, now known as DDPM.

GPT-3 論文發表幾周後，伯克利大學的一個團隊發表了一篇名為 "去噪擴散概率模型"（Denoising Diffusion Probabilistic Models）的論文，也就是現在的 DDPM。

The paper showed for the first time that it was possible to generate very high quality images using a diffusion process, where pure noise is transformed step-by-step into realistic images.

這篇論文首次表明，利用擴散過程可以生成非常高質量的影像，在這個過程中，純噪聲被逐步轉化為逼真的影像。

The core idea behind diffusion models is pretty straightforward.

擴散模型的核心思想非常簡單。

We take a set of training images and add noise to each image step-by-step until the image is completely destroyed.

我們獲取一組訓練影像，然後逐步向每幅影像添加噪聲，直到影像完全被破壞。

From here we train a neural network to reverse this process.

在此基礎上，我們訓練一個神經網絡來逆轉這一過程。

When I first learned about diffusion models, I assumed that the models would be trained to remove noise a single step at a time.

當我第一次瞭解擴散模型時，我以為這些模型經過訓練後可以一步一步地去除噪音。

Our model would be trained to predict the image in step 1, given the noisier image in step 2, trained to predict the image in step 2, given the noisier image in step 3, and so on.

我們將對模型進行訓練，以預測第 1 步中的影像，並給出第 2 步中更噪點的影像；再對模型進行訓練，以預測第 2 步中的影像，並給出第 3 步中更噪點的影像，以此類推。

When it came time to generate an image, we would pass pure noise into our model, take its output, and pass it back into its input again and again.

當需要生成影像時，我們會將純噪聲輸入模型，得到它的輸出，然後一次又一次地將它傳回輸入端。

And after enough steps, we would have a nice image.

經過足夠多的步驟後，我們就會得到一個漂亮的影像。

Now it turns out that this naive approach to building a diffusion model really does not work well.

現在事實證明，這種建立擴散模型的天真方法確實行不通。

Virtually no modern models work like this.

幾乎沒有任何現代模型能像這樣工作。

These are the training and image generation algorithms from the Berkeley team's paper.

這些是伯克利團隊論文中的訓練算法和影像生成算法。

The notation is a bit dense, but there are some key details we can pull out that will help us understand what it takes to make these models really work.

這些符號有點繁瑣，但我們可以從中找出一些關鍵細節，幫助我們理解如何才能讓這些模型真正發揮作用。

The first thing that surprised me is that the team added random noise to images not just during training, but also during image generation.

首先讓我感到驚訝的是，研究小組不僅在訓練過程中，而且在生成影像的過程中，在影像中添加了隨機噪聲。

Algorithm 2 tells us that when generating new images, at each step, after our neural network predicts a less noisy image, we need to add random noise to this image before passing it back into our model.

算法 2 告訴我們，在生成新影像時，每一步都需要在神經網絡預測出噪點較少的影像後，向該影像添加隨機噪點，然後再將其傳回模型。

This added noise turns out to matter a lot in practice.

在實際應用中，這種額外的噪音非常重要。

If we take a popular diffusion model like Stable Diffusion 2, and use the Berkeley team's image generation approach, known as DDPM sampling, we can get some really nice images.

如果我們採用像穩定擴散 2 這樣的流行擴散模型，並使用伯克利團隊的影像生成方法（即 DDPM 採樣），我們就能得到一些非常漂亮的影像。

Here's the image we get when prompting the model with this prompt, asking for a tree in the desert.

下面是我們在提示模型時得到的影像，要求在沙漠中找到一棵樹。

Now if we remove the line of code that adds noise at each step of the generation process, we end up with a tiny, sad, blurry tree.

現在，如果我們去掉在生成過程中每一步都添加噪音的那行代碼，我們就會得到一棵又小又可憐又模糊的樹。

How is it that adding random noise while generating images leads to better quality, sharper images?

為什麼在生成影像時添加隨機噪聲會使影像品質更高、更清晰？

The second thing that surprised me when I encountered the Berkeley team's approach was that the team wasn't training models to reverse a single step in the noise addition process.

伯克利團隊的方法讓我感到驚訝的第二點是，該團隊並沒有訓練模型來逆轉噪聲添加過程中的任何一個步驟。

Instead, the team takes an initial clean image, which they call X0, and adds scaled random noise to the image, which they call epsilon.

相反，研究小組先獲取一張乾淨的初始影像（他們稱之為 X0），然後在影像中添加一定比例的隨機噪音（他們稱之為ε）。

And from here, they train the model to predict the total noise that was added to the original image.

在此基礎上，他們對模型進行訓練，以預測添加到原始影像中的總噪聲。

So the team is effectively asking the model to skip all the intermediate steps, and make a prediction about the original image.

是以，研究小組實際上是要求模型跳過所有中間步驟，對原始影像進行預測。

Intuitively, this learning task seems much more difficult to me than just learning to make a noisy image slightly less noisy.

憑直覺，在我看來，這項學習任務要比學習如何讓嘈雜影像的嘈雜聲稍小一些困難得多。

The Berkeley team's paper and approach was a landmark result that put diffusion on the map.

伯克利團隊的論文和研究方法是一項具有里程碑意義的成果，它將擴散問題推上了風口浪尖。

Why does adding random noise while generating images, and training the model like this, work so well?

為什麼在生成影像時添加隨機噪音，並對模型進行這樣的訓練能取得如此好的效果？

The DDPM paper draws on some fairly complex theory to arrive at these algorithms.

DDPM 論文借鏡了一些相當複雜的理論來得出這些算法。

I'll include a link to a great tutorial in the description if you want to dig deeper into the theory.

如果您想深入瞭解相關理論，我將在說明中提供一個精彩教程的鏈接。

Happily, it turns out that there's a different but mathematically equivalent way of understanding what diffusion models are really learning, that we can use to get a visual and intuitive sense for why the DDPM algorithms work so well.

令人欣慰的是，我們發現了一種不同的、但在數學上等價的方法來理解擴散模型真正在學習什麼，我們可以利用這種方法來獲得一種形象直觀的感覺，瞭解為什麼 DDPM 算法如此有效。

The key will be thinking of diffusion models as learning a time-varying vector field.

關鍵在於將擴散模型視為學習時變向量場。

This perspective also leads to a more general approach, called flow-based models, which have become very popular recently.

這種觀點還引出了一種更普遍的方法，即最近非常流行的基於流量的模型。

To see how diffusion models learn this time-varying vector field, let's temporarily simplify our learning problem.

為了瞭解擴散模型是如何學習這個時變向量場的，讓我們暫時簡化一下我們的學習問題。