Name: 變壓器詳解了解 GPT、BERT 和 T5 背後的模型 (Transformers, explained: Understand the model behind GPT, BERT, and T5)
Uploaded: 2025-07-07T11:48:27.000Z
Duration: 9 min 11 s

程度副詞

The neat thing about working in machine learning is that every few years somebody invents something crazy that makes you totally reconsider what's possible.

在機器學習領域工作的好處在於，每隔幾年就會有人發明出一些瘋狂的東西，讓你完全重新考慮什麼是可能的。

Like models that can play Go or generate hyper-realistic faces.

比如可以下圍棋或生成超逼真人臉的模型。

And today the mind-blowing discovery that's rocking everyone's world is a type of neural network called a transformer.

今天，震驚世界的重大發現是一種名為 "變壓器 "的神經網絡。

Transformers are models that can translate text, write poems in op-eds, and even generate computer code.

變形金剛是一種可以翻譯文字、在專欄文章中寫詩，甚至生成計算機代碼的模型。

These have been used in biology to solve the protein folding problem.

這些技術已在生物學中用於解決蛋白質摺疊問題。

Transformers are like this magical machine learning hammer that seems to make every problem into a nail.

變形金剛就像一把神奇的機器學習錘，似乎能把所有問題都變成釘子。

If you've heard of the trendy new ML models BERT or GPT-3 or T5, all of these models are based on transformers.

如果您聽說過時髦的新型 ML 型號 BERT 或 GPT-3 或 T5，那麼所有這些型號都是基於變壓器的。

So if you want to stay hip in machine learning and especially in natural language processing, you have to know about the transformer.

是以，如果你想在機器學習，尤其是自然語言處理領域保持領先地位，就必須瞭解變壓器。

So in this video I'm going to tell you about what transformers are, how they work, and why they've been so impactful.

是以，在這段視頻中，我將向大家介紹什麼是變壓器、變壓器是如何工作的，以及為什麼變壓器具有如此大的影響力。

It's a type of neural network images, videos, audio, and text.

它是一種神經網絡影像、視頻、音頻和文本。

But there are different types of neural networks optimized for different types of data.

但是，針對不同類型的數據，有不同類型的神經網絡進行了優化。

Like if you're analyzing images, you'd typically use a convolutional neural network, which is designed to vaguely mimic the way that the human brain processes vision.

比如，如果你要分析影像，通常會使用卷積神經網絡，其設計目的是模糊模仿人腦處理視覺的方式。

And since around 2012, neural networks have been really good at solving vision tasks, like identifying objects in photos.

大約從 2012 年開始，神經網絡在解決視覺任務（如識別照片中的物體）方面大顯身手。

But for a long time, we didn't have anything comparably good for analyzing language, whether for translation or text summarization or text generation.

但長期以來，無論是翻譯、文本摘要還是文本生成，我們都沒有類似的語言分析工具。

And this is a problem because language is the primary way that humans communicate.

這是一個問題，因為語言是人類交流的主要方式。

You see, until transformers came around, the way we use deep learning to understand text was with a type of model called a recurrent neural network, or an RNN, that looks something like this.

要知道，在變形金剛出現之前，我們使用深度學習理解文本的方式是使用一種叫做遞歸神經網絡（或 RNN）的模型，它看起來就像這樣。

Let's say you wanted to translate a sentence from English to French.

假設您想將一個句子從英語翻譯成法語。

An RNN would take as input an English sentence and process the words one at a time and then sequentially spit out their French counterparts.

RNN 將英語句子作為輸入，一次處理一個單詞，然後按順序吐出對應的法語單詞。

In language, the order of words matters, and you can't just shuffle them around.

在語言中，詞語的順序很重要，不能隨意更改。

For example, the sentence, Jane went looking for trouble, means something very different than the sentence, trouble went looking for Jane.

例如，"簡去找麻煩了 "與 "麻煩去找簡了 "這兩個句子的意思截然不同。

So any model that's going to deal with language has to capture word order, and recurrent neural networks do this by looking at one word at a time sequentially.

是以，任何處理語言的模型都必須捕捉詞序，而遞歸神經網絡是通過一次連續查看一個詞來實現這一點的。

First, they never really did well at handling large sequences of text, like long paragraphs or essays.

首先，他們在處理大段文字（如長段落或文章）時總是不盡如人意。

By the time they were analyzing the end of a paragraph, they'd forget what happened in the beginning.

當他們分析一個段落的結尾時，他們會忘記開頭髮生了什麼。

And even worse, RNNs were pretty hard to train.

Because they process words sequentially, they couldn't paralyze well, which means that you couldn't just speed them up by throwing lots of GPUs at them.

由於它們是按順序處理單詞的，是以無法很好地癱瘓，這意味著你無法通過向它們投擲大量 GPU 來加快它們的速度。

And when you have a model that's slow to train, you can't train it on all that much data.

而當你的模型訓練速度很慢時，你就無法用那麼多數據來訓練它。

This is where the transformer changed everything.

They're a model developed in 2017 by researchers at Google and the University of Toronto, and they were initially designed to do translation.

它們是由谷歌和多倫多大學的研究人員於 2017 年開發的一種模型，最初是為了進行翻譯而設計的。

But unlike recurrent neural networks, you could really efficiently paralyze transformers.

但與遞歸神經網絡不同的是，你可以真正有效地癱瘓變壓器。

And that meant that with the right hardware, you could train some really big models.

這意味著，只要有合適的硬件，就能訓練出一些真正的大型模型。

Remember GPT-3, that model that writes poetry and code and has conversations?

還記得那個會寫詩、寫代碼和進行對話的模型 GPT-3嗎？

That was trained on almost 45 terabytes of text data, including like almost the entire public web.

這是在近 45 TB 的文本數據上進行的訓練，其中包括幾乎整個公共網絡。

So if you remember anything about transformers, let it be this.

如果你對變形金剛還有什麼印象，那就記住它吧。

Combine a model that scales really well with a huge data set, and the results will probably blow your mind.

將擴展性極佳的模型與龐大的數據集相結合，結果可能會讓你大吃一驚。

那麼，這些東西究竟是如何工作的呢？

From the diagram of the paper, it should be pretty clear.

從論文的圖表來看，應該很清楚了。

Actually, it's simpler than you might think.

事實上，這比你想象的要簡單得多。

There are three main innovations that make this model work so well.

這一模式之所以如此成功，主要有三個創新點。

Positional encodings and attention, and specifically a type of attention called self-attention.

位置編碼與注意力，特別是一種名為自我注意力的注意力。

Let's start by talking about the first one, positional encodings.

我們先來談談第一種，位置編碼。

Let's say we're trying to translate text from English to French.

假設我們要將文本從英語翻譯成法語。

Positional encodings is the idea that instead of looking at words sequentially, you take each word in your sentence, and before you feed it into the neural network, you slap a number on it.

位置編碼的概念是，在將句子中的每個單詞輸入神經網絡之前，先給它加上一個數字，而不是按順序查看單詞。

One, two, three, depending on what number the word is in the sentence.

一、二、三，取決於單詞在句子中的數字。

In other words, you store information about word order in the data itself rather than in the structure of the network.

換句話說，您是在數據本身而不是網絡結構中存儲詞序資訊。

Then as you train the network on lots of text data, it learns how to interpret those positional encodings.

然後，當你在大量文本數據上訓練網絡時，它就會學會如何解釋這些位置編碼。

In this way, the neural network learns the importance of word order from the data.

這樣，神經網絡就能從數據中瞭解詞序的重要性。

This is a high-level way to understand positional encodings, but it's an innovation that really helped make transformers easier to train than RNNs.

這只是理解位置編碼的一種高層次方法，但這一創新確實讓變換器比 RNN 更容易訓練。

The next innovation in this paper is a concept called attention, which you'll see used everywhere in machine learning these days.

本文的下一個創新點是一個名為 "注意力 "的概念，你會發現如今機器學習中到處都在使用這個概念。

In fact, the title of the original transformer paper is attention is all you need.

事實上，原變壓器論文的標題就是關注就夠了。

So the agreement on the European economic area was signed in August 1992.

是以，1992 年 8 月簽署了歐洲經濟區協定。

That's the example sentence given in and remember, the original transformer was designed for translation.

這就是其中的例句，請記住，最初的變壓器是為翻譯而設計的。

Now imagine trying to translate that sentence to French.

現在試著把這句話翻譯成法語。

One bad way to translate text is to try to translate each word one for one.

翻譯文本的一個糟糕方法是嘗試逐字翻譯。

但在法語中，有些單詞是翻轉的。

Like in the French translation, European comes before economic.

與法文譯文一樣，"European"（歐洲）位於 "Economic"（經濟）之前。

Plus French is a language that has gendered agreement between words.

另外，法語是一種單詞之間有性別區分的語言。

So the word European needs to be in the feminine form to match with la zone.

是以，"European"（歐洲）一詞需要使用女性形式，以便與 "la zone"（區域）相匹配。

The attention mechanism is a word in the original sentence when making a decision about how to translate a word in the output sentence.

在決定如何翻譯輸出句中的某個單詞時，注意力機制就是原句中的某個單詞。

In fact, here's a nice visualization from that paper that shows what words in the input sentence the model is attending to when it makes predictions about a word for the output sentence.

事實上，這篇論文中有一個很好的可視化圖表，顯示了模型在預測輸出句中的某個單詞時，會關注輸入句中的哪些單詞。

So when the model outputs the word European, it's looking at the input words European and economic.

是以，當模型輸出 European（歐洲）一詞時，它正在研究 European（歐洲）和 economic（經濟）這兩個輸入詞。

You can think of this diagram as a sort of heat map for attention.

你可以把這張圖看作是一種注意力熱圖。

And how does the model know which words it should be attending to?

那麼，模型又是如何知道它應該關注哪些詞語的呢？

It's something that's learned over time from data.

這是隨著時間的推移從數據中總結出來的。

By seeing thousands of examples of French and English sentence pairs, the model learns about gender and word order and plurality and all of that grammatical stuff.

通過觀察成千上萬的法語和英語句子，該模型可以瞭解性別、詞序、複數和所有文法知識。

So we talked about two key transformer innovations, positional encoding and attention.

是以，我們談到了兩個關鍵的變壓器創新，即位置編碼和注意力。

But actually, attention had been invented before this paper.

但實際上，在這篇論文發表之前，注意力就已經被髮明出來了。

The real innovation in transformers was something called self-attention, a twist on traditional attention.

變壓器的真正創新在於一種叫做 "自我關注 "的東西，它是對傳統關注的一種創新。

The type of attention we just talked about had to do with aligning words in English and French, which is important for translation.

我們剛才談到的注意力類型與英語和法語中的單詞對齊有關，這對翻譯非常重要。

But what if you're just trying to understand the underlying meaning in language so that you can build a network that can do any number of language tasks?

但如果你只是想了解語言的基本含義，從而建立一個可以完成各種語言任務的網絡呢？

What's incredible about neural networks like transformers is that as they analyze tons of text data, they begin to build up this internal representation or understanding of language automatically.

像變形金剛這樣的神經網絡令人難以置信的地方在於，當它們分析大量文本數據時，會開始自動建立這種內部表徵或對語言的理解。

They might learn, for example, that the words programmer and software engineer and software developer are all synonymous.

例如，他們可能會知道程序員、軟件工程師和軟件開發人員都是同義詞。

And they might also naturally learn the rules of gender and tense and so on.

他們還可能自然而然地學會性別和時態等規則。

The better this internal representation of language the neural network learns, the better it will be at any language task.

神經網絡對語言的內部表徵學得越好，就越能勝任任何語言任務。

And it turns out that attention can be a very effective way to get a neural network to understand language if it's turned on the input text itself.

事實證明，如果把注意力放在輸入文本本身上，就能非常有效地讓神經網絡理解語言。

Versus looks like I just crashed the server.

Versus 看起來像是我把服務器搞崩潰了。

The word server here means two very different things.

這裡的服務器一詞有兩種截然不同的含義。

And I know that because I'm looking at the context of the Self-attention allows a neural network to understand a word in the context of the words around it.

我之所以知道這一點，是因為我正在研究 "自我關注 "的上下文，它能讓神經網絡在周圍詞語的上下文中理解一個詞語。

So when a model processes the word server in the first sentence, it might be attending to the word check, which helps it disambiguate from a human server versus a mental one.

是以，當模型處理第一句中的服務器一詞時，它可能會注意到單詞檢查，這有助於它區分是人類服務器還是心理服務器。

In the second sentence, the model might be attending to the word crashed to determine that the server is a machine.

在第二句中，模型可能會注意到 "crashed "這個詞，從而確定服務器是一臺機器。

Self-attention can also help neural networks disambiguate words, recognize parts of speech, and even identify word tense.

自我注意還能幫助神經網絡辨別單詞、識別語篇，甚至識別單詞時態。

This in a nutshell is the value of self-attention.

一言以蔽之，這就是自我關注的價值所在。

So to summarize, transformers boil down to positional encodings, attention, and self-attention.

是以，概括地說，變壓器可以歸結為位置編碼、注意力和自我注意力。

Of course, this is a 10,000 foot look at transformers, but how are they actually useful?

當然，這只是對變壓器的萬丈高樓平地起，但它們究竟有什麼用呢？

One of the most popular transformer-based models is called BERT, which was invented just around the time that I joined Google in 2018.

最流行的一種基於變壓器的模型叫做 BERT，它的發明時間正好是我 2018 年加入谷歌的時候。

BERT was trained on a massive text corpus and has become this sort of general pocket knife for NLP that can be adopted to a bunch of different tasks, like text summarization, question answering, classification, and finding similar sentences.

BERT 是在一個龐大的文本語料庫中訓練出來的，它已成為 NLP 的一把萬能小刀，可用於各種不同的任務，如文本摘要、問題解答、分類和查找相似句子等。

It's used in Google Search to help understand search queries, and it powers a lot of Google Cloud's NLP tools, like Google Cloud AutoML Natural Language.

谷歌搜索使用它來幫助理解搜索查詢，它還為谷歌雲的許多 NLP 工具（如谷歌雲 AutoML 自然語言）提供動力。

BERT also proved that you could build very good models on unlabeled data, like text scraped from Wikipedia or Reddit.

BERT 還證明，你可以在未標記的數據上建立非常好的模型，比如從維基百科或 Reddit 搜刮來的文本。

This is called semi-supervised learning, and it's a big trend in machine learning right now.

這就是所謂的半監督學習，也是目前機器學習的一大趨勢。

So if I've sold you on how cool transformers are, you might want to start using them in your app.

如果我已經向你介紹了變壓器有多酷，你可能就想在自己的應用程序中使用它們了。

字幕列表影片播放

變壓器詳解了解 GPT、BERT 和 T5 背後的模型 (Transformers, explained: Understand the model behind GPT, BERT, and T5)

sort

process

context

grab