Name: DeepSeek是在 "抄襲 "美國嗎？ 它的出現對中美AI競爭的影響 DeepSeek ｜ LLM ｜Open AI ｜ 中國 ｜美國 ｜人工智能競爭 ｜開源模型 20250126金融汪 (DeepSeek是在“抄袭”美国吗？它的出现对中美AI竞争的影响 DeepSeek ｜ LLM ｜Open AI ｜ 中国 ｜美国 ｜人工智能竞争 ｜开源模型 20250126金融汪)
Uploaded: 2025-01-29T01:04:10.000Z
Duration: 20 min 9 s

the AI race between China and the U.S. and what's at stake?

中國和美國之間的人工智能競賽及其利害關係？

動名詞

Okay so first of all China has a lot of disadvantages in competing with the U.S.

首先，中國在與美國競爭時有很多劣勢。

Number one is the fact that they don't get access to all the hardware that we have access to here.

第一，他們無法使用我們這裡的所有硬件。

So they're kind of working with lower-end GPUs than us.

是以，他們使用的 GPU 比我們低端。

It's almost like working with the previous generation GPUs crappily.

這幾乎就像是在蹩腳地使用上一代 GPU。

So and the fact that the bigger models tend to be more smarter naturally puts them at a disadvantage.

是以，更大的機型往往更聰明，這自然使它們處於劣勢。

But the flip side of this is that necessity is the mother of invention.

但反過來說，必要是發明之母。

Because they had to go figure out workarounds, they actually ended up building something a lot more efficient.

因為他們不得不去想變通的辦法，所以實際上他們最終建造的東西效率要高得多。

It's like saying, hey look you guys really got to get a top-notch model and I'm not going to give you resources.

這就好比說，嘿，你們真的需要一個頂級模特，但我不會給你們資源。

Unless it's impossible, unless it's mathematically possible to prove that it's impossible to do so, you can always try to like come up with something more But that is likely to make them come up with a more efficient solution than America.

除非這是不可能的，除非在數學上可以證明這是不可能的，否則你總可以試著想出一些更有可能讓他們想出比美國更有效解決方案的辦法。

And of course they've open sourced it so we can still adopt something like that here.

當然，他們已經將其開源，所以我們仍然可以在這裡採用類似的東西。

But that kind of talent they're building to do that will become an edge for them over time, right?

但他們正在培養的這種人才，隨著時間的推移，會成為他們的優勢，對嗎？

The leading open source model in America is Meta's Lama family.

美國領先的開源模式是 Meta 的喇嘛家族。

It's kind of like a model that you can run on a computer.

它有點像一個可以在電腦上運行的模型。

But even though it got pretty close to GPT-4 and Asana at the time of its release, the model that was closest in quality was a giant 405B, not the 70B that you could run on your computer.

不過，儘管它在發佈時已經非常接近 GPT-4 和 Asana，但在品質上最接近的型號是巨大的 405B，而不是可以在電腦上運行的 70B。

And so that was still not a small, cheap, fast, efficient, open source model that rivaled the most powerful closed models from Okinawa and Tropic.

是以，這仍然不是一個小型、廉價、快速、高效的開源模型，無法與沖繩和熱帶最強大的封閉模型相媲美。

And then these guys come out with like a crazy model that's like 10x cheaper in API pricing than GPT-40 and 15x cheaper than Sonnet, I believe.

然後，這些傢伙又推出了一個瘋狂的型號，其 API 價格比 GPT-40 便宜 10 倍，比 Sonnet 便宜 15 倍。

And pretty much equal or better in some benchmarks than worse than some others, but like roughly in that ballpark of 4OS quality.

在某些基準測試中，它的性能相當或更好，而在其他一些基準測試中，它的性能則較差，但大致與 4OS 的品質相當。

And they did it all with like approximately just 2048H800 GPUs, which is actually equivalent to like somewhere around 1500 or 1500H100 GPUs.

他們使用大約 2048H800 GPU 完成了這一切，這實際上相當於大約 1500 或 1500H100 GPU。

That's like 20 to 30x lower than the amount of GPUs that GPT-4 is usually trained on.

這比 GPT-4 通常使用的 GPU 數量要低 20 到 30 倍。

And they did roughly $5 million in total compute budget.

他們的計算預算總額約為 500 萬美元。

They did it with so little money and such an amazing model.

他們用如此少的錢和如此出色的模型做到了這一點。

And definitely it makes us all question like, okay, like if we have the equivalent of DOGE for like model training, this is an example of that, right?

這肯定會讓我們產生這樣的疑問：好吧，如果我們有等同於 DOGE 的模型訓練方法，這不就是一個例子嗎？

So fraction of the price, fraction of the time, dumbed down GPUs essentially.

是以，只需幾分之一的價格，幾分之一的時間，基本上就是傻瓜式 GPU。

What was your surprise when you understood what they had done?

當你明白他們的所作所為時，你有什麼驚訝？

So my surprise was that when I actually went through the technical paper, the amount of clever solutions they came up with.

是以，讓我感到驚訝的是，當我真正翻閱技術文件時，他們提出了大量巧妙的解決方案。

First of all, they trained a experts model.

首先，他們訓練了一個專家模型。

There's a lot of like, the main reason people find it difficult to catch up with OpenAI, especially the MOE architecture, is that there's a lot of irregular loss spikes.

人們發現 OpenAI（尤其是 MOE 架構）難以追趕的主要原因是，有很多不規則的損失峰值。

So often like you've got to restart the training checkpoint again.

所以，你經常要重新開始訓練檢查點。

And a lot of infrastructure needs to be built for that.

為此需要建設大量的基礎設施。

And they came up with very clever solutions to balance that without adding additional hacks.

他們想出了非常巧妙的辦法，在不增加額外黑客的情況下實現了平衡。

And they also figured out 8-bit training, at least for some of the numerics.

他們還想出了 8 位訓練方法，至少對某些數字進行了訓練。

And they cleverly figured out which has to be in higher precision, which has to be in lower precision.

他們巧妙地算出了哪個精度要高，哪個精度要低。

And to my knowledge, I think floating point 8 training is not that well understood.

據我所知，我認為浮點 8 訓練還沒有得到很好的理解。

Most of the training in America is still running.

And maybe OpenAI and some people are trying to explore that, but it's pretty difficult to get it right.

也許 OpenAI 和一些人正在嘗試探索這一點，但要做到這一點非常困難。

So because necessities, I'm going to mention, because they don't have that much memory, that many GPUs, they figured out a lot of numerical stability stuff that makes their work.

是以，我想說的是，因為他們沒有那麼多內存和 GPU，所以他們想出了很多數值穩定性的方法，使他們的工作變得更加穩定。

And they claimed in the paper that the majority of the training was stable, which means they can always rerun those training runs again on more data or better data.

他們在論文中聲稱，大部分訓練是穩定的，這意味著他們隨時可以在更多數據或更好的數據上重新運行這些訓練運行。

Usually the wisdom, or I would say the myth, is that Chinese are just good at copying.

通常，人們認為中國人只是善於模仿，或者說是神話。

So if we stop writing research papers in America, if we stop describing the details of our infrastructure, our architecture, and stop open sourcing, they're not going to be able to catch up.

是以，如果我們停止在美國撰寫研究論文，如果我們停止描述我們的基礎設施和架構的細節，停止開放源代碼，他們將無法迎頭趕上。

But the reality is, some of the details in DeepSea 3 are so good that I wouldn't be surprised if Meta took a look at it and incorporated some of that in Llamas 4.

但實際情況是，《深海 3》中的一些細節非常出色，如果 Meta 借鏡了《深海 3》，並在《傲骨賢妻 4》中加入一些細節，我也不會感到驚訝。

但問題是，情況正在發生變化。

We don't know exactly the data that it was trained on, even though it's open source.

儘管它是開源的，但我們並不清楚其訓練所依據的數據。

We know some of the ways and things it was trained on, but not everything.

我們知道它的一些訓練方式和訓練內容，但不是全部。

And there's this idea that it was trained on public chat GPT outputs, which would mean it just was copied.

還有一種說法是，它是在公開哈拉 GPT 輸出上訓練出來的，這意味著它只是被複制了。

Yeah, look, they've trained it on 14.8 trillion tokens.

是啊，你看，他們已經在 14.8 萬億個代幣上進行了訓練。

If you actually go to any LinkedIn post or X post now, most of the comments are written by AI.

如果你現在真的去看 LinkedIn 上的帖子或 X 帖子，大部分評論都是人工智能寫的。

Even with an X, there's a Grok tweet enhancer.

即使是 X，也有一個 Grok 鳴叫增強器。

或者在 LinkedIn 中，有一個人工智能增強器。

Or in Google Docs and Word, there are AI tools to rewrite your stuff.

或者在谷歌文檔和 Word 中，也有人工智能工具來改寫你的內容。

So if you do something there and copy-paste it somewhere on the internet, it's naturally going to have some elements of a chat GPT-like training.

是以，如果你在那裡做了一些事情，然後複製粘貼到互聯網上的某個地方，自然會有一些類似哈拉 GPT 培訓的元素。

And there's a lot of people who don't even bother to strip away that I'm a language model part.

有很多人甚至懶得剝去我是語言模型的部分。

所以他們就把它粘貼在某個地方。

So I wouldn't disregard their technical accomplishment just because for some prompts, like, who are you?

是以，我不會因為某些提示，比如你是誰？

Or which model are you in response to that?

For a long time, we thought, I don't know if you agreed with us, China was behind in AI.

在很長一段時間裡，我們都認為--我不知道你是否同意--中國在人工智能方面落後了。

Can we say that China is catching up or hasn't caught up?

我們能說中國正在追趕還是沒有追趕嗎？

I mean, if we say the meta is catching up to open AI and entropic, if you make that claim, then the same claim can be made for China catching up to America.

我的意思是，如果我們說元語言正在追趕開放式人工智能和熵，如果你這麼說，那麼同樣的說法也可以適用於中國追趕美國。

A lot of papers from China that have tried to replicate O1.

In fact, I saw more papers from China after O1 announcement that tried to replicate it than from America.

事實上，在 O1 公佈後，我看到中國試圖複製 O1 的論文比美國還多。

And the amount of compute DeepSeq has access to is roughly similar to what PhD students in the US have access to.

DeepSeq 所能使用的計算量與美國博士生所能使用的計算量大致相當。

This is not meant to criticize others, even for ourselves.

這並不是要責備別人，即使是責備我們自己。

For Perplexity, we decided not to train models because we thought it's a very expensive thing.

對於 Perplexity，我們決定不訓練模型，因為我們認為這是一件非常昂貴的事情。

And we thought there's no way to catch up with the rest.

而我們卻認為沒有辦法趕上其他人。

Will you incorporate DeepSeq into Perplexity?

你們會將 DeepSeq 納入 Perplexity 嗎？

I assume they have an API and they have open source of AI so we can host it ourselves too.

我想他們應該有一個應用程序接口，而且他們還開放了人工智能的源代碼，這樣我們也可以自己託管。

And it's good to try to start using that because it actually allows us to do a lot of the things at lower cost.

嘗試開始使用這種方法是件好事，因為它實際上可以讓我們以更低的成本完成很多事情。

But what I'm kind of thinking is beyond that, it's just like, okay, these guys actually could train such a great model with a good team.

但我的想法是，除了這一點之外，這些人其實還可以和一支優秀的團隊一起訓練出如此出色的模型。

And there's no excuse anymore for companies in the US, including ourselves, to not try to do something like that.

包括我們自己在內的美國公司再也沒有藉口不嘗試這樣做了。

You hear a lot in public from a lot of thought leaders and generative AI, on the research side, on the entrepreneurial side.

在公開場合，你會聽到很多思想領袖和生成式人工智能的觀點，無論是在研究方面，還是在創業方面。

Elon Musk and others say that China can't catch up.

埃隆-馬斯克等人說，中國趕不上。

DeepSeek是在 "抄襲 "美國嗎？ 它的出現對中美AI競爭的影響 DeepSeek ｜ LLM ｜Open AI ｜ 中國 ｜美國 ｜人工智能競爭 ｜開源模型 20250126金融汪 (DeepSeek是在“抄袭”美国吗？它的出现对中美AI竞争的影响      DeepSeek ｜ LLM ｜Open AI ｜ 中国 ｜美国 ｜人工智能竞争 ｜开源模型 20250126金融汪)

字幕列表影片播放

stuff

essentially

extremely

figure