Name: GPU 與網路的AI硬體NPI集群加速驗證，效率大提升！ (Accelerating AI Hardware NPI Cluster less Validation of GPUs and Networking)
Uploaded: 2025-11-15T00:37:16.000Z
Duration: 19 min 15 s

情態助動詞 B1

We might be keeping you between lunch, so hopefully we don't drown on too much.

我們可能會佔用大家午餐的時間，所以希望我們不會講太久。

So we're going to Talk about validating GPUs and NICs or GPUs and networking or AI accelerators, depending on what you want to call them.

所以我們要來聊聊驗證 GPU 和 NIC，或是 GPU 和網路，或是 AI accelerators，看大家習慣怎麼稱呼它們。

XPUs, Broadcom, like CUs, without clusters, without.

XPUs、Broadcom，像是 CUs，沒有叢集，沒有。

Which is really hard because fundamentally you need a cluster.

這其實很難，因為基本上你需要一個叢集。

So this work is on behalf of what we've done at Meta, myself and a few of my colleagues.

所以這項工作是基於我們在 Meta 所做的成果，由我和我的幾位同事共同完成。

Arvind couldn't make it today, but there's multiple people behind us.

Arvind 今天無法前來，但背後還有許多人。

So I wanna thank everyone who contributed to this.

所以我想感謝所有對此做出貢獻的人。

So this is somewhat of a continuation from the talk we gave in 2023.

所以這其實是延續我們在 2023 年的演講內容。

We talked about evalu all to all collectives on our Zion systems.

我們談論了在 Zion systems 上進行 eval all to all collectives。

It was much harder than we thought when we started.

這比我們一開始想的還要困難許多。

It started with, okay, the performance isn't anywhere near roofline.

一開始是，好吧，效能離 roofline 還有好大一段距離。

Ultimately, we dug into the collective state machine, how the network was configured.

最終，我們深入研究了 collective state machine，以及網路是如何設定的。

We built a complex test bed with multiple racks.

我們建立了一個包含多個機架的複雜測試平臺。

In the end, you know, we had to drill into packet captures because we're getting switches from the signal or signals from our switches that indicated that the buffers are filling up.

最後，你知道，我們必須深入研究 packet captures，因為我們從 switches 那裡收到了訊號，顯示 buffers 正在填滿。

Incast was due to microbursts and ultimately it was very complicated.

Incast 是因為 microbursts 造成的，最後發現非常複雜。

So one of the call to actions there is how do we do it better?

所以其中一個 call to actions 就是我們該如何做得更好？

In that test bed, we were limited by a few racks and it was very hard to measure.

在那個測試平臺裡，我們受限於幾個機架，測量起來非常困難。

How can we go about this a different way?

我們有沒有別的方法可以處理這個問題？

So that's kind of the subject of this talk.

So there has to be a better way than we did last time.

所以我們上次的做法肯定有更好的方法。

所以我們內部就開始著手處理。

We collaborated with keysight and this is what we came up with.

我們跟 keysight 合作，然後就有了這個成果。

Like, normally in a collective's test, you have a bunch of nodes.

通常在 collective's test 中，你會有一堆 nodes。

By definition, the job of the collectives is to exchange data between them.

根據定義，collectives 的工作就是在它們之間交換資料。

But there's no reason why you have to simultaneously test every NIC and accelerator.

但沒有理由說你必須同時測試每一個 NIC 和 accelerator。

So the idea here is to focus on testing one NIC and one accelerator and mock everything else.

所以這裡的想法是專注於測試一個 NIC 和一個 accelerator，然後 mock 其他所有東西。

So imagine magical device that can behave just like your switch or just be like your rack.

所以想像一個神奇的裝置，它可以表現得就像你的 switch 或你的 rack 一樣。

Think of it as a hardware collectives emulator, or a traffic generator, whatever you want to call it.

你可以把它想成是一個硬體的 collectives emulator，或是一個 traffic generator，隨你怎麼稱呼它。

And instead of doing a cluster level test, you reduce your like sort of cluster under test to system under test, or even device under test.

然後，你不再做一個 cluster level test，而是將你的 cluster under test 縮減成 system under test，甚至 device under test。

And then there's some questions, like there's no perfect Answer to.

然後會有一些問題，像是沒有完美的答案。

It's like when I'm testing a NIC in an accelerator, do I also test the network, like the switch?

這就像我在測試一個 NIC 和一個 accelerator 時，我是否也需要測試網路，像是 switch？

The signaling from the switch is fundamentally part of it.

從 switch 來的訊號，基本上是它的一部分。

We're gonna both present data on a system under test with just mocking different endpoints and just a device under test, just the accelerator and the NIC by itself.

我們會同時展示一個 system under test 的數據，透過模擬不同的 endpoint，以及一個 device under test，也就是單獨的 accelerator 和 NIC。

And there's one other thing I wanna talk about.

One of the complexities that we talked about last time is how the collective communications library works.

我們上次提到的複雜性之一，是如何運作 collective communications library。

So, so we went over the collective state machine, how buffers are exchanged, and

所以，我們看過了 collective state machine，以及 buffers 是如何交換的，

that was actually a contributing factor to some of what we were seeing with InCast.

這其實也是我們看到 InCast 的一些問題的成因之一。

So in these results, we replaced the vendor CCL with our own.

所以在這些結果中，我們將 vendor 的 CCL 換成了我們自己的。

We have our own collective communications library, at least for all to all.

我們有自己的 collective communications library，至少是 for all to all 的。

So we wanted to reduce one factor from the equation.

所以我們想從方程式中減少一個變數。

We're calling this one arm, you know, AI validation or one ARM testing, whatever you want to call it.

我們就稱它為 one arm，你知道的，AI validation 或 one ARM testing，你想怎麼叫都行。

The idea is you mock, like I said before, everything but the device under test.

就像我之前說的，這個概念就是模擬，除了被測設備之外的所有東西。

And you might think, how is this possible?

你可能會想，這怎麼可能辦到？

Like, how can I make my device running a complex collective algorithm or an AI workload talk to something else?

像是，我怎麼讓我的設備跑著複雜的 collective algorithm 或 AI workload，去跟別的東西溝通呢？

It's actually simpler than you think when you get into it.

其實實際操作起來，比你想的還要簡單。

The nice thing about collectives is, is the whole everyone participating in that collective is aware of what everyone else is doing.

collectives 的好處是，參與 collective 的每一個節點都知道其他節點在做什麼。

So like node A is sending to node B, node B knows it's receiving from node A and so on and so forth.

就像節點 A 傳送給節點 B，節點 B 知道它正在接收來自節點 A 的資料，以此類推。

So you kind of know at the beginning of a test, you know what all the patterns are.

所以你大概會在測試開始時就知道所有的模式是什麼。

So all you have to do to make this work is your mocks need to do what the device under test is expecting.

所以，要讓這一切順利運作，你只需要確保你的 mock 模擬出被測設備所期望的行為。

Basically it's got to pass the duck test.

基本上，它必須通過「鴨子測試」。

Like from the accelerator under test point of view, if it looks like a duck, quacks like a duck, it must be a duck.

就像從被測加速器的角度來看，如果它看起來像鴨子，叫起來也像鴨子，那它就一定是鴨子。

So in this case, keysight is a very expensive.

所以，在這個情況下，keysight 的價格非常昂貴。

So Ashutosh will talk more about how exactly this works and some of the details of it, but

所以 Ashutosh 會更詳細地說明這具體是如何運作的，以及一些細節，但是

think of it as a collective emulator that can take input from our collective communications library and execute it in a similar fashion.

你可以把它想像成一個 collective 模擬器，它可以接收來自我們 collective 溝通函式庫的輸入，並以類似的方式執行。

Okay, so now we're going to talk about some applications of this.

好的，現在我們要來談談這個的一些應用。

Application we're going to talk about is measuring collective completion time.

我們要講的應用是測量 collective 的完成時間。

So one thing you commonly do in NPI is you'll measure the performance and you know, just functionality.

所以，在 NPI 中你常做的一件事就是測量效能，你知道的，還有功能性。

Sometimes just getting to performance is hard of.

Of all to all collectives or some other collective with different rank sizes.

像是 all-to-all collectives 或是有不同 rank 大小的其他 collective。

In this example here, we used a fixed message size.

在這個例子裡，我們用了固定的訊息大小。

So then the all to all size changes to correspond to that and we evaluate it.

所以 all-to-all 的大小就會跟著改變，然後我們來評估它。

I can't read our first data point here, but you know, anywhere from.

我看不清楚我們第一個數據點，但你知道的，從。

從 8 個 ranks，到 24 個 ranks，依此類推。

So with that I'm going to hand it off to Ashutosh, who's going to go into more details of the results and interesting things we've discovered.

所以接下來我要把時間交給 Ashutosh，他會更詳細地說明結果以及我們發現的有趣之處。

So we have the accelerator and the neck surrounded by the mocks.

所以我們有加速器和交換器，周圍圍繞著 mock。

So we usually care about two kinds of flows, right?

所以我們通常關心兩種流量，對吧？

Ones that originate from the NIC and land on the mocks, which is NIC TX in red.

一種是從 NIC 發起並到達 mock 的流量，也就是紅色的 NIC TX。

And the ones that originate from the mocks and land on the nic, which is NIC rx.

另一種是從 mock 發起並到達 NIC 的流量，也就是 NIC RX。

This of course flows between the mocks themselves, but that's not conjugation for now.

當然，這也包括 mock 之間的流量，但我們暫時不討論這個。

You see a green line which represents the ideal trend as we go from 8 ranks all the way to 24 ranks, right?

你看到一條綠線，代表著從 8 個 ranks 一直到 24 個 ranks 的理想趨勢，對吧？

Nick RX we see as we go from 8 ranks to 24 ranks, we're able to achieve 90 to 97% of the ideal line rate.

我們看到從 8 個 ranks 到 24 個 ranks 的 NIC RX，我們能達到理想線速的 90% 到 97%。

And the observation for NICK TX trend here is we see that there are some flows that are actually outliers, which is they unexpectedly achieve or exercise higher bandwidth rate than we expect them to.

而 NIC TX 的趨勢觀察是，我們看到有些流量實際上是離群值，也就是它們意外地達到了比我們預期更高的頻寬。

So let's first look at the observation with NIC rx, which is the blue line, right, where we were able to achieve 90 to 97% of the ideal rate.

所以我們先來看看 NIC RX 的觀察，也就是藍線，對吧，我們能達到理想速率的 90% 到 97%。

Let's pick up the use case for 24 ranks, where we have one accelerator and NIC surrounded by 23 physical mocks.

我們來挑選 24 個 ranks 的使用案例，其中有一個加速器和 NIC，周圍有 23 個實體 mock。

它們都是 400 gig 的連接埠速度。

Now, since this is all to all, each mock should be transmitting at 17.4 Gbps, to the NIC, right?

因為這是 all to all，每個 mock 應該以 17.4 Gbps 的速度傳輸到 NIC，對吧？

And based on the trend we saw before this, we're achieving around 97% of the rate, which is 17.0 Gbps.

而根據我們之前看到的趨勢，我們達到了大約 97% 的速率，也就是 17.0 Gbps。