Name: 人工智能擴展計算互聯技術的性能評估 UAL vs UALoE SU (Performance Evaluation of Interconnect Technologies for AI Scale Up Computing UAL vs UALoE SU)
Uploaded: 2025-10-25T23:22:10.000Z
Duration: 19 min 49 s

So today we are going to talk about the interconnects for scale ups.

那今天我們要來聊聊 scale ups 的 interconnect。

關係子句

So over the last couple of days we heard a lot of keynote speakers and presentations that talk about the importance of interconnects for scale ups.

在過去這幾天，我們聽了很多關於 scale ups interconnect 重要性的 keynote speakers 和演講。

So here we want to talk about the two technologies, what we know about the UAL and Ethernet and we're gonna talk about the three approaches to doing that UL over Sue and Rocky and I'm Sri Vengarisna and here's my colleague Srinivas Gangam.

所以今天我們要來聊聊兩種技術，也就是我們知道的 UAL 和 Ethernet，還有我們要講的三種方法，也就是 UL over Sue 和 Rocky，我是 Sri Vengarisna，這是我的同事 Srinivas Gangam。

First let's look at the two approaches of doing that.

首先，我們來看看這兩種方法。

Like once we have the UAL which is based on memory semantics that use load store and atomic kind of memory transactions and we have Ethernet which has UL over Ethernet or scale up Ethernet and these are basically again memory transactions which are encapsulated over optimized Ethernet header.

就像我們有了 UAL，它是基於使用 load store 和 atomic 類型的 memory transactions 的 memory semantics，還有 Ethernet，它有 UL over Ethernet 或 scale up Ethernet，這些基本上都是封裝在 optimized Ethernet header 裡的 memory transactions。

So as we heard it recently too that okay, the memory transactions can be multiple transactions which can be packed in a given packet so that we can amortize the overhead of the IPG and the Ethernet header.

所以就像我們最近聽到的，memory transactions 可以是多個 transactions，可以打包在一個 packet 裡，這樣我們就可以 amortize IPG 和 Ethernet header 的 overhead。

For RDMA transport it basically has the complete header stack of IP Ethernet, IP and the IB header.

對於 RDMA transport，它基本上包含了完整的 IP, Ethernet, IP 和 IB header stack。

So this distinction is very important as we basically go over the performance results what we're going to show later.

所以這個區別非常重要，因為我們接下來要講的效能結果，我們將會進一步說明。

So looking at the key advantages for scale up networking, the first one is load balancing.

所以，看看 scale up networking 的主要優勢，第一個是負載平衡。

When we say load balancing, the load balancing can happen at a very fine granular.

當我們說負載平衡時，負載平衡可以發生在非常細的粒度上。

Basically we can do load balancing at 256 bytes.

基本上，我們可以做到 256 位元組的負載平衡。

All the memory transactions that have been read from the compute can be load balanced across all the Rails.

所有從計算機讀取的 memory transactions 都可以跨所有 Rails 進行負載平衡。

So the Rails all can be optimized for high bandwidth.

所以所有 Rails 都可以針對高頻寬進行優化。

The second one is the latency characteristics for UAL.

If you look at it, the DL Flit is always 640 bytes.

如果你仔細看，DL Flit 總是 640 位元組。

But for Ethernet as we heard from the previous presentation too, the latency can be variable depending on the packet size, depending on how many memory transactions you're packing within a given packet.

但對於 Ethernet，就像我們在之前的簡報中聽到的，延遲可能是可變的，這取決於封包大小，也取決於你在給定的封包中打包了多少 memory transactions。

And with memory semantics one of the biggest advantage what we can have is it can be push model.

而使用 memory semantics，我們可以擁有的最大優勢之一就是它可以是推播模型。

When you say push model, once the COMPUTE is done, we can directly push the data from the compute tile to the networking tile and it basically doesn't incur the knock RTT delay what we see in an soc and all the transactions what are basically started at the compute tile, they all can be completed out of order.

當你說推播模型時，一旦運算完成，我們就可以直接將資料從運算單元推送到網路單元，而且基本上不會產生我們在 SoC 中看到的往返延遲，而且在運算單元啟動的所有交易，都可以亂序完成。

When I say completed out of order, all the transactions are load balanced across the rails and all these transactions can complete by using the maximum Bandwidth and then the completion can be done after that.

當我說亂序完成時，所有交易都會在各個通道之間進行負載平衡，所有這些交易都可以利用最大頻寬完成，然後之後再進行完成。

So we can get the maximum utilization of the bandwidth.

所以我們可以獲得最大的頻寬利用率。

From the implementation point of view, a lot of choices about this implementation depends on the scale up switch, what you're going to deploy the network, for example, if at all the switch is capable of doing packets per second with no limitation, then the packing logic, what you can have in the accelerator can be very efficient.

從實施的角度來看，很多關於此實施的選擇取決於你將部署網路的 scale up switch，例如，如果 switch 能夠處理每秒封包數且沒有限制，那麼加速器中的封包邏輯就可以非常有效率。

It can basically pack only one memory transaction within a packet.

它基本上可以在一個封包中只打包一個 memory transaction。

But that also adds the inherent bandwidth penalty because we have the overhead for a given transaction which is the Ethernet header.

但這也會增加固有的頻寬損失，因為對於給定的交易，我們有 overhead，也就是 Ethernet header。

And for the QS behavior, like depending on the number of traffic classes you have, especially the no loss traffic classes.

而對於 QS 行為，例如取決於你擁有的 traffic class 數量，特別是 no loss traffic classes。

These traffic classes are required in Ethernet because we need to separate the request and response channels, which are different for Ethernet.

這些 traffic classes 在 Ethernet 中是必需的，因為我們需要分離 Ethernet 的請求和響應通道，它們是不同的。

But for UL we already have a request and response channel which are separated out at the EPLI layer.

但對於 UL，我們已經在 EPLI 層分離了請求和響應通道。

So it can get better performance without any QAS properties.

所以它可以在沒有任何 QAS 特性的情況下獲得更好的效能。

But here for Ethernet we need to have separate queues and these need to be scheduled properly for end to end workload.

但在這裡，對於 Ethernet，我們需要有獨立的佇列，而且這些佇列需要妥善排程，以實現端對端的工作負載。

All these implementation choices have a direct impact on latency and the bandwidth.

所有這些實作選擇都會直接影響延遲和頻寬。

What we can observe in the network, this is a typical simple framework.

我們在網路上觀察到的，這是一個典型的簡單框架。

What we are showing up, we have seen it multiple flavors of it.

我們展示的，我們已經看過很多不同形式的了。

Here we can see on the left hand side we have a compute tile.

在這裡，我們可以看到左邊有一個運算單元。

Right hand side we have a networking tile.

And within the networking tile the TX and the RX are connected to the switch.

在網路單元內，TX 和 RX 連接到交換器。

And here the switch can be any out of the shelf ethernet switch.

而這裡的交換器可以是任何現成的 Ethernet 交換器。

And here as the experience done and wanted to transfer the payload, it basically needs to load balance across all the networking tiles.

當有經驗的人想要傳輸負載時，它基本上需要跨所有網路單元進行負載平衡。

And here as I was saying before, the load balancing can happen very efficiently because it can load balance at 256 bytes.

而正如我之前說的，負載平衡可以非常有效地進行，因為它可以以 256 位元組為單位進行負載平衡。

And as the transactions arrive at the ethernet tile, it basically needs to go to the packing logic first.

然後當這些 transactions 到達 ethernet tile 時，它基本上需要先經過 packing logic。

When you talk about the packing logic, what it is trying to do is pack multiple of these memory transactions within an Ethernet packet.

當你說到 packing logic 時，它的作用是將多個 memory transactions 包裝到一個 Ethernet packet 裡面。

而且它可以是 work conserving 的。

But at the same time work conserving and not packing basically will affect your bandwidth.

但同時，work conserving 和不 packing 會影響你的頻寬。

But if at all you pack more, you inherit adds latency.

但如果你 pack 更多，就會增加 latency。

So that's kind of the drawback of packing.

所以這就是 packing 的一個缺點。

Then it goes to the queuing and scheduling logic.

接著會到 queuing and scheduling logic。

And here again as we talked about the queuing, we need to have separate the requests and response channels so that we basically can have the end to end workflow.

這裡一樣，就像我們之前說的 queuing，我們需要區分 requests 和 response channels，這樣我們才能有 end to end 的 workflow。

Here it can be an optimized L2 header, it can be etherne or it can be a SUE header.

這裡可以是 optimized L2 header，可以是 ethernet，也可以是 SUE header。

Then at the link layer it basically needs to have end to end reliability.

所以連結層基本上需要有端對端的可靠性。

So we basically need to at least support the LLR capability within the link layer so that we can at least have link to link protected.

所以我們基本上至少需要在連結層支援 LLR 功能，這樣我們至少可以有連結到連結的保護。

The reason is it only corrects bit errors but not end to end reliability.

原因是它只修正位元錯誤，而不是端對端的可靠性。

Then defies common across Ethernet and ul.

然後定義了 Ethernet 和 UL 的共同點。

And on the receive side we basically have a parsing logic that can parse the packet and then it needs to unpack and basically write the packets into the memory.

而在接收端，我們基本上有一個解析邏輯，可以解析封包，然後需要解開封包並將封包寫入記憶體。

對於 UL 框架來說，它看起來比較簡單。

The reason it is simpler is that okay, it doesn't have the packing logic, it doesn't have the scheduling logic.

它比較簡單的原因是，它沒有封裝邏輯，也沒有排程邏輯。

All this stuff is to some extent incorporated in the DLNDL layer because DL inherently has separate channels for requests and responses.

所有這些東西在某種程度上都包含在 DLNDL 層，因為 DL 本身就有獨立的請求和回應通道。

So here, the biggest takeaway from this here is the logic is simplified here because the interfaces are much more similar because the NOC already has an AXI kind of interface or a load store kind of interface which ties very well with the UPLI interface here.

所以在這裡，最大的重點是邏輯在這裡被簡化了，因為介面更相似，因為 NOC 已經有 AXI 或 load store 類型的介面，這與 UPLI 介面非常契合。

The amount of logic what we have and the area saving what we can have is significant because of this.

因為這樣，我們擁有的邏輯量和節省的面積都相當可觀。

And now we go with the rocky workload here.

而現在我們要來處理 rocky workload。

Like we have a simplified rocky workload.

像是我們有一個簡化的 rocky workload。

Basically talking about the control path which is not performance critical here.

基本上是說控制路徑在這裡不是效能的關鍵。

The connection, the memory registration and the key exchanges can happen at a very slow rate.

連線、記憶體註冊和金鑰交換都可以用非常慢的速度進行。

From the core operations point of view, we only need to support RDMA write and write with immediate.

從核心操作來看，我們只需要支援 RDMA write 和 write with immediate。

The reason for that is okay, all the transactions are bulk transfers and we only wanted to do transfers by load balancing the large payload.

這樣做的原因是，所有的交易都是大量傳輸，而我們只想透過負載平衡大型酬載來進行傳輸。

And what is not required is the send and receive operations.

而不需要的是 send 和 receive 操作。

So from the feature set point of view, multipathing is very important because large IO needs to be load balanced across all the paths.

所以從功能集的角度來看，多重路徑非常重要，因為大型 IO 需要在所有路徑之間進行負載平衡。

So all the packets need to be sprayed across all the paths which are available.

所以所有封包都需要分散到所有可用的路徑上。

When I say out of order delivery it means every packet needs to have an address so that it can be placed in the destination GPU as and when the packet arrives without waiting for reorder or any other complex logic.

我說亂序傳輸，意思就是每個封包都需要有位址，這樣它才能在抵達時就放到目標 GPU 上，而不需要等待重排序或其他複雜的邏輯。

From the congestion point of view we need to have selective acks and nacks.

從壅塞的角度來看，我們需要有選擇性的 ACK 和 NACK。

The packets are basically written into the memory.

They can be ACK back and NACS is for any of the congestion related issues when a packet is dropped or then we can send an AC so that the end to end reliability will resend the packet and from the completion point of View we can do a RDMA write with immediate so that completion can be sent to the destination accelerator.

它們可以被 ACK 回來，而 NACK 是針對任何與壅塞相關的問題，當封包遺失時，我們可以傳送一個 AC，這樣端對端的可靠性就會重新傳送封包，而從完成點的角度來看，我們可以進行 RDMA write with immediate，這樣完成就可以傳送到目標加速器。

So from the rocky semantics point of view, the biggest thing is the load balancing.

所以從 rocky semantics 的角度來看，最重要的一點就是負載平衡。

As we talked about, a large I O now needs to be partitioned across multiple DMA engines within the networking tile.

正如我們所討論的，大型 IO 現在需要在網路 tile 內的許多 DMA 引擎之間進行分割。

And depending on the I O size, the the partition of these jobs across the networking tiles can be chunky if at all.

而根據 IO 的大小，這些工作在網路 tile 之間的分割如果有的話，可能會是塊狀的。

It's a small transfer that needs to be done, then the load balancing is not efficient across the networking tiles.

這是個小傳輸，負載平衡在網路 tile 之間就不會有效率。

And here most of the ROCKY engines are a pull model.

而這裡大部分的 ROCKY 引擎都是採用 pull model。

When I say pull model, usually the transactions, the DMA thread is sitting in the networking tile wherein it needs to pull the data from the host memory or the GPU memory and then it needs to reorder the transactions and form a packet and send it over the interface.

我說 pull model，通常是交易，DMA 執行緒會待在網路 tile，它需要從主機記憶體或 GPU 記憶體拉取資料，然後重排交易，組成封包並透過介面傳送。

Here from the rail optimization point of view, like the bandwidth across these rails may not be completely load balanced to the extent of what memory semantics can achieve.

從 rail optimization 的角度來看，這些 rail 的頻寬可能無法像記憶體語義那樣達到完全的負載平衡。

Because here the IO chunking can be little bit more grad like little more chunkier.

因為這裡的 IO 分塊可以稍微更精細一點，更小塊一點。

And for the ACK generation again we have the same issue.

而 ACK 的產生，我們又面臨同樣的問題。

When we are writing the packet at the receiving side to write into the memory, it needs to wait for all the acks of the memories which memory transactions sent on the NOC for the acks to come back to generate the ack for the packet.

當我們在接收端寫入封包到記憶體時，它需要等待所有在 NOC 上傳送的記憶體交易的 ACK 回來，才能產生封包的 ACK。

And from the resiliency point of view, again as we talked about, is a hierarchical kind of a load balancing IO chunk is basically partitioned to multiple smaller chunks within the networking tile.

從彈性角度來看，正如我們之前討論的，這是一種階層式的負載平衡，IO 塊基本上被分割成網路 tile 內的許多小塊。

And if at all a link is lost within a networking tile, then the load balancing across the networking tiles needs to be a little bit adjusted and that gets a little bit more complicated.

如果網路 tile 內的連結遺失了，那麼網路 tile 之間的負載平衡就需要稍微調整，這會變得有點複雜。

Going over the simple again, the XPU kind of a framework here is the same thing what we talked about before, we have a DM engine here which is kind of reading the transactions from the memory and then sending it out.

再簡單說一下，這裡的 XPU 框架和我們之前談到的相同，我們有一個 DMA 引擎，它會從記憶體讀取交易，然後傳送出去。

And we have the queuing and scheduling logic and most of the logic can be similar down the road here I talked about having Ethernet IP and AIB header, but

而我們有佇列和排程邏輯，大部分邏輯在後續都會很相似，我之前提過有 Ethernet IP 和 AIB header，但

it can be a SU optimized header too here which can reduce the load.

它也可以是 SU optimized header，這樣可以減輕負載。

Okay, from the workflow comparison point of view, like I'll just walk through that quickly, we went over that.

好的，從工作流程比較的角度來看，我會快速帶過，我們已經講過了。

Like load balancing, it can be very efficient for UAL and for rocky it can be a little bit chunkier for the switch latency can be the same across all it will be the 200-300Ns the write responses are very efficient for UL and for Rocky they can be a little bit more delayed because of the ACK accumulation.

像是負載平衡，對 UAL 來說可以很有效率，對 rocky 來說可能會有點零散，交換器的延遲在所有情況下都可能相同，會是 200-300Ns，寫入回應對 UL 來說非常有效率，而對 Rocky 來說，因為 ACK 的累積，可能會有點延遲。

And the bandwidth utilization is very good for UL from the functionality comparison.

從功能比較來看，UL 的頻寬利用率非常好。

Again, here is the same thing what we talked about before.

再說一次，這裡跟我們之前談到的相同。

The dmitradius pull and push and the packing logic is there for UL over Ethernet, which potentially can add some latency.

dmitradius 的 pull 和 push 以及打包邏輯是用於 UL over Ethernet，這可能會增加一些延遲。

And for channel based scheduling for ual, as we talked about, the requests and responses have separate channels.

而對於 ual 的基於通道的排程，正如我們所討論的，請求和回應有獨立的通道。

But for Ethernet we need to separate them out to separate queues from the attributes.

但對於 Ethernet，我們需要將它們分開到不同的佇列，從屬性來看。

字幕列表影片播放

人工智能擴展計算互聯技術的性能評估 UAL vs UALoE SU (Performance Evaluation of Interconnect Technologies for AI Scale Up Computing UAL vs UALoE SU)

assume

critical

multiple

basically