Name: 人工智能織物的源路由選擇 (Source Routing for AI Fabrics)
Uploaded: 2025-10-25T00:04:14.000Z
Duration: 15 min 41 s

程度副詞

I'm a Product Manager at Marvell, handling data center software Sonic Inside.

我是 Marvell 的 Product Manager，負責資料中心軟體 Sonic Inside。

I am also Director of Software Product Management at Marvell.

我也是 Marvell 的 Director of Software Product Management。

So today we're going to present an alternate approach to managing flows in an AI cluster.

所以今天我們要提出一個管理 AI cluster 中流量的替代方法。

I want to be very clear when we say alternate, we're not saying better, we're just saying different.

我想說清楚，當我們說替代方法時，我們不是說更好，我們只是說不同。

So we'll give you a brief introduction to the AI training traffic background, the fabrics stuff that David has probably already mentioned.

所以我們會簡單介紹一下 AI training traffic 的背景，以及 David 可能已經提過的 fabrics 相關內容。

The impact of job completion time, especially when it comes to congestion.

job completion time 的影響，特別是在遇到壅塞的時候。

Other latency dynamics within the fabric.

Some existing approaches to improve JCT as well as our proposal and our call to action.

一些現有的改善 JCT 的方法，以及我們的提案和 call to action。

So with that I'll hand it over to my partner here.

那麼，我就把時間交給我這位夥伴。

Okay, so before we go into the solution of what we are proposing, we wanted to kind of set a base of the different AI models.

好的，在我們進入我們提出的解決方案之前，我們先來介紹一下不同的 AI models。

We have different network architectures required.

所以，如果你們看這張投影片。

So there were like typically the old generation ones where we had small AI trainings which could have been accommodated in just a single chassis with all the CPUs and the GPUs required with all the PCIe links.

所以，過去通常是舊世代的，那時候我們有小型 AI 訓練，這些訓練可以容納在單一機箱內，包含所有需要的 CPU 和 GPU，以及所有的 PCIe 連結。

And then we have the next generation where our models kind of expanded, we needed bigger clusters.

然後我們進入下一代，我們的 models 規模擴大了，我們需要更大的 clusters。

So there we went into kind of the medium and large AI training fabrics where you had dedicated AI fabrics.

所以我們進入了中大型 AI training fabrics，那時候有專用的 AI fabrics。

It is only dedicated for AI training and it has always made sure that you have only the AI workloads running on them.

它只專門用於 AI training，而且一直確保只有 AI workloads 在上面運行。

So it could have dedicated networks and it could have even just proprietary networks.

所以它可能有專用的網路，甚至可以有專有的網路。

But what we are seeing from there, going ahead with a lot of 2023 becoming genai, more and more AI applications coming into the market and more and more people integrating AI into their applications.

但我們從那裡看到的，進入 2023 年，越來越多的 generative AI，越來越多的 AI applications 出現，越來越多人將 AI 整合到他們的 applications 中。

We see that we need a multi tenant fabric because every application today, every software application, whether it is cloud or enterprise, they are looking at integrating AI into their whole software suit.

我們看到我們需要一個 multi tenant fabric，因為現在的每一個 application，每一個軟體 application，無論是雲端還是企業，他們都希望將 AI 整合到他們的整個軟體套件中。

So what that means is we can't have a completely dedicated network just for AI.

所以這意味著我們不能有一個完全專用的網路只用於 AI。

That is where the multi tenant aspect of the network comes in.

這就是網路的 multi tenant aspect 的地方。

We have to cater for different kinds of applications, different kinds of AI workloads.

我們必須滿足不同類型的 applications，不同類型的 AI workloads。

And even like even in the AI, the requirement for each AI applications are different.

甚至在 AI 裡面，每個 AI applications 的需求也不同。

So what we need to make sure is that we cater for all of these and every company doesn't have the bandwidth to put dedicated AI networks because their LLMs might not be as huge as typically what we have seen in a chatgpt AI, right?

所以我們需要確保的是，我們能夠滿足所有這些需求，而且並非所有公司都有足夠的頻寬來設置專用的 AI 網路，因為他們的 LLMs 可能不像我們通常在 ChatGPT AI 中看到的那麼龐大，對吧？

So that is where multi tenant networks becomes more important.

這就是 multi tenant networks 變得更重要的原因。

And with multi tenant networks becoming more important, what we are seeing is Ethernet becomes a default because you have different kinds of traffic, different kinds of requirements of applications.

隨著 multi tenant networks 變得越來越重要，我們看到乙太網路成為預設選項，因為你會有不同類型的流量，不同類型的應用程式需求。

We see that Ethernet becomes more adaptable to these kind of networks instead of having a dedicated and proprietary solution into the AI networks.

我們看到乙太網路比專用且專有的解決方案更能適應這類網路，而不是用於 AI 網路。

So when we think about AI, if you think how the AI workloads are placed, how they're interacting with each other, there are multiple steps goes even before it comes onto the server, right?

所以當我們談論 AI 時，如果你想想 AI workloads 的放置方式，它們之間的互動方式，在進入伺服器之前，其實還有很多步驟，對吧？

So there are dedicated servers who are planning the whole AI orchestration, the work job orchestration.

所以有專用的伺服器負責規劃整個 AI 的協調，工作任務的協調。

There is a mapper sitting on a server thinking, okay, I have this much workload which I have to work on and I have this much capacity on my CPUs and my GPUs, so where can I place them and how do I interact to get all the results?

伺服器上會有一個映射器，它會思考，好的，我有這麼多的工作量需要處理，我的 CPU 和 GPU 有這麼大的容量，那麼我可以在哪裡放置它們，以及我該如何互動才能獲得所有結果？

One of the key factors for that is job completion time.

其中一個關鍵因素是 job completion time。

How can I get the job completed at the fastest time possible?

我該如何以最快的速度完成任務？

所以對於所有這些事情，實際上

the mapper has already mapped all the workloads, usually on your data center, very specific to the racks and the servers, everything mapped.

這個映射器已經將所有的工作負載都映射好了，通常是在你的資料中心，精確到機架和伺服器，全部都映射好了。

So it has a complete view and it will make sure that it knows the ideal state, where it wants to be and how it can place its jobs on different infrastructure products.

所以它有一個完整的視圖，並且會確保它知道理想狀態，它想要達到的目標，以及它如何在不同的基礎設施產品上放置它的工作。

So if you think of that, the training goes in multiple stages.

所以如果你仔細想想，訓練會抽成好幾個階段。

So we have the initial testing parameters, we provide all the inputs into the learning model, we get intermediate results, we work on them multiple iterations to kind of get into the final trained model.

我們有初始的測試參數，將所有輸入提供給學習模型，得到中間結果，然後我們進行多次迭代來得到最終訓練好的模型。

All of this has a lot of east west traffic, and

所有這些都會產生大量的東西向流量，而且

all of these east west traffic, if it is not planned, tends to have congestion on some of the resources.

所有這些東西向流量，如果沒有事先規劃，就容易在某些資源上造成壅塞。

Because every result you are getting out of each training step is going into one or the other resource which is concentrating all these results together.

因為你從每個訓練步驟中得到的每個結果都會進入某個資源，將所有這些結果集中在一起。

What happens there is we hit the congestion and we have to mitigate these congestion and plan ahead.

那時候我們就會遇到壅塞，我們必須緩解這些壅塞並提前規劃。

So there are multiple ways which is kind of addressing all these congestion issues already.

所以有很多方法可以解決這些壅塞問題。

So it is like we have the packet spraying method, we have the VOQ method, we have the telemetry adaptive method.

像是我們有封包噴灑法、VOQ 方法，還有遙測自適應方法。

All of these are great in different scenarios.

So one of the other alternative we are trying to propose today is because you have a mapper already, you have someone who is already looking into your network and all of your resources from a higher level and it knows exactly where the job is going to be placed and how these jobs are going to interact.

所以我們今天想提出的另一個替代方法是，因為你已經有一個 mapper，有一個人在從更高層級關注你的網路和所有資源，而且他確切知道工作將會被放置在哪裡，以及這些工作將如何互動。

We are saying that it might be better or it is a alternative approach to look at the network and the resources from a higher level.

我們說，從更高層級來看待網路和資源，可能是個更好的方法，或者說這是一個替代方法。

Don't just think from a network or a server or a GPU or XPU level, but

不要只從網路、伺服器、GPU 或 XPU 層級思考，而是

And if you already know how you're interacting with each of these modules, then maybe it is better to configure upfront what the flows are.

如果你已經知道你如何與這些模組中的每一個互動，那麼也許最好預先設定好 flow 是什麼。

So to go deeper into the whole proposal itself, I'll hand it over to Kishore who can take you into more details on this.

為了更深入地探討整個提案本身，我將把它交給 Kishore，他可以為你提供更多細節。

So like Pratya mentioned, this is an alternative approach.

所以，正如 Pratya 所提到的，這是一個替代方法。

There is a mapping function that takes pieces of your workload and puts it maps it to certain compute elements in your network in your infrastructure.

有一個對應功能，它會將你的工作負載片段，對應到你網路、你基礎設施中的特定運算元件。

So our approach is to kind of piggyback on that.

所以我們的方法就是順著這個來。

So assuming that there is a controller or a mapper, there will be a way for us to pre program the flows, the traffic flows in the fabric from point A to point B.

所以假設有一個控制器或對應器，我們就能夠預先設定好 fabric 中從 A 點到 B 點的 flow、流量 flow。

So that approach is something that we're calling flow path routing or source routing.

這個方法我們稱之為 flow path routing 或 source routing。

The idea is to assign the paths to the flows in advance before you actually start your workload pre program your fabric.

概念是在你實際開始工作負載、預先設定你的 fabric 之前，就將路徑指派給 flow。

Then we use the controller to prep the fabric, install all the routes and install the paths into your fabric as an initial step.

然後我們使用控制器來準備 fabric，安裝所有路由，並在初始步驟中將路徑安裝到你的 fabric 中。

Your controller can load balance your flows using source routes.

你的控制器可以使用 source routes 來負載平衡你的 flows。

Assuming it has some information about your network.

假設它對你的網路有一些資訊。

We'll talk about that a little bit later.

Your fabric nodes as defined in your fabric as the top or your leaf spine layer do not really need to do a lookup.

在你的 fabric 中定義的 fabric nodes，也就是你的 top 或 leaf spine 層，其實不需要做 lookup。

And we'll talk about how we go about that.

Your destination ports can be embedded into your packet ahead of time just like any old source route, similar to what you see in SRV6.

你的目的地 port 可以像舊式的 source route 一樣，預先嵌入到你的封包裡，這跟你在 SRV6 裡看到的很像。

And we'll talk about the difference what this proposal has versus SRV6 in a little bit.

我們稍後會聊聊這個提案和 SRV6 之間的差異。

And we can eliminate lookup latency in your fabric.

我們可以消除你 fabric 中的 lookup latency。

And the idea here is to use the information in your packet to immediately make the decision.

這樣做的想法是利用封包中的資訊立即做出決策。

就像你 fabric node 裡的線路一樣。

So how does this new packet format look like and how is it embedded into a typical IP?

那麼這個新的封包格式看起來是怎麼樣的？又是怎麼嵌入到一般的 IP 裡的呢？

So on the left hand side you have your layer 2 header, your IP header and your usual transport protocol of choice.

所以在左邊，你有你的 layer 2 header、你的 IP header，還有你常用的 transport protocol。

Whether it's udp, tcp, Rocky, rdma, doesn't matter.

無論是 udp、tcp、Rocky、rdma，都沒關係。

Right in the middle we are introducing a new shim or what we're calling our flow Path routing header in between the layer 2 and IP.

中間我們加入了一個新的 shim，或者說我們稱之為 flow Path routing header，放在 layer 2 和 IP 中間。

And the reason why we put it here is we do not want to dictate new hardware implementations on the Mac, just take advantage of what you already have, right?

我們把它放在這裡的原因是，我們不想強制要求 MAC 進行新的硬體實作，只是利用你現有的東西，對吧？

And we don't change the ip, we don't change the transport.

我們也不會改變 IP，也不會改變 transport。

In fact, we don't use the IP or the transport at all as a part of the switching decision making.

事實上，我們在做交換決策時，根本不會用到 IP 或 transport。

On the right hand side, this is the header that we're defining.

右邊這邊，就是我們定義的這個 header。

As you can see, there are five 16 bits fields which we're calling hop IDs.

你可以看到，有五個 16 位元的欄位，我們稱之為 hop IDs。

Now your hop ID maps to a physical port number on your device.

你的 hop ID 會對應到你裝置上的實體連接埠號碼。

So this implies that your controller has to have intimate knowledge of not just your fabric and what you're connecting to, but

所以這表示，你的控制器必須非常瞭解你的 fabric 和你連接的對象，而且

it also has to have an intimate knowledge of the actual switch itself and what physical ports map to your front panel.

它還必須非常瞭解實際的交換器本身，以及哪些實體連接埠對應到你的前面板。

So it's a little bit more involved work onto the controller, but

所以這對控制器來說是比較複雜一點的工作，但是

it's something that we believe is achievable.

So on the left hand side in this diagram you'll see a server connecting through a two tier fabric to another server, sending traffic from point A to point B.

所以各位可以看到，在這個圖的左邊，有一臺伺服器透過一個兩層式的 fabric 連接到另一臺伺服器，從 A 點傳送流量到 B 點。

We have defined roles for each port in this fabric.

我們為這個 fabric 中的每個連接埠定義了角色。

One role is the access port, the other role is the fabric port.

一種角色是存取連接埠 (access port)，另一種角色是 fabric 連接埠 (fabric port)。

Your access port's responsibility is to perform a lookup of your flow.

你的存取連接埠的責任是執行你流量的查找 (lookup)。

And your flow can be Flow can be determined by an arbitrary set of fields up to you, the one who defines the fabric.

而你的流量可以由你來決定，也就是定義 fabric 的人，用任意的一組欄位來判斷。

Right now we're assuming source port, sipdip, maybe Q pair source and Q pair destination in the case of Rocky.

現在我們假設是來源連接埠、sip、dip，在 Rocky 的情況下，還有 Q pair 來源和 Q pair 目標。

字幕列表影片播放

人工智能織物的源路由選擇 (Source Routing for AI Fabrics)

assume

approach

process

dedicated