Name: 介紹 Dagster+ (Introducing Dagster+)
Uploaded: 2025-03-22T20:01:18.000Z
Duration: 29 min 51 s

大家好，我是 Dagster 實驗室的首席執行官皮特。

Today we are launching Dagster Plus, the next evolution of Dagster Cloud.

今天，我們推出了 Dagster Plus，它是 Dagster Cloud 的下一代產品。

基礎被動語態

Dagster Plus is a major rethinking of what a data orchestrator is, where it sits in the stack, and how data platforms are constructed.

Dagster Plus 是對什麼是數據協調器、數據協調器在堆棧中的位置以及如何構建數據平臺的一次重大反思。

Since our launch in August of 2022, we've seen enormous growth in both the Dagster Core open source project and our commercial product.

自 2022 年 8 月推出以來，Dagster Core 開源項目和我們的商業產品都取得了巨大的發展。

We now have hundreds of commercial customers and the fastest growing open source community in the category.

現在，我們擁有數百家商業客戶和增長最快的開源社區。

With this growth comes broader and deeper relationships with our diverse set of customers.

隨著業務的增長，我們與不同的客戶建立了更廣泛、更深入的關係。

As we engaged with these customers throughout 2022 and 2023, we heard a few key challenges.

在 2022 年和 2023 年與這些客戶的接觸中，我們聽到了一些關鍵的挑戰。

They were spending too much money on infrastructure, especially on the data warehouse, ETL tools, and AI workloads, and struggled to understand, optimize, and control their spend.

他們在基礎設施上花費了太多資金，尤其是在數據倉庫、ETL 工具和人工智能工作負載上，並且難以理解、優化和控制其支出。

Their stacks were sprawling and complex with hundreds of different integrations and point solutions and varying levels of maintenance, and no single pane of glass for understanding the data platform as a whole.

他們的堆棧既龐雜又複雜，有數百個不同的集成和點解決方案，維護水準也參差不齊，沒有一個單一的窗口來了解整個數據平臺。

Inspired by the data mesh architecture, they wanted to empower individual teams to work autonomously end-to-end, but struggled to balance this with the centralized needs of the data organization.

受數據網格架構的啟發，他們希望賦予各個團隊端到端自主工作的能力，但又難以在數據組織的集中需求之間取得平衡。

We realized that we could solve many of these problems for customers by making a significant investment in tooling that sits on top of the core data orchestration engine.

我們意識到，通過對核心數據協調引擎之上的工具進行大量投資，我們可以為客戶解決許多此類問題。

So we spent the last 12 months spinning up a team dedicated to these problems, building furiously, and iterating with design partners.

是以，在過去的 12 個月裡，我們組建了一個團隊，專門負責解決這些問題，狂熱地進行構建，並與設計合作伙伴一起不斷迭代。

I'm proud of what the team has accomplished, and I'm excited to present the fruits of our labor to you today.

我為團隊所取得的成就感到驕傲，今天我很高興向大家展示我們的勞動成果。

So let's dig into what Dagster Plus is specifically and why Dagster Labs is uniquely positioned to deliver this type of product.

是以，讓我們來具體瞭解一下 Dagster Plus 是什麼，以及為什麼 Dagster Labs 在提供此類產品方面具有獨特的優勢。

First and foremost, Dagster Plus is the next evolution of Dagster Cloud, our enterprise-class hosted product.

首先，Dagster Plus 是企業級託管產品 Dagster Cloud 的升級版。

This means it includes everything that was part of Dagster Cloud, including enterprise access control, seamless scale up to tens of thousands of assets and hundreds of thousands of partitions, and serverless or hybrid on-prem managed infrastructure.

這意味著它包含了 Dagster Cloud 的所有功能，包括企業訪問控制、無縫擴展至數萬個資產和數十萬個分區，以及無服務器或混合內部託管基礎架構。

This also means that we're retiring the Dagster Cloud brand today.

這也意味著我們將在今天退出 Dagster Cloud 品牌。

Second, Dagster Plus is a suite of tools that sits on top of Dagster's data orchestration engine and benefits from this deep integration.

其次，Dagster Plus 是 Dagster 數據協調引擎之上的一套工具，可從這種深度集成中獲益。

Today we'll be focused on four primary areas of the product.

今天，我們將重點討論產品的四個主要方面。

We've built a brand new data catalog and data discovery capability in Dagster Plus, which enables all users of the data platform to discover and leverage data autonomously.

我們在 Dagster Plus 中構建了全新的數據目錄和數據發現功能，使數據平臺的所有用戶都能自主發現和利用數據。

Dagster Plus builds on Dagster Core's asset checks feature, introducing additional capabilities around anomaly detection, freshness, and alerting.

Dagster Plus 以 Dagster Core 的資產檢查功能為基礎，引入了異常檢測、新鮮度和警報等附加功能。

We've made massive improvements to our branch deployments feature from Dagster Cloud and are launching a killer new capability, change tracking, with Dagster Plus.

我們對 Dagster Cloud 的分支部署功能進行了大量改進，並通過 Dagster Plus 推出了一項極具殺傷力的新功能--變更跟蹤。

And finally, Dagster Insights enables data teams to manage their spend on tools like Snowflake, Fivetran, and OpenAI without doing any additional integration work.

最後，Dagster Insights 使數據團隊能夠管理其在 Snowflake、Fivetran 和 OpenAI 等工具上的支出，而無需進行任何額外的集成工作。

With Dagster Plus, Insights is now generally available and comes with many new features and integrations.

隨著 Dagster Plus 的推出，Insights 現在已普遍可用，並具有許多新功能和集成。

The upcoming presentations will dive into each of these in detail, but before we do, I wanted to touch on why we, rather than our competitors, are uniquely positioned to address these challenges.

在接下來的演講中，我們將逐一深入探討這些問題，但在此之前，我想先談談為什麼我們而不是我們的競爭對手在應對這些挑戰方面具有獨特的優勢。

Dagster has historically been considered a data orchestrator.

Dagster 歷來被視為數據協調器。

All of Dagster's main alternatives, such as Apache Airflow, proudly declare themselves as workflow-oriented tools.

所有 Dagster 的主要替代產品，如 Apache Airflow，都自豪地宣稱自己是面向工作流程的工具。

They are primarily focused on the scheduling of black box tasks.

它們主要側重於黑盒任務的調度。

While this makes these tools quite flexible and suitable for a variety of non-data applications, they are not specialized to the data domain.

雖然這使得這些工具相當靈活，適用於各種非數據應用，但它們並不是數據領域的專用工具。

In fact, we believe that there is an impedance mismatch between the workflow-oriented tools like Airflow, which deals primarily with tasks, and the rest of the data platform, which deals primarily with data assets, like database tables, machine learning models, and files.

事實上，我們認為，Airflow 等面向工作流的工具主要處理任務，而數據平臺的其他部分主要處理數據資產，如數據庫表、機器學習模型和文件，兩者之間存在阻抗失配。

Dagster is the only widely adopted asset-oriented data orchestrator, built from the ground up with data assets and data awareness at its core.

Dagster 是唯一被廣泛採用的面向資產的數據協調器，其核心是數據資產和數據意識。

This unique architecture enables Dagster Plus to deliver a category-redefining, integrated experience, bringing together unique capabilities into one tool in order to speed up your development work, reduce your costs, and simplify your data platform.

這種獨特的架構使 Dagster Plus 能夠提供定義類別的集成體驗，將獨特的功能整合到一個工具中，從而加快您的開發工作，降低您的成本，簡化您的數據平臺。

Let's hear from Jared, who will tell us about the data catalog.

下面請賈裡德為我們介紹數據目錄。

I'm Jared, the head of product at Dagster Labs.

我是賈裡德，Dagster 實驗室的產品主管。

Today we're announcing a brand new data catalog as part of Dagster Plus, but before we get into that, let's talk about the unfulfilled promise of data catalogs.

今天，我們將發佈一個全新的數據目錄，作為 Dagster Plus 的一部分，但在此之前，讓我們先談談數據目錄尚未實現的承諾。

As your organization scales, the number of assets you have to manage also scales.

隨著企業規模的擴大，需要管理的資產數量也在增加。

It doesn't take long before you end up with a large number of rarely used data models with poor documentation and questionable provenance.

用不了多久，就會出現大量很少使用的數據模型，這些數據模型的文檔不完善，來源也有問題。

The central premise of data catalogs is to resolve all this by helping teams avoid rework and duplication, and help data practitioners work more independently by enabling them to discover and use trusted data assets.

數據目錄的核心前提是通過幫助團隊避免返工和重複來解決所有這些問題，並通過使數據從業人員能夠發現和使用可信的數據資產來幫助他們更加獨立地工作。

Unfortunately, because standalone data catalogs don't have a native understanding of data operations, they have to create observations about the state of data assets from the exhaust of the loosely integrated tools that comprise the data platform.

遺憾的是，由於獨立的數據目錄對數據操作沒有原生的理解，是以它們必須從組成數據平臺的鬆散集成工具的排氣中創建有關數據資產狀態的觀察結果。

As a result, data engineering team ends up constantly troubleshooting problems of accuracy and timeliness in the syncing the state of their data assets to the data catalog.

是以，數據工程團隊在將數據資產的狀態同步到數據目錄時，需要不斷排除準確性和及時性方面的問題。

All the while, they are trying to coax the rest of their stakeholders to follow a new set of development practices to ensure that it remains in sync.

與此同時，他們還在努力哄騙其他利益相關者遵循一套新的開發實踐，以確保保持同步。

Typically, the end result is a lack of adoption from the very downstream stakeholders that the data team was trying to help.

通常情況下，最終結果是數據團隊試圖幫助的下游利益相關者不採用數據。

Unlike traditional workflow orchestrators, Dagster is asset oriented.

與傳統的工作流協調器不同，Dagster 以資產為導向。

It places data assets at the core of the development framework.

它將數據資產置於開發框架的核心位置。

The result is a powerful combination of context about data operations and the outputs of those operations.

其結果是將數據操作的上下文與這些操作的輸出結果有力地結合在一起。

This view of data assets and operations makes Dagster a natural tool for data discovery and documentation, creating a data catalog that serves the needs of all practitioners.

這種數據資產和操作視圖使 Dagster 成為數據發現和記錄的天然工具，創建的數據目錄可滿足所有從業人員的需求。

With the launch of Dagster+, teams who want to spend less time managing integrations for their tools and more time getting things done will find it particularly useful as a new system of record.

隨著 Dagster+ 的推出，那些希望花更少的時間管理工具集成、花更多的時間完成工作的團隊會發現它作為一個新的記錄系統特別有用。

And while we're still early in our journey, we are already seeing that, for many teams, this eliminates the need for traditional point solution data catalogs.

雖然我們還處於起步階段，但我們已經看到，對許多團隊來說，這消除了對傳統點解決方案數據目錄的需求。

The first major set of improvements focuses on the kinds of data that Dagster collects and displays at the asset definition level.

第一組主要改進集中於 Dagster 在資產定義層面收集和顯示的數據類型。

On the asset details page, you'll now be able to find a new variety of information about your data asset that helps you understand the status, technical state, definition, ownership, location, and more all at a glance.

現在，您可以在資產詳細信息頁面找到有關數據資產的各種新資訊，幫助您瞭解數據資產的狀態、技術狀態、定義、所有權、位置等，所有資訊一目瞭然。

You can see things like the latest status of your asset, including the data quality testing configured on the asset, its description, who owns the asset, what groups and tags are associated with it, as well as where the asset is located, its compute details, and the associated resources and compute kind.

您可以查看資產的最新狀態，包括資產上配置的數據品質檢測、資產描述、資產所有者、與資產相關的組和標籤，以及資產的位置、計算詳細信息、相關資源和計算類型。

And for structured assets, you'll now find a new rich set of details, including any raw SQL that defines the asset structure, metadata about column names, descriptions, and data types.

對於結構化資產，你現在可以找到一組新的豐富細節，包括定義資產結構的任何原始 SQL、有關列名、描述和數據類型的元數據。

And an exciting new change that we're launching for enterprise customers is the ability to explore the lineage of individual columns on the page.

我們正在為企業客戶推出一項令人興奮的新功能，即探索頁面上各欄的脈絡。

When automatically ingested from DBT or derived from user-provided metadata, Dagster can compute the relationship of individual columns and represent them on the page similar to our asset lineage graph.

當從 DBT 自動攝取或從用戶提供的元數據中導出時，Dagster 可以計算各個列之間的關係，並將它們表示在頁面上，類似於我們的資產脈絡圖。

Column lineage provides a clear understanding of how data moves and transforms within a system, allowing you to track things like the proliferation of PII or tracing a mistake through its various downstream tables.

通過列系，可以清楚地瞭解數據在系統內的移動和轉換情況，從而跟蹤 PII 的擴散或通過各種下游表追蹤錯誤等情況。

This new feature will help you manage data quality, compliance, and increase the speed of your troubleshooting.

這項新功能將幫助您管理數據品質和合規性，並加快故障排除速度。

But our feature release doesn't stop with the asset data model and UI.

但我們發佈的功能並不僅限於資產數據模型和用戶界面。

We're also releasing a new, more powerful search experience to help you quickly locate the most relevant data.

我們還將推出功能更強大的全新搜索體驗，幫助您快速查找最相關的數據。

From this interface, you can search for data assets by metadata, including their name, their compute kind, the asset group, the asset owner, or by any associated tag that you add to your assets, and see information about all of your data assets by their associated metadata.

在這個界面上，你可以通過元數據搜索數據資產，包括它們的名稱、計算類型、資產組、資產所有者，或通過你添加到資產中的任何關聯標籤，並通過它們的關聯元數據查看所有數據資產的資訊。

I want to call out a feature here that might get passed over, which is definition-level tags.

我想在這裡指出一個可能會被忽略的功能，那就是定義級標籤。

With definition-level tagging, you can more easily add cross-sectional data to organize and group your data assets.

通過定義級標籤，您可以更輕鬆地添加橫截面數據，對數據資產進行組織和分組。

So now you can tag all of your intermediary assets, or your assets with PII, or the assets that support your finance team, and make these assets easily discoverable by their relevant stakeholders.

是以，現在您可以標記所有中介資產、帶有 PII 的資產或支持財務團隊的資產，並使相關利益方能夠輕鬆發現這些資產。

Dagster's data catalog experience is powerful because of how it combines context about data pipelines and the data assets they produce.

Dagster 的數據目錄體驗之所以強大，是因為它結合了數據管道及其產生的數據資產的上下文。

But we know that not everyone needs to look under the hood inside Dagster.

但我們知道，並不是每個人都需要查看 Dagster 內部的引擎蓋。

Sometimes your data practitioners just need to know where to find the right asset.

有時，您的數據從業人員只需要知道在哪裡可以找到正確的資產。

And that is why we're also launching a new capability we're calling Catalog Mode, which allows you to hide most of the operational views of Dagster, and instead focus on the assets that Dagster has documented.

是以，我們還推出了一項新功能，我們稱之為 "目錄模式"，它允許您隱藏 Dagster 的大部分操作視圖，而專注於 Dagster 已記錄的資產。

This new feature can be turned on by default for your viewers in your organization, or turned off when they need to go deeper.

這項新功能可以默認為組織中的查看者打開，也可以在他們需要深入查看時關閉。

So that's our high-level introduction to the brand new data catalog and search features in Dagster+.

以上就是 Dagster+ 中全新數據目錄和搜索功能的高級介紹。

We expect this addition will make it easier for all teams on the data platform to find information about their data assets and status, and to operate more autonomously, while reducing the operational burden on the platform team.

我們預計，這一新增功能將使數據平臺上的所有團隊更容易找到有關其數據資產和狀態的資訊，並更自主地運行，同時減輕平臺團隊的營運負擔。

You might find it helps eliminate the need for standalone tools which will reduce context switching, help break down data silos, and reduce the total cost of ownership of your data platform.

您可能會發現，它有助於消除對獨立工具的需求，從而減少上下文切換，幫助打破數據孤島，降低數據平臺的總體擁有成本。

We know that data cataloging is an important capability for teams, and we're excited to be rolling out this first version, but we're not stopping here.

我們知道數據編目是團隊的一項重要功能，我們很高興能推出第一個版本，但我們不會就此止步。

Sooner Catalog will also consume data about assets that are not managed by Dagster, and represent them in the data catalog as external assets.

Sooner Catalog 還將消耗有關非 Dagster 管理資產的數據，並在數據目錄中將其表示為外部資產。

I hope this first overview has you interested in the new data cataloging capabilities in Dagster+, but check out the companion blog to find out more.

我希望這第一份概述能讓您對 Dagster+ 的新數據編目功能產生興趣，但要了解更多資訊，請查看配套博客。

I'm Sandy, the lead engineer on the Dagster project.

我是桑迪，Dagster 項目的首席工程師。

Today I'm going to talk about the new data reliability features that we're launching as part of Dagster+, and how they help you deliver trustworthy data to stakeholders across your entire data platform.

今天，我將談談我們作為 Dagster+ 的一部分推出的新數據可靠性功能，以及它們如何幫助您在整個數據平臺上向利益相關者提供可信賴的數據。

The job of a data orchestrator is to help you produce and maintain the data assets that your organization depends on.

數據協調器的工作是幫助您生成和維護組織所依賴的數據資產。

Whether those assets are tables, machine learning models, or reports, in order to be useful they need to be trustworthy and reliable.

無論這些資產是表格、機器學習模型還是報告，要想發揮作用，它們就必須是可信和可靠的。

That is, they need to consistently contain up-to-date and high-quality data.

也就是說，它們需要始終包含最新的高質量數據。

But bugs are inevitable, and pipeline developers don't control the data that enters their pipelines.

但錯誤是不可避免的，管道開發人員無法控制進入管道的數據。

So the only realistic way to achieve trustworthy data is with monitoring and issue detection, so that issues can be addressed before they affect downstream consumers.

是以，實現可信數據的唯一現實途徑是進行監控和問題檢測，這樣就能在問題影響下游消費者之前加以解決。

Most data teams struggle with data reliability because they don't have robust monitoring.

大多數數據團隊之所以在數據可靠性方面舉步維艱，是因為他們沒有強大的監控系統。

Workflow-oriented orchestrators like Airflow will report if your data pipeline hits an error during execution, but don't offer visibility into the quality, completeness, or freshness of the data that they're updating.

Airflow 等以工作流為導向的協調器會報告數據管道在執行過程中是否出錯，但並不提供對所更新數據的品質、完整性或新鮮度的可視性。

Teams that adopt stand-alone data reliability tools often end up abandoning them, because they're difficult to fit into practices for operating data.

採用獨立數據可靠性工具的團隊最終往往會放棄這些工具，因為它們很難融入數據操作實踐中。

Data engineers end up needing to visit both their orchestrator and their reliability tool to understand the health of their data pipeline.

數據工程師最終需要同時訪問他們的協調器和可靠性工具，以瞭解數據管道的健康狀況。

And if they want to trace data reliability alerts to the DAGs that generated the data, they need to try to integrate their data reliability tool with their orchestrator.

如果他們想將數據可靠性警報追蹤到生成數據的 DAG，就需要嘗試將數據可靠性工具與協調器集成。

This usually means contending with query-tagging schemes that are fragile to communicate with or configure and maintain.

這通常意味著要與查詢標記方案抗爭，而這些方案在通信、配置和維護方面非常脆弱。

At Dagster Labs, we believe orchestration and data reliability go hand-in-hand.

在 Dagster Labs，我們相信協調和數據可靠性是相輔相成的。

Dagster's asset-oriented approach to data orchestration enables Dagster Plus to offer a full set of data reliability features.

Dagster 面向資產的數據協調方法使 Dagster Plus 能夠提供一整套數據可靠性功能。

These help you monitor the freshness of your data, the quality of your data, and changes to the schema of your data.

這些功能可幫助您監控數據的新鮮度、數據品質以及數據模式的變化。

Dagster helps monitor both the source data that feeds your pipelines and the data produced by your pipelines.

Dagster 可幫助監控為管道提供數據的源數據和管道生成的數據。

Monitoring source data helps you find out early if your pipeline is going to run on stale, bad, or unexpected data.

監控源數據可幫助您及早發現您的管道是否會在陳舊、糟糕或意外的數據上運行。

And monitoring output data helps you ensure that the final product is on time and high-quality.

對輸出數據的監控有助於確保最終產品的準時和高質量。

At the center of Dagster's data reliability stack is a feature called asset checks.

Dagster 數據可靠性堆棧的核心是一項名為資產檢查的功能。

An asset check is responsible for determining whether a data asset meets some property, such as whether it contains the expected set of columns, contains no duplicate records, or is sufficiently fresh.

資產檢查負責確定數據資產是否符合某些屬性，如是否包含預期的列集、不包含重複記錄或足夠新鮮。

Asset checks can be executed in line with data pipelines or scheduled to execute independently.

資產檢查可與數據管道同步執行，也可按計劃獨立執行。

Optionally, Dagster can halt your data pipeline when an asset check fails to avoid propagating bad data.

作為選項，Dagster 可以在資產檢查失敗時停止數據管道，以避免傳播不良數據。

Asset checks were introduced as an experimental feature last year, and we've marked them as generally available in Dagster's recent 1.7 release.

資產檢查是去年推出的一項實驗性功能，我們已在 Dagster 最近發佈的 1.7 版本中將其標記為全面可用。