Name: Delta Lake 是什麼？🤔 (What is this delta lake thing?)
Uploaded: 2025-12-21T12:45:20.000Z
Duration: 6 min 58 s

And in this video, we're going to talk all about Delta Lake and Delta Tables.

在這支影片裡，我們要來聊聊 Delta Lake 和 Delta Tables。

If you find this for the very first time, be sure to hit that subscribe button to stay up to date from all the videos from both Adam and this guy.

如果你是第一次看到我們，記得按下訂閱按鈕，隨時掌握 Adam 和這位仁兄的最新影片。

All right then, thank you so much for coming back.

我們在那支影片上獲得了很多迴響。

And in this video we're going to talk about this Delta Lake, these delta tables.

而這支影片我們要來聊聊這個 Delta Lake，還有這些 delta tables。

Just continuing the video series of building this modern lakehouse data warehouse.

繼續我們之前建立 modern lakehouse data warehouse 的系列影片。

所以，請告訴我們，什麼是 Delta Lake？

So Delta Lake is files that is changeable.

Delta Lake 其實就是可以修改的檔案。

While previously you would have a parquet file and that parquet file cannot be changed, a delta table builds a framework around it which allows you to update, insert or delete values to it and make it asset compliant.

以前你會有一個 parquet file，那個 parquet file 是不能修改的，但 delta table 在它周圍建立了一個框架，讓你能夠更新、插入或刪除裡面的值，並且讓它 asset compliant。

Consistency、isolation 和 durability。

So it brings all of the asset compliancy to a parquet file, like in a relational database.

所以它會把所有的 asset compliant 都帶到 parquet 檔案裡，就像在關聯式資料庫一樣。

All right, so why would I use this delta though?

好的，那為什麼我會想用這個 delta 呢？

So why wouldn't I just sage it up into a file system or into a database and override it and then select from it?

那為什麼我不能直接把它儲存到檔案系統或資料庫裡，然後覆蓋它再從裡面選取呢？

Why wouldn't I just do it the old school way?

我為什麼不能就用老方法就好？

If you would like to have incremental changes to your data in a data lake, you need to use delta.

如果你想在 data lake 中對你的資料進行增量變更，你就需要使用 delta。

There's a file as such is called immutable.

You need to do insert updates and deletes against it.

你需要對它進行 insert、update 和 delete 操作。

The framework from Delta allows you to do that.

Delta 的框架讓你可以做到這一點。

It allows you to have the full history of any changes that are happening to your table inside that data lake, which allows you to basically look back in time as well.

這讓你能夠完整記錄 data lake 中表格的所有變更歷史，這樣你就可以回溯時間軸了。

If I'm loading a fact table instead of reloading the entire table, I can look at the delta and only pull the changes.

如果我要載入 fact table，而不是重新載入整個表格，我可以查看 delta，只抓取變更的部分。

If I'm loading my dimensions, I could use a delta to say give me the new rows for my dimension.

如果我要載入 dimensions，我可以使用 delta 來指定給我新的 dimension 列。

If I'm using the type 2, I can say expire the old row and look what's there and give me this new row and we'll talk about dimensions and facts and what they are and stuff like that in some upcoming videos.

如果我使用 type 2，我可以說讓舊列失效，然後看看裡面有什麼，並給我新的列，我們會在之後的影片中討論 dimensions 和 facts 以及它們是什麼等等。

We need to know how to create these delta tables.

我們需要知道如何建立這些 delta tables。

我們到 Stan 的筆記型電腦那邊看看。

So we're back into our SYNAPSE analytic workspace and we're going to go head over to the Develop tab.

所以我們回到 SYNAPSE analytic workspace，然後我們要前往 Develop tab。

You can see this thing called Notebooks and we've got this delta table initialization notebook.

你可以看到這個叫做 Notebooks 的東西，我們有這個 delta table initialization notebook。

We're going to first load that data frame based upon the data we loaded last time out of that SQL Server, which is our fact movement, data here format parquet.

我們會先載入這個 data frame，這會基於我們上次從 SQL Server 載入的資料，也就是我們的 fact movement，資料格式是 parquet。

And we're going to load it to a data frame.

然後我們會將它載入到一個 data frame。

Takes about 1 minute and 26 seconds just to start the session and load that data frame.

啟動 session 並載入該 data frame 大約需要 1 分 26 秒。

And so once we've loaded that data frame, we can just use that data frame to write it to a delta table, which is really not that hard.

所以一旦我們載入該 data frame，我們就可以直接使用該 data frame 將它寫入 delta table，這真的不難。

你只需要說我們的 data frame dot write。

The format is going to be delta mode is override.

格式將是 delta，模式是 override。

Override just basically means if there's already a folder there that is delta, drop it, recreate that folder, and

Override 基本上就是說，如果那裡已經有一個資料夾是 delta，就刪除它，然後重新建立那個資料夾，並且

then save it at display specific location.

And you can see that we're specifying that lake house silver here, Delta table fact movement.

你可以看到我們指定了 lake house silver 這裡，Delta table fact movement。

If you're not familiar with notebooks and Spark, you should go watch the video we did talking about what's a notebook and what is spark and why you would use Spark, which is some cool stuff.

如果你對 notebooks 和 Spark 不熟，建議你去看看我們之前做的影片，裡面有講 notebook 是什麼、Spark 是什麼，以及為什麼要用 Spark，這些都很酷。

But in that bronze layer, Stan, that's our staging.

不過在 bronze layer，Stan，那就是我們的 staging 區域。

That's where we just grabbing all the data from our source, we drop it in that bronze layer.

我們就是把所有資料從來源抓進來，然後丟到 bronze layer。

But then just now in your Pyspark script, I saw you, I'm guessing you're initializing the delta, like creating the delta for the first time and then moving that data to the silver layer.

但剛剛在你的 Pyspark 腳本裡，我看到你，我猜你是在初始化 delta，也就是第一次建立 delta，然後把資料移到 silver layer。

So normally you would do some steps in between where you're going to cleanse and enrich that data to move it to that silver layer.

所以通常你會做一些中間步驟，像是清理和豐富資料，然後再移到 silver layer。

We knew that the data was okay, so we're just initializing that delta table there as it is.

我們知道資料沒問題，所以就直接初始化那個 delta table。

And that basically is in a format called delta, where we can update, insert and delete that values to any data that's coming in.

這基本上是一種叫做 delta 的格式，我們可以更新、插入和刪除進來的任何資料。

We're going to push to that same format that we have in Silver.

我們會推送到和 Silver layer 相同的格式。

So everything in Silver will be the same format.

所以 Silver 裡的一切都會是相同的格式。

它可以是我們想要的任何格式。

And then from that silver layer, we go to that gold layer.

然後從 Silver layer，我們會到 Gold layer。

And this is the layer that our Power BI users would connect to to consume this data or data scientists will connect to to use all that data.

這就是我們的 Power BI 使用者會連線來取用這些資料的層級，或是資料科學家會連線來使用所有這些資料的層級。

So bronze is our raw data, our silver is our delta where we enable it so we can do all of our time travel and stuff.

所以 Bronze 是我們的原始資料，Silver 是我們的 delta，我們啟用它，這樣我們就可以進行所有的 time travel 和其他操作。

And then we have our goal layer, which is, you know, the serving layer for all of our report consumers.

然後我們有我們的目標層級，也就是，你知道的，所有報表使用者的服務層級。

All right, so I'm not familiar with this Pyspark.

好的，所以我不太熟悉這個 Pyspark。

I'm a SQL guy, and you guys are talking about how flexible SYNAPSE is.

我是 SQL 的人，你們一直在談論 SYNAPSE 有多靈活。

Is there any way that I could just do SQL instead of learning PySpark?

有沒有辦法讓我只用 SQL，而不是學習 PySpark？

That's a cool thing because that Spark engine can also read SQL.

這很酷，因為那個 Spark 引擎也可以讀取 SQL。

Okay, so here you see that we created that data frame into a delta table.

好的，所以這裡你們可以看到我們把那個 data frame 建立成一個 delta table。

Now, if I want to use the SQL language here, I can basically use SQL language as well to do the same statements I did above.

現在，如果我想在這裡使用 SQL 語言，我基本上也可以使用 SQL 語言來執行我上面做的相同語句。

So you can write it in PySpark, you can write it in SQL even.

所以你可以用 PySpark 寫，甚至可以用 SQL 寫。

Net, there's different languages you can use to write this.

所以，你可以用不同的語言來寫。

We need a little bit of Pyspark code to turn this data frame that we have here into a temp view.

我們需要一點 Pyspark 程式碼，把我們這裡的 data frame 變成一個 temp view。

And that basically makes it accessible from the SQL language inside Spark.

這基本上讓 Spark 裡的 SQL 語言可以存取它。

Once we've done this, we can now write SQL code.

一旦我們做完這個，我們就可以寫 SQL 程式碼了。

The first thing you need to do is specify percent percent SQL to say, hey, this is a SQL code cell.

第一件事，你需要指定 percent percent SQL，告訴它這是一個 SQL 程式碼儲存格。

If you've already created the table, you can say drop table if exists, drop database if exists, create a database which is called Kyne Cube.

如果你已經建立了表格，你可以說 drop table if exists，drop database if exists，建立一個叫做 Kyne Cube 的資料庫。

Create that table, add kyemecube with the dot fact movement using delta at this specific location.

建立那個表格，加入 kyemecube 加上 dot fact movement，使用 delta 在這個特定的位置。

You can see here that we're also specifying that silverware and selecting from fact movement, which is that temp view we created over here.

你可以在這裡看到，我們也指定了那個 silverware，並且從 fact movement 查詢，也就是我們在這裡建立的 temp view。

And once we've done this, you can just say, hey, give me another SQL cell and select from that data.

然後我們做完這一步之後，你就可以直接說，嘿，給我另一個 SQL 儲存格，然後從那個資料裡選取。

So from that gym Q fact movement, you can see you can read that data inside that notebook.

所以從那個 gym Q fact movement，你可以看到你可以在那個 Notebook 裡讀取那個資料。

And then one more really cool thing is remember that we created a database here.

然後還有一個很酷的地方是，記得我們在這裡建立了一個資料庫。

That database will also be underneath your lake databases over here, and

那個資料庫也會在你的 lake databases 底下，然後

that will contain that table called Fact movements.

它會包含那個叫做 Fact movements 的表格。

That metadata is synced across your SYNAPSE workspace, which makes it really easy to work with from there on.

那個中繼資料會同步到你的 SYNAPSE workspace，這讓之後在那裡工作變得非常容易。

So instead of doing all that SSIs and staging and stuff like that, I can use Data Factory to quickly just, you know, dynamically, like we did in the first video, dump all the data into the stage environment.

所以，與其做所有那些 SSIs 和 staging 之類的，我可以用 Data Factory 快速地，你知道的，動態地，就像我們在第一個影片裡做的那樣，把所有資料都丟到 stage 環境。

Then I can write some scripts, some pyspark or even some SQL.

然後我就可以寫一些腳本，一些 pyspark 甚至是 SQL。