Name: 2 1 分佈式文件系統 15 50 (2 1 Distributed File Systems 15 50)
Uploaded: 2021-01-14T07:02:42.000Z
Duration: 15 min 51 s
Description: 【看影片學英語】數萬部 YouTube 影片，搭配英漢字典即點即查，輕鬆掌握單字發音與用法，長久累積看電影不必再看字幕。

I'm Anand Rajaraman and today's topic is Map-Reduce.

In the last few years Map-Reduce has emerged as a leading paradigm for

But before we get into Map-Reduce proper, let's spend a few minutes trying to

understand why we need Map-Reduce in the first place.

Now we're all familiar with the basic computational model of CPU and

The algorithm runs on the CPU, and accesses data that's in memory.

Now we may need to bring the data in from disk into memory, but

once the data is in memory, fits in there fully.

So you don't need to access disk again, and

the algorithm just runs in the data that's on memory.

Now there's a familiar model that we use to implement all kinds of algorithms, and

big, that it can't all fit in memory at the same time.

Look at the disk in addition to looking at CPU and memory.

you can only bring in a portion of the data into memory at a time.

And you can process it in batches, and you know, write back results to disk.

And this is the realm of classical data mining algorithms.

But sometimes even this is not sufficient.

So think about Google, crawling and indexing the web, right?

Let's say, google has crawled 10 billion web pages.

And let's further say, that the average size of a web page is 20 KB.

Now, these are representative numbers from real life.

Now if you take ten billion webpages, each of 20 KB,

Now, when you have 200 TB, let's assume that they're using

the classical computational model, classical data mining model.

And all this data is stored on a single disk, and

we have read tend to be processed inside a CPU.

Now the fundamental limitation here is the bandwidth,

the data bandwidth between the disk and the CPU.

The data has to be read from the disk into the CPU, and

the disk read bandwidth for most modern SATA disk representative number.

So, so we can read data at 50MB a second.

How long does it take to read 200TB at 50MB a second?

the answer is 4 million seconds which is more than 46 days.

Remember, this is an awfully long time, and

is just the time to read the data into memory.

To do something useful with the data, it's going to take even longer.

You can't take four to six days just to read the data.

Now the obvious thing that you think of is that it can split the data into chunks.

And you can have multiple disks and CPUs.

you, you stripe the data across multiple disks.

And you can read it, and, and process it in parallel in multiple CPUs.

For example, if you had a 1,000 disks and CPUs, in four thousa-,

And we were completely in parallel, in 4 million seconds, you could do the job in,

4 million by 1,000, which is 4,000 seconds.

And that's just about an hour which is, which is very acceptable time.

this is the fundamental idea behind the idea of cluster computing.

this tiered architecture that has emerged for

cluster computing is something like this.

You have the racks consisting of commodity Linux nodes.

As you go with commodity Linux nodes because they are very cheap.

And you can, you can buy thousands and thousands of them and, and rack them up.

Each rack has 16 to 64 of these commodity Linux nodes and

and, the, the, the switch in a rack is typically a gigabit switch.

So there's 1 Gbps bandwidth between any pair of nodes in rack.

Of course 16 to 64 nodes is not sufficient.

the racks themselves are connected by backbone switches.

is a higher bandwidth switch can do two to ten gigabits between racks.

Right? So so we have 16 to 64 nodes in a rack.

And then you, you rack up multiple racks, and, and you get a data center.

So this is the standard classical architecture that has emerged over

For you know, for storing and mining very large data sets.

Now once you have this kind of cluster this doesn't solve the problem completely.

Because cluster computing comes with it's own challenges.

But before we get there, let's get us, you know, ideal of the scale, right?

In 2011 somebody estimated that Google had a million machines,

In stacked up you know, is, is somewhat like this.

So, so it gives, so that gives you a sense of the scale of modern data centers and,

This is what, it looks like inside a data center.

So the, the, what you see there is, is the back up racks, and

you can see the connections, between, between the racks.

字幕列表影片播放

2 1 分佈式文件系統 15 50 (2 1 Distributed File Systems 15 50)

individual

assume

scenario

massive