short unlike our guest, Greg, who is quite tall. And--yeah. This actually--it rose out

of a discussion we were having between some Kernel folks that I know, and Bryan [INDISTINCT]

and right over here, about sort of the nature of the embedded system--what's that?

>> It might not [INDISTINCT] >> Well, no, I mean, like from outside the

company, inside the company like--donít--I'm speaking here. All right?

>> Go ahead. >> Anyway, and so, you know, at the end of

this today, I want you to come out and speak. It would be fun for us and fun for you and

stuff. So, here we are, Greg. >> HARTMAN: Thanks. Is this working? All right.

I'm Greg Kroah Hartman, Iím a Linux Kernel developer. Iím a sub-system maintainer of

the Kernel for USB, driver Core, I used to do PCI, I like to see all the weird things.

I'm one half of the stable Kernel maintainer group which I'll talk about later. So here,

we're going to talk about the Kernel, the process, who's doing it, how itís being done,

why you should care and why Google should care and things like that. Please tackle--ask

questions in the middle, I like it. Make fun of me. I'm going to make fun of you. So, beat

them. So, first thing. This is the most important thing, donít get anything. Realize this.

This is what the Linux Kernel do for the past year and a half from the 2.6.20 to 2.6.25

Kernel. This is what was changed per day in the Kernel. This is a stable Kernel. This

is a stable development process. This is freaking insane. It is the fastest moving so--project--so--software

project known to anybody. Largest project of its kind known to anybody but it's also

created a result thatís never been done before and we have an operating system that supports

more devices than any of our operating system ever has. We support more processors than

any other operating system ever has in the history of computing. And we're doing it in

an open manner, in a very chaotic manner. Moving at a rate, that fast; shipping something

thatís stable, that everybody, including yourself, uses everyday. We broke all the

software development rules and we're continuing to break it. If that doesnít make you guys

scared, it should. The biggest thing to that make you guys--should make you guys scared

is, if you keep your code outside the Kernel, thatís going to pass you by. Do you think

you can keep up with that kind of rate of change, figure out what's going on with that

rate of change? If you get your code into the Kernel, that rate of change will happen

to you automatically. If you donít, it will blow you by. So, get your code in the Kernel.

It's the main thing I want to talk about. So, thatís it. Thatís crazy. And also, this

is the first question people ask like, "Oh, thatís jus drivers, right? Drivers are changing

that fast.î No. It's the whole Kernel. I looked at it, Iíve broke it down by the proportion

of the code--size of the Kernel. The Kernel is about nine million lines of code. The Core--the

Kernelís five percent, driverís a little over 50%, 25%. If you look at the rate of

change at each individual portion, it's exactly proportion--itís exactly proportional to

the amount of code thatís in the Kernel for that section. So, the Core of the Kernel is

changing at 5% of that rate. The driver section is changing at 55% of that rate. The architecture

specific spot is going up--what is that? I donít know, it's going at 30% of that rate.

So, it's uniform. It's flat across the whole thing. So, donít think that just--that the

drivers are changing, it's the whole thing; it's everything. So, and along with that rate

of change, it breaks down to this. It's going really fast. Three point sixty nine changes

an hour, 24 hours a day, seven days a week. During the 2.5, the 2.6 development cycle,

we're running about 2.5 changes an hour. And that was unheard of; no way that Iíd ever

thought we could do that. Since then, we've been going up. I donít know why. We're getting

faster. We're getting faster. We're doing more rate of change and it's getting bigger.

So, this is some of itís staple. So, here's our graph. If you want to see all the--that

lines of code that are added and modified, diluted over time, I just did for the 26 [INDISTINCT]

and long. If you look--if you used that, it was a little bit less. So, it's--itís really--itís

kind of staying around the same but at lines out, it keeps going up. Thatís interesting.

Thatís not really trend worthy but it is continuously going up. But itís trend worthy

is the changes per hour. It's continuing to go up. 2.6.25 changes per hour is amazing

. Well, that rate of change does not included moving files around. Moving files around the

Kernel doesnít affect it at all. It does not change at all. So, we're going faster,

doing more rate of change per hour per release. Everybody happy? Nobodyís questioning that?

Oh, come on. All right. So, how we do this? Hereís how we do this. Extensively, it starts

off at the developer on the bottom to make a change to a file, to--they find a bug, they

want to add a new driver, do something. Then it takes that change, the path, should they

send it off to the maintainer or the driver of the file. In the Kernel, we have about

600 unique maintainers that are listed on who owns what. You canít find what--who owns

what, you can look in the file using git and see who modified it last. Whoever touched

the floppy driver last is the maintainer. Never touch the floppy driver. So, that personís

responsible for looking at it and then they see it, they review it and then they sign

off by it. They say, ìThis looks good to me. I think itís okay.î And they add their

signature called Sign Off. And Iíll talk a little bit more about that later. So, the

developer says, the sign off, itís good. They say itís good. They send it off on to

the subsystem maintainer. Subsystem maintainer like USB, PCI, BFS, security, Core, things

like that. They consolidate all the patches from all data people and then they hold on

to them. They create this little mini trees everywhere. These little mini trees are either

in gits, using quilt, somebody--I think thatís all everybody uses, some tarballs maybe. And

then all that, every night gets pulled into the Linux-next Tree. Itís from Steven Rothwell

in Australia. Which--for IBM, the Aus labs. He pulls it in every night, the Linux-next

Tree merges them all together and sees if it builds or not. Sends you angry emails or

you break the bill or you break the merge, itís great. Itís awesome. Itís testing

all the subsystem maintainer stuff for all our development things. Andrew at the same

is creating--is now relying on Linux next and sucking everything in the host tree, about

once a week or so. And then also, he pulls things from the Linux Kernel Mailing List,

a sort of mailing list that happen to get drop by subsystem maintainers or whatnot,

he gets--he sucks those under his tree and everybody--and then he does a release every

couple of weeks. So thatís testing, this is the developmental stuff. So all the crazy

stuff the subsystem maintainers are holding on to, all the stuff thatís coming across

the mailing list but Andrew picks it up, gets pulled to these releases. Then when Linus

says itís the merge tree--windows open. Iíll talk more about that later. All the subsystem

maintainers hit Linus. We got two weeks to hit Linus with all the changes you want. We

all go to Linus. Andrew sends a bunch of stuff to Linus after the subsystem maintainers get

there and then we go on, the cycle releases. Iíll talk more about releases later. So,

nice little tree structure. So, Linus and Andrew and the next tree. Does that make sense?

Nice hierarchy, right? Nice and pretty. So, Iíd try to graph this. Looking at the relationship

between people and the work--way patches flow to the tree. I think itíd be a nice pyramid

or whatnot. It turned out, it isnít. I made a graph that was about 40 feet long and 4

feet tall, of one Kernel release. It was a mess. Itís a big giant spaghetti mess. It

looks like an internet routing system. And which is good because it means that, if this

person doesnít pay attention, we route the packets around them, Andrew picks it up and

then sends it to subsystem maintainer, it sends it to Linus or this person sends it

directly to there and goes around the maintainer or this person disappears for a week. So,

itís very good. Itís lossy. We can handle people disappearing, handle people come--reappearing

and it works out really well. It isnít a strict hierarchical development system which

is good and it shows how we can keep up this massive rate of change so fast. Itís kind

of scary. If you own a file or a driver, you need to accept the fact that other people

are going to be changing it, extensively, without you knowing it but hopefully you should

be paying attention and you just have to accept that so other people can do that. You can

always change it back if you didnít like it. But thatís the way we get things done.

Thatís how we can work so fast. Make sense? Cool. So, hereís a release cycle. So, Linus

does the release based on something. We say itís good. So then, we have two weeks. We

can throw anything we want at Linus. All the subsystem maintainers throw everything we

want. Extensively, itís been tested in the Linux-next Tree, before that I have been testing

the MM3 from Andrew. So, we hopefully expect that the code has been tested, it builds,

it hopefully compiles sometimes and all merges. So, thatís when the hundreds of thousands

of patches go in at once, two weeks. Linux releases an RC1 Kernel. Then the subsystem

maintainers that--were on vacation or forgot about the merge window being open, wake up

and say, ìOh, no. I forgot my stuff.î They send it to Linus again so, then goes an rc2

a couple of weeks later. Hopefully, thatís getting better. People are slowing down. Everything

happens between rc1 and rc2. Then, its bug fixes. We start doing bug fixes, we track

bug fixes, regressions, we do RC1, 3, 4, 5, whatever. Until we think things are slowing

down and good enough and boom when it says, ìOkay. This oneís good. Out the door.î

And then we start the whole cycle over again. There is no stable and unstable development

trees anymore. You step odd was unstable and development even was stable, not anymore.

For the past four years, weíve been doing this model. When we finally realize that this

model really worked good, we said, we should try and do release every two to three months,

right? If you look, weíre releasing on an average of every two and three quarter months,

for the past three to four years, like Clockwork, you can trust that a new Kernel will come

out between two and three months. This is a really, really good thing. Unless the up

stream just rose, figure out in their schedules when things are going to be done within a--within

a reasonable time frame that can plan when things are going to come out. The [INDISTINCT]

can figure that out, developers can figure that out, ìI miss this merge window, whenís

the next one going to come around?î You don't have to worry that their selves are going

to get drop for forever. It works really, really well. One thing that happened though

is we realized during these two to three month time, what's happens about security bugs?

These are people that actually use the Kernel.org--Kernel. What happens then? Thereís bugs, security

problems happen, what are we going to do? So, we formed--two people formed the stable

tree which is me and Chris Wright who works for Red Hat and we do security updates or

bug fixes; 0.1, 0.2, 0.3, 0.4, 0.5 and we have a real strict list of rules of what we

will accept, what we won't accept. It's pretty easy. It has to be bug fixes only and security

fixes. The patches have to be in the upstream tree already, unless thereís some really,

really odd thing, things like that. It must be small, obvious. We do those releases every

couple of weeks, whenever we get around to it. Sometimes we're slacking, sometimes we're

not. Security updates, we can do it usually if we got an inbox. We can do it in couple

of hours. So, we release things as soon as possible if we can. And that's working out

really, really good. The fun things is, after the--we drop them, after 2.6.20 comes out,

it might be one more 2.6.1.9 stable and then we stop. We don't have to do it. Weíll have

to maintain that for forever. You can maintain it for forever if you want to. Somebody picked

up the 2.6.1.6 Kernel, decided to maintain it on his own for a long, long time. We all

thought he was crazy. He got a job out of it. So, maybe he wasnít so crazy at all.

So that he could do things over time. So, it worked out really well. If you want--if

anybody wants the 2.6.2.4 Kernel to maintain it for a while, let me know. Weíll give it

to you. We don't care. So, that's how it works. If itís stable, releases happen, development

and releases happen and then we drop stable trees. So, we don't have to maintain them

for forever, which given our rated change previously is a good thing to do. Makes sense?

Everybody happy with that? Is it working? All right.

>> So, you guys don't have an open analysis [INDISTINCT] every blogger really preferred

at least [INDISTINCT] >> HARTMAN: We don't have a notion of a preferred

release anymore, no. Every release is a preferred one, the latest one. Based on the rate of

change, if you report a bug--a bug problem on 0.2--on 0.2 Kernels ago, it's given how

fast Kernelís often changing. We're going to ask you, can you reproduce this on the

current version? It's the first thing we're going to ask you. So, yeah. So, everythingís

preferred. The last one is preferred. Now, note when the distros do releases, we know

what Kernels are going to release. So, sometimes those Kernels hang around a little bit longer.

Fedora and OpenSUSE are based on 2.6.2.5 right now. Odds are that Kernelís going to hang

around for a little while because Chris Wright works Red Hat and I work for SUSE. So, weíll

probably want to hold on to that Kernel for a little while longer, make sure bug fixes

get in there and it's good. But, if you want the new features and things like that, you

should go to the latest one. [INDISTINCT] Yeah.

>> Why use the two-week merge window, why not the [INDISTINCT]