Everybody welcome back to another data analysis last night.

It's science tutorial with python and pandas were gonna be continuing our work with the avocado data set before we leave it for a new and exciting data set in the next tutorial.

Just a few more things I want to show with the avocado data set.

Uh, especially just issues you might run into overtime.

So let's go ahead and jump in s.

So the first thing we're gonna do is we're just gonna basically recreate where we were import pandas as p d d f equals p D ri PT Read C S v data sets of kado dot c S v Albany D f equals D f eagles DF where d f region is all Benny.

Not all Pandey often e d f dot set index, as dates in place is, he will be true in there.

Finally, uh, Albany DF don Actually, we're just gonna do average price price dot plot And I didn't look up the command, uh, soldiers from this twice and there we have it.

So pretty graph.

But there's a few issues that we're having right out of the gate.

The first issue is these, like, these dates are running over each other.

And when you see something like this, it probably means pandas doesn't actually realize it's a date.

So the first thing that I would do is convert that to a date time.

So when we read in the C S V, if you actually have a date, the pride of the big thing you'd want to do is go ahead and just say, D f date or whatever the column name is equals p dot to daytime, Uh, and then d f date.

So, um, this will probably try to think we've done done one in the 1st 1 But I don't think so.

Is our first time where we're actually reassigning the value of that column.

This is No, this is no different than any other function.

Really like.

You can just map functions to the to the column here like that.

The other thing you could use, like map and apply and stuff like that, and we'll talk a little more about that.

But this is a nice built in pan dysfunction that's gonna convert this column.

Um, and then we're just reassigning that value because this is actually just gonna return a bunch of values for us.

So pretty cool.

Anyway, uh, now what if we go to graph that we could see?

Okay, they're nice and slanted, and it appears that pandas actually knows.

Hey, that these air dates So awesome.

So that's one thing.

The next thing is, this graph is crazy, crazy, busy looking.

So the first thing I would think of to smooth it out is to use some sort of moving average.

So, um, if you don't know what, like a moving averages, it's like any point along the way.

Let's say we're gonna do a 25 moving average, so we'll take every point we're gonna say, Okay at this point.

So this point and the previous 24 points, what's the average?

So 25 point average.

And then we go one point over and do the exact same thing.

We just keep slowly rolling.

That and what that's gonna wind up doing is convincing these like crazy fluctuations, and it will hopefully smooth things out a little bit.

So the way the entity that is Albany d f on, then we'll do average prize and then dot rolling.

So this is like, any rolling thing.

Then you would pass a number for the window and then what kind of rolling?

So you could say it rolling some, or in this case, we're gonna say a ruling mean and then we're gonna plot it Cool, except not cool, because this is pretty ugly.

Something went wrong.

So the first thing I would think of when I would see a chart like this is that possibly things were out of order.

So what if we said all Benny D f don't let's do a head, uh, 25.

So when we look here, we could see Okay, we start.

So it's in reverse order, and then we scroll down.

And at least in this case, it does look like everything is in proper order.

Uh, let's do dot index, but you can see here at least right out of the gate.

Um, the index does start at late December and then goes down and reverse.

So you would think, Oh, this is reverse chronological.

But apparently it's not because then we have 2018 which again goes in reverse chronological order.

So something is wrong.

So the first thing that I would do is like, Okay, let's make sure these dates are improper.

Order.

So what I would say is, um let's do all Benny D f dot sort index in place evils troop.

You can also sort data frames by specific columns and that will also adjust the index.

Just for the record, we do get a little bit of a warning here.

Uh, I will talk about that warning in a little bit.

So all I want to do now is actually just sort that thing.

Craft that again.

Okay.

And now we get a pretty graph, and I guess the warning doesn't show.

I don't know what you've already kind of sort of the index and stuff.

Anyway, we'll see the warning soon enough.

So now we get a nice, smooth out graft That actually is a little different than the other graph.

There are kind of like these two humps, but the second hump is quite a bit higher than the first.

Where is in that first graph?

It was clearly like, this one is clearly wrong.

And then even this one looks to be pretty darn wrong.

So, uh, pretty cool.

Uh, Now we got that fixed.

So the next thing that I want to dio is what if we actually want this to be a column in our data frame?

Because we can later output this data frame to a C S V again.

And it doesn't have to look exactly the same as it came in.

It could be anything.

So we decided, Hey, this is a valuable meaningful column.

So, actually, what I want to do is copy this column, come down here and I say Albany D f.

And I'm just gonna sign a new name.

I'm gonna say price 25 m A is equal to that value.

Okay, so we'll run that.

See that warning again?

I will explain that momentarily Albany DF dot head.

And then we'll just do like three here and we can see him initially.

We do see Nan's.

That's because tthe e rolling periods required 25.

So the 1st 25 are gonna be nan because there's nothing to calculate later you can set.

I think it's literally men periods or something like that, and you can overcome that.

Otherwise, what we could say is all been e d f dot tale three and boom.

We've actually got values alternatively, um, will almost certainly use this again in the future.

But you can say Albany DF dot dropping a for example on.

Then we'll just say dot head three.

And in this case, what dropping is gonna do for us is remove any rose that have any values and a not any or any Anais.

Anyways, if removes, um, Vienna's so later if you have, like, gaps for whatever reason, or you just want to drop missing data for what N a n a N is not a number.

So anything that doesn't have values, you can just drop them.

And so you could do that.

And you can also drop in a in place.

It was true.

If you want, I'm not gonna drop him here, but just know that you can.

So now that we've done that, let's talk about this air.

So first of all, it's not in there.

I can't believe I just called in an air.

Smart part of my point is it's not in there.

Um, so a little bit of ah, warning pandas is really happy to serve warnings like you're going to see tons of warnings from pandas.

Same thing with psychic learn we're gonna use.

I can't learn a little bit later in, like part five, probably in these two libraries.

For some reason, I just love to issue warnings about like, deprecation is possibly, like future double deprecation way far in the future.

Uh, so you'll see the warnings for, like years.

And anyway with pandas, What this warning is telling you is a value is trying to be set on a copy of a slice from a data frame and then it says, Try using dot Lok.

Um, I don't think that's actually useful.

I don't usually warnings are pretty useful.

That will tell you about future deprecation, tze and new functions that you should use or whatever so read them.

Um, but I don't really think this is useful.

Personally, I don't think that's the right way to do it, But anyway, the warning still stands.

Hey, it's just letting you know you're doing something that might screw you up later.

So basically what it's telling you is like, as you begin to do work like we've actually kind of forked the original data frame into this Albany data frame and they were changing values.

And what Panas is trying to tell us is like if you were just to go perfectly in order and on Lee modify a data frame and always keep it D f you would never have a problem.

But if at some point you fork and then you start changing values over here, in here, in here, well, don't forget that that original it came from that original data frame.

And you might be either one changing values that you didn't expect or not changing values that you did expect.

So that's what that warning is there to tell you.

So how could we not see this warning?

Because later you might be doing things and you might be iterating, and then you'll see like hundreds of these warnings.

So one way that you could do is by telling pandas I get it.

It's a copy.

So how could we do that?

Well, I'm so glad you asked Joan Allman, E d f equals d f dot copy.

And now what that does is returned to us a new copy of the data frame, because later we could modify that data frame.

Um and it won't impact Albany DF but pandas wants us to know it won't impact Albany DF.

So instead, we just do DF copy which returns the data frame.

So now we can reference that, you know, we can treat it just like a data for him right away.

So we could say what d f dot copy?

Where d f, um, region region equals Albany.

Then we do all the same things that we did before.

So, for example, we can take this, and, uh, I can't remember when we actually okay, so we set the index afterwards, So let's go ahead and set the index.

And we also sorted the index.

I believe afterwards.

So set index, sort index.

Let's just run that real quick.

Okay?

Cool.

Uh, and now we don't have any warnings.

So basically, I just wanted to show you guys how you get away with, you know, not sure in the mornings.

Okay, so we've got this graph Now, the next thing you might think about is like, Hey, I want to graph all the regions on one graph and see them compared.

Okay, so how would we do that?

So the first thing we we need to do is somehow iterated over all the regions butts.

We have to figure out all of the region's.

So the cool thing about pandas is at the end of the day, it is rows and columns, and you're Each individual column is just an array.

So Pan is at the end of day can be an array, you know, a multi dimensional array homicide, array of a raise, like a list of lists.

Anyways, um, it could be a multi dimensional, right?

So And how might that happened, by the way, uh, for example, because they left out values boom.

And now you can see this has actually been converted to an array.

We could do the exact same thing with a column so we could say, D f region dot values and we see we get an array.

Now it's, uh, just doesn't give us the length.

But what we can say is, Well, it's gonna be the length of the data from which I think was like, 18,000 or something.

It's not gonna be interesting to know that.

So Wayne also convert it to a list to list, and then you can see okay, We got quite a bit values here.

Too many values.

So how do we get, like, just the unique items?

While one option would be quite literally just converted to a set and now we can see.

Okay, these are all the regions, and then we could convert it to a list two, and then we could generate over that list.

So Okay, the problem is, that's messy, and there's a simpler way.

So a lot of times in panda's, there's just an easier way than whatever we're doing.

So in this case there is it will be DF region dot unique, done.

So on this way we can get all the unique values.

And then we could actually generate over these.

And for that reason to, uh, don't forget about the panties docks scroll through the docks.

Just get an idea for all the things that are available to you before you start trying to do it in python and moving it back because again, if you can keep things in panda's, it's probably gonna be faster than what you write.

What you write will may end up with the exact same outcome, but pandas is likely painters is using c++.

You're gonna be using python, which is slow.

So So Yeah.

Okay.

Do you have region unique?

Cool.

So we have the unique values.

Now, how might we graph every single one?

So long pants are on Metal it lib.

Basically, you've got this canvas in the background and then any time you plot something, it just kind of ads.

But we're in this, like Jupiter notebook.

So it's gonna like not yet.

Well, it's Jupiter lab now, but anyways, uh, probably what we want to do is actually create a new data frame, and they just plot that data frame.