字幕列表 影片播放 列印英文字幕 Welcome to this next class in Pattern Recognition, we have been looking at density estimation. So, let us briefly recall, what we have been doing in the last couple of classes. We have been looking at, how to estimate densities for given IID samples, for a particular density. We first looked at the maximum likelihood estimation method. For the last couple of classes, we have been considering the Bayesian estimation of density function. So, given a density function f x given theta where, theta is the parameter, we are considering Bayesian estimate for the parameter theta. As I have already told you, the main difference between the maximum likelihood estimation and the Bayesian estimation is that, in the Bayesian estimation, we look at the parameter theta, which may be a vector or a scalar, the parameter theta itself is viewed as a random variable. And because, we view it as a random variable, it has a prior density, which captures our initial uncertainty. Our knowledge or lack of knowledge about the specific value of the parameter is captured through a prior density f theta. So, f theta gives us some idea about, what we think or the possible values for the parameter. Given the given the prior density, we are going to use the data likelihood that is, f D given theta to calculate the posterior density f theta given D. Once again, I would like to drive your attention to a caution on the notation, for simplicity, the densities of all kind of random variables we are using the same symbol f. So, f of theta, f of D given theta, f of theta given D, all these are densities, purely as mathematical notation looks same because, f is the same function. But, we are we are using f as a notation to denote density and density of, which random variable it is, is clear from context. Thus, f x given theta is the density of x conditioned on theta, which is the parameter f of theta, is the density of the parameter theta and so on, f theta given D is the conditional density of theta, conditioned on data D. So, even though we are using the same symbol f, for all densities so, I hope you understand that, the f used in different times is a different function. It refers to densities of different random variables and just to keep the notation uncluttered, we are calling it as the f. So, once again, essentially we start with a prior density f of theta, for the parameter theta then, use the data likelihood of D given theta to calculate the posterior f theta given D, we have seen a couple of examples of this process earlier. And the idea of Bayesian estimation by now, you would have seen is, to choose a right kind of prior, the right kind of prior for us is, what is called the conjugate prior. The conjugate prior is that prior density, which results in the posterior density belonging to the same class of densities. For example, as we saw when, we are estimating the mean of a Gaussian random variable or mean of a Gaussian density where, the variance is assumed known, we choose Gaussian density for the prior then, the posterior is also Gaussian density. So, for that particular estimation problem, the prior density happens to be Gaussian similarly, for a Bernoulli problem where, we have to estimate the parameter p namely, the probability of the random variability taken value 1. For parameter p, the appropriate prior density turns out to be beta appropriate in the sense that, if I take prior density to be beta then, the posterior also becomes beta, so such a prior is called a conjugate prior right. The conjugate prior is that prior density, which results in the posterior density also, to be of the same class of densities, the use is of conjugate prior makes the process of calculation of posterior density easier. As we have seen let us say, in the case of the Bernoulli parameter, that we have seen earlier, if we start with some beta a 0, for the beta a 0 b 0 for the prior then, the posterior is also beta density with possibly some other parameters a and b n. So, calculation of the posterior density is simply a matter of parameter updation, given the parameters of the prior now, we update them into parameters of the posterior. Having obtained the posterior density, we can finally use either the mean or the mode of the posterior density at the final estimate. We have seen examples of both or we can also calculate f of x given D, that is the actual class conditional density, conditioned on data by integrating the posterior density and we have seen example of that also. So, this class, we will look at a couple of more examples of Bayesian estimation and then, closed Bayesian estimation. As you would have by now seen, Bayesian estimation is a little more complicated mainly because, you have to choose the right kind of prior and different kind of estimation problems make different priors of the conjugate. So, we will start with another example, this is the multinomial example that is, we consider estimating the mass function of a discrete random variable, which takes one of M possible values say, a 1 to a M where, p i is probability Z takes the value a i. So, essentially Z takes value a 1 with probability p 1, a 2 with probability p 2, a M with probability p M and we want to estimate this p 1, p 2, p M, given a sample of n iid realizations of Z. We already considered this problem in in the maximum likelihood case and there we told you, this is particularly important in certain class of pattern recognition problem. Especially those to do with say, the text classification and so on where, the discrete random variables for feature that is, features that take only finitely many values are important. We have seen, how to do the maximum likelihood estimation for obtaining this p 1, p 2’s that, p 1, p 2, p M that characterize the mass function of this discrete random variable Z. Now, we will look at, how to do the same thing using Bayesian estimation, I hope the problem is clear, this already is done earlier so, we will we will quickly review the notation that we used earlier. So, as earlier, we represent any realization of Z by M-dimensional Boolean vector x, x has M components x superscript 1, x superscript M. The transpose there is because, as I said, all vectors for us are column vectors, each of these components of this vector x, x superscript i is either 0 or 1, and summation of all of them is 1. That means, essentially x takes only the unit vectors 1 0 0 0, 0 1 0 0’s and so on, the idea is that, if z takes the i th value a i then, we represent it by a vector where, i th component is 1 and all others are 0. We have already seen that, this is a interesting and useful representation for maximum likelihood with this same, we use the same representation here. And now, p i turn out to be, the probability that the i th component of this vector is 1 that is what, I wrote there p x subscript i is equal to 1. Because, p i is the probability with where, Z takes the i th value a i and when Z takes the i th value a i, the i th component of x i becomes 1 where, i th component of x becomes 1. Also, as I told you last time, the reason, why we use the superscripts to denote the components of x is because, subscripts of x are used as to denote different data or data is x 1 to x n that is why, we are using superscripts to denote the components of particular data. So, we also seen last time that, for this x, the mass function with the single vector parameter p, is product i is equal to 1 to M p i x i because, in any given x, only one component of x is 1 and that is the one that, survives this product. So, if x is 1 0 0 0 then, for that x, f of x given p will become p 1 to the power 1 and p to the power 0 and so on that is, p 1. So thus, this correctly represents the mass function, that we are interested in and p is the parameter of the mass , that we need to estimate. As usual, our data has n samples x 1 to x n, and each sample x i is a vector of M components, the components are shown by superscripts. And each component is either 0 or 1, and in each data it is M x i that is, each M vector x i, if I sum all the components, it becomes 1. That simply means, because, each component is on 0 1 and sum is 1 means, exactly one component is 1 and all others are 0 so, this is the nature of our representation and we have n such data items. So, given such data, we want to estimate p 1, p 2, p M thus essentially, what we are doing is, we are estimating the parameters of a multinomial distribution. As you know, a multinomial binomial takes only binomial is important, when there is a random experiment that is repeated, which takes only two values success or failure. In the multinomial case, it is the same thing independent realizations of a random experiment that takes more than two values say, M values. If a random variable takes M different values, I can think of it as a random experiment, which can result in one of M possible outcomes right. So, many samples from that random variable are like, I have a multinomial distribution that is, I repeat a random experiment, that can take one of M possible outcomes, n number of times. So, some of which will be well result in first outcome and so on, some of which will results in second outcome and so on. So, I know for each repetition, what outcome has come and given those things, we want to estimate the multinomial parameters p 1, p 2, p M. Now because, we are in the Bayesian context, the first question we have to answer is, what is the conjugate prior in this case right. Now, as we have already seen from our earlier examples, for this we should examine the form of the data likelihood. So, we already have our model, we for this x, we have the mass function, that we that we have seen earlier that is the mass function so, given this mass function, what is the data likelihood, that is easy to calculate. So, f D given p, p is product of this over i is equal to 1 to n now, we can substitute for f of x i, if I substitute for f of x i, f of x i is once again product p j x i j. Now, if I interchange i and j summation then, p j to the power x i j product over i can be written as product over j p j to the power n j where, n j is summation over i x i j. What does that mean, in any given x i, the j th component is 1, if that particular outcome of Z represents the j th value of Z. So, this n j tells you, out of the n, how many times Z taken the j th value so, for example, n 1 plus n 2 plus n M will be equal to n, the total number of samples. So, out of n samples, n 1 times the first value of Z has come, n 2 times the second value of Z has come or looking at as a multinomial distribution, n 1 times the first outcome has occurred, n 2 times the second outcome has occurred and so on. So, now, in terms of this n’s, the data likelihood is given by product over j, p j to the power n j. Now, if this is the data likelihood, we multiply this with a prior and we should get another expression of the same form, as the prior so, what should be our prior. So, this is the data likelihood so, we expect the prior to have a some density, this proportional to a product of p j to the power a j. If the prior is proportional to product of p j to the power a j and then, I multiply with data likelihood, I get another product of p j to the power some a j prime so, once again that posterior will belong to the same density of the prior right. Let us still remember that, p is a vector parameter, p has M components with all of them are probabilities so, they are greater than or equal to 0. And sum of p a is equal to 1 because, p 1 is the probability of that Z takes first value and so on so, this is needed for the mass function of Z. So, we need a density, which which is a density defined over all p, that satisfy this and that should have a form, which is product p j to the power a j. Now, such a density is, what is known as a Dirichlet density, if you remember when there are only two outcomes when you looking at Bernoulli, the the prior happen to be what is called the beta density. So, we will see that, the Dirichlet density is a kind of generalization of the beta density so, the Dirichlet density is given by f of p, is gamma of a 1 plus a 2 plus a M by gamma of a 1 into gamma of a 2 into gamma of a M, product j is equal to 1 to M p j to the power a j minus 1. Where, this gamma is the gamma function, that we already seen when we discuss the beta function the beta density, gamma function gamma of a is integral 0 to infinity x to the power a minus 1, e power minus x d x. In this, the the parameters of this Dirichlet density or this a j’s that is that is, a 1 a 2 a M, all of them are assumed to be greater than equal to 1. Also, this density has this value, only for those p that satisfy all components greater than or equal to 0 or some of the components is 1, outside of those p’s the density is 0. That means, the density is concentrated on that subset of power M, which satisfies p i greater than or equal to 0 and summation p equal to 1, which is essentially called as simplex. Those you know what is simplex is, they are simplex but, anyway even, if you do not know what is simplex is, this density is non-zero only for those p’s that satisfy p i greater than or equal to 0, summation p i is equal to 1 otherwise, the density value is 0. So, that is the Dirichlet density so, if M is equal to 2, this becomes gamma a 1 plus a 2 by gamma a 1 into gamma a 2, and p 1 to the power a 1 minus 1 and p 2 to the power a 2 minus 1 and that is the beta density. So, when M is equal to 2, this density becomes the beta density and this Dirichlet density happens to be the conjugate prior here. So, as you can see, if I am estimating the parameter of a Bernoulli density then, my conjugate prior happens to be beta. Whereas, if Iam estimating parameters for a multinomial distribution rather than binomial one then, the prior happens to be Dirichlet, which is a kind of a neat generalization of the beta density to, a to more than two case. Of course, this is a strange expression and we have first show that, this is a density on that particular set of p, that we we mentioned. Using the using similar methods, as in the case of beta density we can show that, this is a density, the other thing that we want is, just like in the beta density case, ultimately because, my posterior will be a Dirichlet density. We need to know the movements of the Dirichlet density so that, I I can correctly use my posterior to get my estimates. Once again without proofs, I I just put down these movements so, if p 1, p 2, p M have joint densities as Dirichlet, with parameters a 1, a 2, a M then, expected value of any component say, p j is a j by a 0 where, a 0 is a 1 plus a 2 plus a M. Variance of p j happens to be a 0 into a j into a 0 minus a j, by a 0 square into a 0 plus 1 and similarly, this is the co variance. We do not know the co variance but, this kind of gives us all the movements upto the up to order 2. So, for example, if you are going to use the use the mean of the posterior rather or final estimate, we would need this formula. So, with this let us now go on in this case, compute the posterior density, if by taking prior as Dirichlet, the posterior density we have, to for the posterior density, as we know is f p given D is proportional to the product of f D given p, into f p f D given p is the data likelihood, f p is the prior we have taken prior to be Dirichlet. We already have an expression for the likelihood so, if you substitute those two so, this is the expression for the likelihood, product p j to the power n j. And let us say, we have taken the we have taken the prior to be Dirichlet with parameters a 1, a 2, a M then, this becomes the prior so, this product can now be written as, product over p j of n j plus a j minus 1. So, obviously the reason, why we chosen this particular prior is that, the posterior will belong to same class. So, indeed posterior belongs to the same class right, this is product is proportional to product of p j to the power something. So, if my prior is Dirichlet with parameters a 1, a 2, a M then, the posterior is Dirichlet with parameters n j plus a j where, the n j’s come from the data, This is once again very similar to, what happened in the Bernoulli case. Thus, the posterior is also Dirich let with parameters n j plus a j so, if we take for example, the mean of the posterior as our final Bayesian estimate, we already seen, what the mean is a j by sum so, we know summation n j. So, it will be n j plus a j by summation over j n j plus a j, summation over j n j is n, that we have already seen and a 0 is the notation we have given for summation a j’s. So, the the bayesian density, which is taken as the mean of the posterior turns out to be n j plus a j by n plus a 0. Let us recall, that the ML estimate for this was n j by n right. And the ML estimate is very easy to see, we are asking, what is the probability that Z takes the j th value or what is the probability that, the j th outcome occurs that is, equal to the number of times j th outcome occurred by the total number of samples right, n j means summation over i, x i j is the number of times the j th value has occurred. So, n j by n was, as we have already derived was the ML estimate here, instead of it being n j by n, it becomes n j plus a j by n plus a 0 where, a j and a 0, which is a 1 plus a 2 plus a M are determined by our choice of the prior right, our choice of prior determines, what the value a j are. So, just like in the case of the Bernoulli parameters, the nature of the Bayesian estimate is same so, you can think of the prior as saying, that before I collect data in my mind because, I have some idea of, what the values of p j’s are. I have coded them to say that, if I have done a zero repetitions of Z, they are fictitious repetitions a 1 of them would give me first value, a 2 of them will give me second value, a M of them will give me third value. So, I can choose the a 0 as well as a 1 to a M based on my idea of, what these numbers p 1 to p M are. So then, the final estimate is the actual in the data, how many times j has occurred plus how many time j has occurred in the fictitious trials, divided by the total number of actual trials and the fictitious trials. So, when data when the like in the Bernoulli case, if the data is small then, we do not go very wrong because, our paired beliefs we will ensure that, p j’s do not go into unnatural values. For example, p j breaking 1 or 0, when data is very small but, as n increases for any fixed a j and a 0, as n increases ultimately, this becomes n j by n. So, asymptotically the Bayesian estimate will be same as the maximum likelihood estimate and hence, it will be consistent. But, once again like in the Bernoulli case, the the prior allows me, to allow my initial beliefs to properly moderate data especially, when data is not very large. Now, let us look at another example, last class we considered the example of estimating mean of a normal distribution where, we assumed the variance to be known right. Now, let us do it the other way round, we want to estimate the variance of a normal distribution and we assume mean to be known. It might look a little strange to you, when we did the ML estimate, we did not have to do so much trouble, we directly did only one example to estimate both mean and variance of a one dimensional Gaussian distribution. Because, in the ML case, it is a very straight forward thing here, for each kind of parameter, the corresponding prior would be different for example, when we wanted to estimate the mean, the conjugate prior was Gaussian right. Some of you may be thinking that, because we were estimating Gaussian density, the conjugate what happened to be Gaussian, that is not true. If you want to estimate the variance of a Gaussian where, I assume mean known, the conjugate prior cannot be Gaussian right because, variance as a parameter can take only non-zero values. So, it is density cannot be Gaussian then, we may jump to the conclusion say, may be it is a exponential, exponential is a density that is 0 only only when the random variables takes positive values, it is the density is non-zero, only when the random variable takes positive values. But, exponential is only a special case right, we will see that in general, the the prior is not exponential, exponential is only very special case of the prior. Also, the prior will not be unvariance, as it turns out for this case, is better to take 1 by variance at the parameter, it is often denoted by nu and is often called the precision. While I have not done the vector case, in the vector case, the inverse of the sigma matrix is called the lambda matrix and that is called the precision matrix. In the in the scalar case of course, we simply take the 1 by variance at the precision, which is often denoted by nu so, in terms of the parameter nu, the normal density model is given by 1 root nu. Because, is normally 1 by sigma root 2 pi, 1 by sigma is root nu so, this root nu by root 2 pi exponential minus half normally, x minus mu whole square by sigma square and 1 by sigma square is nu. So, it is written as exponential minus nu by 2 into x minus mu whole square. Note that, we are assuming mu is known and that is why, only mu is shown as the conditioning parameter. And we need to find nu said that, nu is always positive so, for example, when we choose a prior density, we choose density that is, the that is 0 on the negative nu and you have to find, what is the prior density, you have to look at the data likelihood. So, let us look at the data likelihood, the data likelihood is given by in terms of nu, as equal to 1 to n, f of x i given nu, f of x given nu is this. So, if I take a product, this will give me nu to the power root nu to the power n that is, nu to the power n by 2. This 1 by root 2 pi to the power n that is, 2 pi to the power minus n by 2 so, I have a two pi to the power minus n by 2 term, I have a nu to the power n by 2 term. And then, when I take a product of this over x i, it will become exponential minus nu by 2 into sum of this. So, exponential minus nu by 2 into sum over i x i minus mu whole square so now ,to ask what should be the right prior, we should ask what kind of function is this of nu, viewed as a function of nu, what kind of function is this. So, we essentially have an exponential nu into something term and we have a nu to the power something term right. So, the conjugate prior should be something, this proportional to nu power something and that, should be proportional to product of a power of nu and an exponential of a linear function of nu. Because, the data density is some constant into nu to the power something and exponential minus some some k times nu. So, if the prior also has nu to the power something and exponential some constant into nu, I mean then, the product will once again be nu to the power something into exponentials of constant nu. So, the prior should be proportional to a product of a power of nu and an exponential of a linear function in nu. And such a density transfer to be, what is known as a gamma density, such a prior would be what is called the gamma density. So, let us look at the gamma density, the gamma density is given by the density function f nu is 1 by gamma a, b to the power of a nu to the power a minus one e to the power of b nu where, a and b are parameters. So, the gamma is once again the gamma function, as a matter of fact, the actual gamma function comes from making this to be a density. By a simple integration, we can show that this to be a density because, this integral will turn out to be the gamma function. The the gamma density has two parameters a and b, the a comes in nu to the power of a minus one and b comes in e power minus b nu, this b power a is needed so that, the density integrates to 1. So, the nu to the power is controlled by a and exponential of the linear function in nu is controlled by b, these two are the parameters and the mean of gamma density is a by b and the mode is a minus 1 by b. If I actually choose a to be 1 then, the density turns out to be b e power minus b nu right now, when a is 1, as you know gamma of a is a minus gamma, if one is one turns out to be 1 by straight forward integration. So, when a is equal to 1 is simply b e power minus b nu that is nothing but, the exponential density so, exponential density is a special case of gamma density with a is equal to 1. So, let us take the prior to be gamma with parameters a 0 and b 0 that means, it becomes nu the power of a 0 minus 1 e to the power of minus b 0 nu, those are the two important terms, the rest is constant. So, the posterior density becomes f of nu given D is proportional f of D given nu into f nu f of D given nu is this and f of nu is this with a as a 0 and b as b 0. So, we get this, f of D given nu is nu to the power forgetting the constants keeping only the nu terms; it become nu to the power n by 2 exponential minus nu by 2 into summation x i minus mu whole square; and from f nu, I get nu to the power of a 0 minus 1 exponential minus b 0 nu. For the reason we chose this as the prior is now, these two new terms will become nu to the power something and these exponential terms will become exponential something into nu. So, if you do that, it becomes nu to the power of a 0 plus n by 2 minus 1 into exponential minus b 0 nu minus nu by 2 into this. So, we once again have nu to the power something exponential minus a linear function of nu so, the posterior as expected, is once again a gamma density. Now, what kind of gamma density is it. The the gamma density is two parameters a and b essentially, forgetting the constants is proportional to nu to the power of a minus 1 e to the power minus b nu right. So for the posterior density, the a parameter is a 0 plus n by 2 and the b parameter is b 0 plus half into this sum right. So, if we think that the posterior is a gamma density with parameters a a n and b n right then, this is what we will get. If the posterior is a is a gamma density with parameters a n and b n. Then a n will be a 0 plus n by 2, 0 plus n by 2 and what will be b n, b n will be b 0 plus half into this summation b 0 plus half into the summation. I can write this summation as, I know 1 by n summation is equal to 1 to n, x n minus mu whole square will be the maximum likelihood estimate for variance plus call it sigma square hat ML. Then, this summation is n times sigma square hat ML so, I can write b n as, b 0 plus n by 2 sigma square hat ML. So, if I chosen the prior to be gamma with parameters a 0 and b 0 then, the posterior will become a gamma with parameters a n and b n. Where a n turns out to be a 0 plus n by 2 and b n turns out to be b 0 plus n by 2 times, sigma square hat ML where, sigma square hat ML is the maximum likelihood estimator for variance in this case. So, recall that sigma square hat ML is the estimate for variance, is the maximum likelihood estimate for variance. So now, if I want to take the mean of the posterior as the final estimate, as we know for the gamma density with parameters a and b, the mean is a by b. So, here the posterior density is gamma with parameters a n and b n so, the mean will be a n by b n. So, our Bayesian estimate for nu nu hat would be a n by b n that is, a 0 plus n by 2 or b 0 plus n by 2 sigma square hat ML right. Remember that, nu is 1 by sigma square right so, if I do not did not have the a 0 and b 0, nu hat is 1 by sigma square Ml so, it is same as the maximum likelihood estimate. Because, nu is actually 1 by sigma square so, the estimate for 1 by sigma square will be 1 by sigma square hat ML. Now, the a 0 and b 0 are determined by our choice of prior right, we are choosing a gamma density at the prior so, the kind of gamma density we want, is what determines the values a 0 and b 0. Now, what can we say about this density, once I have this estimate, once again as n tends to infinity, nu hat converges to sigma square ML right because, as n tends to infinity, n by 2 will be greater than both a 0 and b 0. So, this fraction essentially becomes n by 2 by n by two sigma square hat ML so, I am I am sorry about to the typo nu hat converges to 1 by sigma square hat ML. I am sorry, it is not nu hat converges to sigma square ML but, nu hat converges to 1 by sigma square hat ML. Also note, that the variance of the posterior right for a gamma density with parameters a n and b n, the posterior the variance is a n by b n square. So, this is a n, this is b n so, if you take the square, the numerator goes as n whereas, denominator increases as n square so, that the variance goes to 0, as n tends to infinity. So, as n tends to infinity, the posterior essentially becomes same as the mean and the mean is 1 by sigma square hat ML so, once again just as we expect, the the Bayesian estimate is consistent. But, at any small sample size, it is not only determined by the data thus, 1 by sigma square hat ML but, is also determined by the initial a 0 and b 0, we choose for the prior prior density, which is gaussian with parameters a 0 and b 0. So, we have seen both Bayesian estimation for either the mean or the variance of the Gaussian, mean we seen last time, for variance we are seeing just now. So, when I want to estimate only the mean, assuming that the variance is known then, the prior turns out to be a a Gaussian. When I want to estimate only the variance, assuming that the mean is known, the prior turns out to be gamma. So, if I want to estimate both mean and variance now, I have two parameters once again, from my experience in estimating variance, we will choose nu as the parameter, for parameters in the density model. So, my density model now is f of x given mu nu is this remember, that nu is 1 by sigma square so, if both mu and nu are unknown, we need a prior, there is a joint density on mu and nu. We already know that, if nu is known, only mu is unknown then, the prior density is Gaussian, if mu is known and nu is unknown then, I know the prior density is gamma. So, the joint density should be some combination of Gaussian and gamma, the the algebra turns out be a little cumbersome so, I do not give you all the algebra, I will just give you the final expression. So, then the conjugate prior would be, what is called as Gaussian gamma density, the Gaussian gamma density this, any joint density of any two random variables mu and nu here, can be written as a product of the marginal of nu multiplied by the conditional of mu given nu that is, true of anything. So, this is how, we will model this so, the the Gaussian gamma density model is given here, as you can see what we are saying is, f nu is the first term here, upto here first meaning, the first two terms so, this is the density what we already seen, this is a gamma right. So, f nu is gamma with parameters a 0 and b 0 and the density mu given nu is essentially a Gaussian density with nu as it is precision or 1 by nu as it is variance. So, the conditional of mu given nu is Gaussian, with this is a Gaussian in the in the variable mu with it is own mean mu 0 and the variance being a function of the conditioning random variable. That is, this is not just directly 1 by nu but it is 1 by c 0 nu so that is, the marginal for nu is a gamma density and the conditional density of mu, conditioned on nu is a Gaussian. Actually, by looking at the at the data likelihood, we can find that, this is the kind of dependence we need that, nu can always be expressed in terms of nu to the power something and exponential linear in nu. Whereas, the mu dependence can only be expressed by something that is the function of both nu and mu that is why, we have to model this in this kind of a factorization. When I put equality hereby now, we know, that we do not need the actual constants, we are only looking at the form of this densities. So, by now, we have seen enough examples so, I started misusing the abusing the notation, I just put equality even though, it is not really equal. Because, this thing is not really a density, there will always be a, in the second thing there will be some normalizing constant. There will be one constant to make this gamma density a density, another constant to make this into a proper normal density so, there will be some constant. But by now, we know this constants do not matter so, I just, we are abusing notation by not putting that constant. And also, actually we would have some relation between c 0 and b 0 and a 0 but, it really does not matter, we can choose a slightly bigger class of densities at the conjugate prior. Of course, this prior density is quite involved and doing the Bayesian estimation with this prior density is not easy so, I will skip the details, you people can sit and do the algebra, the algebra is more complicated. But ultimately, you get similar looking final essentially, what we get is a convex combination of the sample mean plus something that depends on the on the prior parameters for the for the estimate of nu. And similarly, for the estimate of nu right, it will be some factor involving 1 by sigma square hat ML and something that, depends on your a 0 b 0 in such a way that, as n tends to infinity, once again the ML estimates and the Bayesian estimates will be same. So, we will we will we, I have just given you the prior for this but, we will not actually derive the final estimates. I have not done any of multidimensional examples, they are not conceptually any more difficult than the one dimensional normally we did. But obviously, as you can see compared to maximum likelihood, obtaining Bayesian estimates has lot more algebra so, just the algebraic notation will be more cumbersome so, we will we will skip that. So, we will simply say that, we can obtain Bayesian estimates like this for many standard densities but, there is one part that is, by now evident, obtaining maximum likelihood estimates and Bayesian estimates is not the same. For maximum likelihood estimates, for almost mechanically, I can calculate the likelihood function, differentiate, equate to 0, find the maximum and I get the estimates. For Bayesian estimate, I have to properly choose the right kind of prior, which is the conjugate prior for that particular problem and only then, right the the expressions are amenable to simplification. And then, I have to look at the look at the parameters of the posterior density and based on that, I I have to obtain my Bayesian estimates. As we have seen the conjugate prior would depend on the form of x given theta as a matter of fact, for the same density in our mind say, Gaussian depending on what is the parameterization that we think, what are the unknown parameters that we think, the prior changes. For example, if we think only the mean of the Gaussian is unknown then, the conjugate prior is Gaussian, if we think only the variance is unknown and we choose the variance in terms of the precision parameter then, the density happens to be gamma. If we think both mean and the precision are unknown then, the prior density, the conjugate prior density turns out to be that gaussian gamma, I told you. So, the conjugate prior would depend very much on the form of f of x given theta, the procedure little more involves certainly than the maximum likelihood estimate. So, what is that we gain with it, why is not maximum likelihood estimate sufficient, that we have already seen, when we when we started the Bayesian estimate, that the reason why we came to Bayesian estimate is that, maximum likelihood estimate blindly believes the data. So, if we have some prior information about the kind of values the parameter can take or because, our first few data are bad and we have very little data. There is no way, we can make any incomplete information we have about the parameter to to bear on the final estimate we get. Whereas, the prior density allows us this the this flexibility so, essentially the prior density allows us to incorporate knowledge, that we may have about the parameter that is, in the form of a conjugate prior. And as we seen in the final expressions, it always comes up with an expression whereby, this small sample problems are automatically handled by trading the part that I get only from data, not trading by combining the part, that I get only from the data, with the part I get from the prior. So, at small sample, our beliefs, kind of moderators in not jumping to too drastic at conclusions based on data, that is the essence of the Bayesian estimation. Now, we will slightly move to a few more related issues in estimation, we seen two specific methods of estimation, we have seen how to do the estimation for different densities. For example, we have derived ML and Bayesian estimates for a few standard densities now, let us look at a few more generic issues about estimation. The first thing that we will do is, to look at what is what is a generic representation for many densities so, there is one form for a density. By by now, I suppose you become familiar to this, that as i said right in the beginning, when we started on our estimation, we use this the word density to mean either density or mass function. Depending on the random variable is discrete or continuous so, we use density in a generic sense so, we are saying, we will look first at at the representation of a density function in terms of some parameters, that captures most of the standard densities, such a form is called the exponential family of densities. It is a it is a very important thing because, as we shall see later on, for exponential family of densities ML estimates become very straight forward. So, we get generic ML estimates for all densities within the exponential family, given the exponential family, for all of them, we can write one kind of generic set of equations to solve, to get the ML estimates, we do not have to do individually. And equally importantly, looking at this, kind of generic notion of a density function allows us to introduce an important notion estimation, which is called the sufficient statistic. So, first let us look at, what we call exponential family of densities suppose, you have a density model for a random variable x with parameters eta. Eta could be a single parameter or many parameters, with many parameter we will think of it as a parameter vector. We will write the the density model as f x given eta as h of x that is, some function with only of x multiplied by some function g of eta, is some function only of the parameter eta multiplied by the exponential of eta transpose u x where u x is a vector of functions of the data. So, given data I can make some new functions of data u 1 x, u 2 x for example, data is x 1, x 2, x n, u 1 x could be summation x i, u 2 x could be summation x i square and so on, you have sorry u 1 x could be x, u 2 x could be x square and so on. So, u x are some vector functions of x; so if the density can be written as a product of a time involving only x and a time involving only eta and a time that involves both eta and u and x in a very special way, exponential eta transpose u x where u is a vector of predefined, a vector of given functions of x. The reason, why it is called the exponential family is that, I can always write it as exponential of something all right. I can write the exponential of eta transpose u x plus this h x factor can be brought inside the exponential by writing it as l n h x similarly, this as l n g x. Because, exponential of l n h x will be h x, exponential of l n g x will be g x because, I can write it like this, it is called a exponential family. The important thing is that, many standard densities Bernoulli, binomial, Poisson, gamma, beta, Gaussian, exponential everything can be put in this form. Of course, among these standard densities, the notable one that cannot be written in this form is, uniform density except from uniform density, almost all the standard densities can be put in this form. So, let us look at a simple example of, how to put a density in in the standard exponential let us consider, the Bernoulli distribution so, the mass function with parameter p is given in terms of p power x into 1 minus p to the power 1 minus x. This is obviously, not in the form of a factor involving only a x, multiplied factor involving only the parameter multiplied by a factor like this right. But the point is, by not thinking of p as the parameter but, something else as the parameter, we would be able to put it in this form. So, let us look at that so, starting with this, we can write this as, I can always write anything so, I want you, I can write this p x into1e minus p to the power 1 minus x as exponential l n of that. So, if I if I take l n and put inside exponential, the l n of this will become x l n p plus 1 minus x l n 1 minus p so, that is what I did, exponential x l n p plus 1 minus x l n 1 minus p. Now, there is, this 1 into l n 1 minus p, exponential of l n 1 minus p will be 1 minus p so, let us let me take that factor out that is, 1 minus p. Now, I have got x l n p and minus x l n one minus p, I can write it as, x into l n p by 1 minus p so, I can write this as 1 minus p exponential of x l n p by 1 minus p. Now, this I can further write as, 1 by 1 plus p by 1 minus p right. So this now, one can immediately see, if I think of p by 1 minus p as a parameter then, I can write this as let us say, that is what I want to call eta then, this is some factor that is dependent only on eta. And this is a factor that depend exponential of let us say, l n p into 1 minus p is my eta then, eta times a function of x namely, x. So, I have to somehow, write this as also l n p by 1 minus p, this is very easy I can always write p by 1 minus p as exponential l n p by 1 minus p. So, I can write this as, 1 by 1 plus exponential eta into exponential eta x where, eta is l n p by 1 minus p right. So, I can write my Bernoulli mass function in as, 1 by 1 plus exponential eta into exponential eta x where, eta is l n p by 1 minus p. So, what does this mean, this is exactly in the form h x into g eta into exponential eta transpose u x right h x is 1, there’ i no factor, there is only dependent on x right so, h x is 1. What is g eta, g eta is this factor 1 by 1 plus exponential eta right that is, g eta and I want exponential eta transpose u x, I have got eta times x. So, eta is a scalar here so, I can simply take u x to be x right. So, if I take eta as l n p by 1 minus p, h x as 1, g eta as 1 by 1 plus exponential eta and u x is equal to x then, it is in this form where, this transpose is of course, is redundant here because, eta happens to be one dimensional. Whereas this means, that the Bernoulli density belongs to the exponential family in the same way, for all the standard densities, we can put them in this general frame work. Sometimes see for example, if I want to represent the Bernoulli mass function with p as the parameter then, it is not in this generic form, h x into g eta into this. But, if I u instead of using p as the parameter, I use l n p by 1 minus p as the parameter then, I can put in this. After all, if you give me eta that is, l n p by 1 minus p, I can calculate p or if you give me p, I can calculate l n p by 1 minus p so, the eta to p transformation is one to one and invertible. So, by that I think of eta as the parameter, as p as the parameter it does not matter but, if I think of eta as the parameter, the mass function comes to a very standard form so, sometimes eta is called the natural parameter for Bernoulli, this particular eta. In the same, many other densities can be put in the exponential form so, for example, for the Gaussian, the normally I will write it with, mu and sigma square as the two parameters like this. But we can also write it like this, some function represents only acts essentially a constant function, some factor that depends only on some parameters, which I call eta 1 and eta 2. And then, exponential eta 1 time some function of x plus eta 2 times some function of x where, eta 1 happens to be mu by sigma square, eta 2 happens to be minus 1 by 2 sigma square, u 1 x happens to be x and u 2 x happens to be x square. The algebra involved is a little more than the algebra involved in showing this by the Bernoulli density but, once again it is just algebra. So, starting from this expression, one can show that, this is same as this expression where, I make this following changes eta 1 is mu by sigma square, eta 2 is minus 1 by sigma square, u 1 x is equal to x, u 2 x is equal to x square. So, under these things, this density once again recommends the form x h g eta exponential eta transpose u x right. As I said h x can be thought of as it is constant function, this is the g eta function and this is exponential eta 1 u 1 x plus eta 2 u 2 x where, eta 1, eta 2, u 1, u 2 are given right. So, once again these are the form h x g eta exponential eta transpose u x so, Gaussian is also in the exponential family and like this, we can show that almost all standard densities belong to exponential class of densities. Now, what is the use of showing many of these densities belong to the exponential family of densities, the the main utility is that, as I said, we get a very standard generic form for the maximum likelihood estimate. And also, what it would mean is the following now. When a density is in this form, if i take the data likelihood right, the data likelihood will depend on n fold product of this that is, product of x h i g eta to the power n. And when I multiply exponential eta transpose u x 1 into exponential transpose u x 2 and so on ultimately, I get exponential eta transpose summation u x i. So, these functions u x or the quantity summation u x i are the only way, the data affects the data likelihood. So, this form gives us a very interesting generic way, in which data affects the data likelihood and hence, gives us a standard method for calculating maximum likelihood estimates for all such densities. So, in the next class, we will we will look at a few of the examples of the exponential family of densities. And how, looking at all of them as exponential family of densities, allows us to obtain maximum likelihood estimates in a in a generic fashion and then, we will introduce the notion of, what is called a sufficient statistic. Thank you.
C1 高級 Mod-03 Lec-08 貝葉斯估計實例;密度的指數族和ML估計。 (Mod-03 Lec-08 Bayesian Estimation examples; the exponential family of densities and ML estimates) 62 7 aga 發佈於 2021 年 01 月 14 日 更多分享 分享 收藏 回報 影片單字