Question

Show that the Dirichlet distribution is a conjugate distribution to the multinomial distribution

Answer 1

A conjugate prior is from the same distributional family as the likelihood itself. In practical terms, it means you can write your posterior down easily, in the sense that “updating” with new data requires relatively little computation.

The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to compute and (b) it in some sense is possible to quantify how much our beliefs have changed after collecting the data. It can certainly be discussed whether these are good reasons to choose a particular prior, as these criteria are unrelated to actual prior beliefs... Nevertheless, conjugate priors are popular, as they often are reasonably flexible and convenient to use for the reasons stated above For the special case of the multinomial distribution, let (P1, ..., Pk) be the vector of multinomial parameters (ie. the probabilities for the different categories). If (P1, ... , Pk) Dirichlet(Q1,..., ak) prior to collecting the data, then, given observations (21, ..., xk) in the different categories, (P1,...,PR) (21,..., xk) Dirichlet(a1 + 21,...,0+ 2k). The uniform distribution is actually a special case of the Dirichlet distribution, corresponding to the case a1 = a 2 =...=al = 1. So is the least-informative Jeffreys prior, for which Q1 =...=ak = 1/2. The fact that the Dirichlet class includes these natural "non-informative" priors is another reason for using it.