Question

Consider an unfair coin which shows heads with probability 0.4 and tails with probability 0.6. Source 1 tosses the coin two times and tells you the number of heads in these two tosses, which is either 0, 1, or 2. In particular, the source is a Binomial (2, 0.4) random variable. Source 2 tosses the coin two times and tells you the result, which is either HH, HT, TH, or TT. In particular, it is two Bernoulli(0.4) random variables. Find the entropy of each source. Which source conveys more information on average?

Answer 1

Entropy of Source 2 is greater than Entroopy of Source 1. The log in the formula has a base of 2.

Do comment for any doubts.