Question

3. Let X be the random variable characterized by a probability distribu tion p (P, ...p) and it can assume one of the values r1,....In with probabilities pi, , pni 0 < pi 1, Σǐpí = 1 The Shannon entropy of the random variable X is defined as Show that 0 s H(X) S log2 so that the maximal value in this inequality is saturated by unifornm probability distribution. What is the level of information for this state of the system? (Hint: Use the Lagrange multiplier method with con straint Σ¡P.-1)

Answer 1

We have to maximize $\fn_jvn H\left ( X \right )=-\sum _ip_i\log_2p_i$ subject to $\fn_jvn \sum _ip_i=1$ . The Lagrangian is

$\fn_jvn L=-\sum _ip_i\log_2p_i&plus;\lambda \left ( \sum _ip_i-1 \right )$

Taking partial derivatives with respect to $\fn_jvn p_i$ and equating to 0,

$\fn_jvn \frac{\partial L}{\partial p_i}=0\\ -\log_2p_i-\frac{1}{\ln 2}&plus;\lambda =0;i=1,2,...,n\\ p_1=p_2=p_3=.....=p_n=1/n$

Thus the maximum entropy occurs, when the distribution is uniform $\fn_jvn p_1=p_2=p_3=.....=p_n=1/n$

The maximum entropy is

$\fn_jvn H\left ( X \right )=-\sum _i\frac{1}{n}\log_2\left ( \frac{1}{n} \right )\\ H\left ( X \right )=\log_2 n$