Question

Consider a Markov chain with state space S = {1,2,3,4} and transition matrix

P =$\begin{bmatrix} p1&p2&p3&p4\\ 1 &0 &0 &0 \\ 0& 1& 0&0 \\ 0&0 &1 &0 \end{bmatrix}$​​​​​​​ where $\sum_{k=1}^{4}pk=1$

(a) Draw a directed graph that represents the transition matrix for this Markov chain.

(b) Compute the following probabilities:

P(starting from state 1, the process reaches state 3 in exactly three-time steps);

P(starting from state 1, the process reaches state 3 in exactly four-time steps);

P(starting from state 1, the process reaches states higher than state 1 in exactly two-time steps).

(c) If the process starts from state 3, provide the states which are not attainable in exactly two-time steps.

Answer 1

S = {1, 2, 3, 4} P = [P_1 P_2 P_3 P_4 1 0 0 0 0 1 0 0 0 0 1 0] (a) directed graph (b) (1) P(starting from state 1, the process reaches state 3 in exactly three- time steps) = P_1^2 P_3 + P_2 times 1 times p_3 = P_1^2 P_3 + P_2 P_3 = P_3(P_1^2 + P_2) (2) P(starting from state 1, the process reaches state 3 in exactly 4 time steps) = P_1^3 P_3 + P_2 times 1 times P_4 times 1 = P_1^3 P_3 + P_2 P_4 (3) P(starting from state 1, the process reaches state 3 in exactly 2 time steps) = P_1 P_3 + P_4 times 1 = P_1 P_3 + P_4 (c) From the directed graph state (2) & (4) is not attainable