Question

Reinforcement learning question.

3. From state x, taking action 1 always produces a reward of 2 and sends you to a state y from which a return of 10 is always received. The discount parameter gamma is 0.9. What is vr(y)? What is q(x,1)?

Answer 1

Since state y, always gives reward of 10, and if the policy $\pi$ is to always remain in state y, then expected reward from state y, with discount gamma = 0.9 will be given as

$v_{\pi}(y) = \sum_{i=1}^{\infty}(\text{Reward})*(\text{discount parameter})^{i-1} = \sum_{i=1}^{\infty}10*(0.9)^{i-1} = \frac{10}{1-0.9}=100$

Since the optimal policy at state x is to take action 1 and go to state y which yields better reward. So $q_*(x,1) = \text{Reward of taking action 1 at state x}+\text{(discount gamma)}*\text{(value of next state)}$

Note that since optimal policy is that at state x, one should take action 1 and go to state y and thereafter remain in state y, thats why $v_{*}(y) = v_{\pi}(y)$ . Hence we could use the previous result of $v_{\pi}(y) = 100$

Please comment for any clarification