Question

Average reward setting is proposed for continuing tasks. Here are some statements regarding this setting. Check all that apply. Average reward \(r(\pi) = \sum_\limits{5}\mu_{\pi}(s)\sum_\limits{a}\pi(as)\sum_\limits{s', r}p(s', |s, a)r\) holds if the MDP meets the ergodicity assumption. For the undiscounted continuing case, all policies that attain the maximal value of \(r(\pi)) are considered to be optimal. The ordering of all policies in terms of value is the same for both discounted and undiscounted tasks when we use the average reward setting Even though \(\gammal) is meaningless in the average reward setting, we cannot choose \(\gammal) to be 0 because then average reward is not well-defined. It is difficult to use the discounted control setting with function approximation in the continuing setting because the policy improvement theorem does not hold any more.

Answer 1

Correct Answer :

For the undiscounted continuing case, all policies that attain a maximum value of \(r(\pi)\) is optimal.

It is difficult to use the discounted control setting with function approximation in the continuing setting because the policy improvement theorem does not hold anymore.

Reason :

1. In the average reward setting for the continuing task, all data policies will be able to attain maximum value.
2. When the maximum value is attained, then this value is considered to be optimal.
3. The ordering of policies is different for the discounted and undiscounted task.
4. The average reward setting will be different for both discounted and undiscounted tasks.
5. It is difficult to use the discounted control setting with function approximation.
6. This is mainly because of the policy improvement theorem does not hold anymore.