Average optimality in infinite horizon optimization.

Abstract

We study three classes of infinite horizon optimization problems: the undiscounted homogeneous Markov decision process, the undiscounted nonhomogeneous Markov decision process, and the undiscounted deterministic problem. To solve the undiscounted homogeneous Markov decision process, we give a sufficient condition for the existence of a stationary optimal strategy using the Doeblin coefficient. We also compare this condition with others in the literature. This condition allows transformation of the original problem into an equivalent discounted homogeneous Markov decision process for which a solution method is known. For the undiscounted nonhomogeneous Markov decision process, it is known that an algorithmically optimal strategy is average optimal. Based on it, we present two solution methodologies using transformations into equivalent problems. Both procedures first transform the original problem into an equivalent discounted nonhomogeneous Markov decision process using the Doeblin coefficient. Then, the first (second) procedure transforms it into an equivalent discounted deterministic problem (discounted homogeneous Markov decision process) whose solution method is known. For undiscounted deterministic problems, it is not known whether an algorithmically optimal strategy is average optimal or not. We present sufficient conditions for it, and apply the result to a production planning problem and a Markov decision process.