Risk-Sensitive Markov Decisions with Discounting and the Replacement of Construction Equipment

Abstract

A general model is presented for the formulation and solution of the risk-sensitive dynamic decision problem that maximizes the certain equivalent of the discounted rewards of a time-varying Markov decision process. The model uses the principle of optimality and stochastic dynamic programming to consider both the immediate rewards and the certain equivalent associated with the remaining transitions of a time-varying Markov process over a finite or infinite time horizon. Assumptions include constant risk aversion and the discounting of future cash flows. Both transient and stationary optimal decision policies are developed depending on the presence or absence of discounting. An example application considers the equipment replacement decision problem that seeks the best age for an existing scraper to be replaced and the optimal age of its replacement. The example illustrates the methodology and the sensitivity of the optimal decision policy to the discount factor and the degree of risk aversion. Higher discount rates produce the same qualitative effect as higher degrees of risk aversion. The stationary certain equivalents depend primarily on the discount rate whereas the optimal replacement policy is more sensitive to the risk aversion coefficient.