Sensitivity and uncertainty analysis of complex simulation models.

Abstract

Discrete-event stochastic simulation is a powerful tool for understanding and evaluating complex systems. Simulation models confront the analyst with the problem of sensitivity analysis: That is, what are the effects of changing the inputs, including parameters of probability distributions and design variables, on the output? Sensitivity analysis is also closely linked to uncertainty analysis, which aims to quantify uncertainty associated with the output response as a result of uncertainties in the input parameters. In this dissertation, we address both sensitivity analysis and uncertainty analysis of complex simulation models. We present a systematic three stage design of experiments (DOE) approach to construct metamodels for the sensitivity analysis of complex simulation models. We examine and apply several recent ideas and methods in the statistical experimental design literature to simulation metamodeling. We evaluate these methods based on several performance criteria and make several recommendations for use in our three stage approach. We also develop a new design criterion for both metamodel discrimination and parameter estimation for a follow-up experiment. Our criterion is computationally more tractable than other similar existing joint criterions. We also present a Bayesian asymptotic approximation of input uncertainty to quantify output uncertainty of simulation models. In addition, we consider the problem of deciding how to allocate additional resources to reduce output uncertainty (i.e. should more simulation replications be conducted or more field data be collected). Based on the asymptotic approximations, we provide closed-form sampling plans for additional data collection activities to effectively reduce the output uncertainty due to input uncertainty.