Simple Partial Models for Complex Dynamical Systems.

Abstract

An agent in an unknown environment may wish to learn a model that allows it to make predictions about future events and anticipate the consequences of its actions. Such a model can greatly enhance the agent's ability to make good decisions. However, in environments like the one in which we live, which is stochastic, partially observable, and high dimensional, learning a model is a challenge.

One approach when faced with a difficult model learning problem is not to model the entire system. Instead, one might focus on the most important aspects of the environment and give up on modeling complicated, irrelevant phenomena. This intuition can be formalized using partial models, which are models that make only a restricted set of predictions in only a restricted set of circumstances. Because a partial model has limited prediction responsibilities, it may be significantly simpler than a complete model.

Partial models have been studied in many contexts, mostly under the Markov assumption, where the agent is assumed to have access to the full state of the world. In this setting, predictions can be learned directly as functions of state and the process of learning a partial model is often as simple as estimating only the desired predictions and omitting the rest from the model. As such, much of the relevant work has focused on the challenging question of which partial models should be learned (rather than how to learn them). In the partially observable case, however, where state is assumed to be hidden from the agent, the basic problem of how to learn a partial model poses significant challenges.

The goal of this thesis is to provide general results and methods for learning partial models in partially observable systems. The main challenges posed by partial observability are formalized and learning methods are developed to address these issues. The methods presented are demonstrated empirically to learn partial models in systems that are too complex for standard, complete model learning methods. Finally, many partial models are learned and composed to form complete models that are used for model-based planning in high dimensional arcade game examples.