Bandit problems with covariates: Sequential allocation of experiments.

Abstract

Consider a Bayesian sequential allocation problem that incorporates a covariate. The goal is to maximize the total discounted expected reward from a population of patients by choosing an appropriate allocation rule. Advantages of a covariate model are illustrated numerically in a finite horizon Bernoulli bandit problem. Theoretical results are obtained for the case of a geometrically discounted infinite population, one-parameter exponential family model. When compared to bandit problems without covariates these show: (a) the approximate solution to the allocation problem is simpler in the presence of a covariate, (b) the exact value of the discount factor is less important and (c) the myopic procedure is asymptotically optimal. The above results extend to the more realistic delayed responses case.