Learning To Use Memory.

Abstract

This thesis is a comprehensive empirical exploration of using reinforcement learning to learn to use simple forms of working memory. Learning to use memory involves learning how to behave in the environment while simultaneously learning when to select internal actions that control how knowledge persists in memory and learning how to use that information stored in memory to make decisions. We focus on two different models of memory: bit memory and gated memory. Bit memory is inspired by prior reinforcement learning literature and stores abstract values, which an agent can learn to associate with task history. Gated memory is inspired by human working memory and stores perceptually grounded symbols. Our goal is to determine computational bounds on the tractability of learning to use these memories. We conduct a comprehensive empirical exploration of the dynamics of learning to use memory models by modifying a simple partially observable task, TMaze, along specific dimensions: length of temporal delay, number of dependent decisions, number of distinct symbols, quantity of concurrent knowledge, and availability of second-order knowledge. We find that learning to use gated memory is significantly more tractable than learning to use bit memory because it stores perceptually grounded symbols in memory. We further find that learning performance scales more favorably along temporal delay, distinct symbols, and concurrent knowledge when learning to use gated memory than along other dimensions. We also identify situations in which agents fail to learn to use gated memory optimally which involve repeated identical observations which result in no unambiguous trajectories through the underlying task and memory state space.