Reinforcement Learning Agents that Discover Structured Representations

Abstract

Deep reinforcement learning (Deep RL) has recently emerged as a powerful method for developing AI that can learn to select actions in the world. One key question in RL is how an agent should learn knowledge that can be transferred to new situations. In this dissertation, I hypothesize that one key to transferring knowledge is the ability to discover structured representations that permit relational reasoning over basic units describing the agent's experience. Recent research in computer vision and natural language processing has shown that structured neural networks with sparse and dynamic information flow enable the discovery of such structured representations, leading to faster learning and improved generalization. The thesis of this dissertation is that we can equip reinforcement learning agents with the ability to discover and exploit structured representations by incorporating structured neural networks with dynamic information flow into the core components of an RL learner. By equipping RL agents with the ability to discover structured representations, we can reduce the amount of experience the agent needs for learning and improve its ability to transfer behaviors across situations. To support this argument, I present the following evidence.

First, I incorporate structured neural networks into an RL agent's transition function and show that this enables the discovery of object representations that capture an object's category, properties, and attributes, while achieving performance comparable to an agent with access to ground-truth object information. Afterward, I incorporate structured neural networks into an RL agent's state function and demonstrate that this enables discovering object primitives that facilitate generalization across three diverse object-centric environments. Next, I incorporate structured neural networks into an agent's value function and show that this enables the discovery of features that enable generalization to combinations of tasks. Finally, I incorporate structured neural networks into an agent's policy and provide a method that transfers to new tasks with hundreds of millions fewer samples compared to other transfer learning baselines. Taken together, this thesis demonstrates that incorporating structured neural networks into the core components of an RL learner can enable structured representation learning that both reduces the amount of experience an agent requires for learning and improves its ability to transfer behaviors across situations.