Leveraging Compositional Structure for Reinforcement Learning and Sequential Decision Making Problems

Abstract

Deep Learning approaches have made tremendous progress toward solving reinforcement learning and sequential decision-making problems. However, current approaches still struggle with long-horizon tasks that require strong generalization. These are tasks that an agent must solve using many actions and may have situations where the agent must generalize its actions from prior experience. A dominant approach is to solve these tasks in a hierarchical manner: a high-level agent decomposes a task into multiple “subtasks” to be individually solved by a low-level agent, which specializes in solving these subtasks. The effectiveness of this approach is enhanced by identifying and utilizing the inherent compositional structures of tasks, which enable more efficient learning and broader generalization. This dissertation introduces novel methodologies that build on task compositionality to address these challenges. Key contributions include: (1) Higher-Order Skill Learning: A hierarchical reinforcement learning framework is proposed, enabling low-level policies to optimize for sequences of subtasks rather than individual ones, resulting in improved efficiency and performance. (2) Parameterized Task Structures: Introducing parameterized subtask graphs to model tasks with compositional structures, enhancing both the efficiency of task inference and generalization to unseen entities. In addition, contributions that show compositional structure can be used through language and large language models (LLMs): (3) Integrating Multimodal Observations in Language Models: Demonstrating that visual observations can be embedded as input tokens for large language models (LLMs), achieving state-of-the-art performance in visually grounded planning tasks. (4) Skill Abstractions for LLM Planning: Highlighting the benefits of providing LLMs with structured descriptions of subtasks, or skills, to improve planning and reasoning capabilities. (5) Code-Augmented Planning: Proposing a method where LLMs use control flow constructs to generate and execute code for solving complex planning tasks, significantly improving task performance. Collectively, these approaches showcase how leveraging task compositionality through hierarchical structures and language from LLMs can improve reinforcement learning and sequential decision-making frameworks.