Contact-based Perception and Planning for Robotic Manipulation in Novel Environments

Abstract

This thesis proposes a framework for autonomous robotic perception and planning for manipulation tasks in unknown environments by leveraging information from purposeful contacts and explicitly reasoning about uncertainty. The high-level goal is to enhance the amount of information that can be extracted from contacts, enabling greater utilization of this sensing modality. We focus on challenging tasks where objects to be manipulated are occluded by the environment, other objects, or themselves, which limits the applicability of purely visual sensing and necessitates contact-based information gathering.

Each chapter of our work tackles a specific challenge arising from collecting information through contact, with the goal of enabling robots to explore autonomously. The first challenge we considered is the limited applicability of long-horizon planning when global perception is lacking. Traps may arise where the state remains in a cycle without accomplishing the goal, and we develop a hierarchical control scheme to detect and escape from traps.

Contact-based exploration is also challenging due to the ambiguity of associating contact points to specific objects in multi-object environments. To resolve this, we present a method that maintains a belief over both current and past contact points without relying on rigid associations. This flexibility allows for the correction of erroneous estimates.

Building on these contact point estimates, we infer the plausible poses of known objects. A key component in our method is the use of negative information—data indicating observed free space—which constrains possible object poses by measuring the discrepancy between these potential poses and the observed point clouds. This approach is especially effective in highly-occluded environments where visual object segmentation often fails.

To integrate our pose estimates into real-time decision-making, we formulate a conditional probability on object poses given the disparity with observed point clouds. We derive a cost function from the mutual information between the object's pose and the occupancy of the workspace points, facilitating its application in closed-loop model predictive control (MPC). Our method also includes a reachability cost function to prevent objects from being pushed out of the robot's workspace and incorporates a stochastic dynamics model to predict information gain changes as the object is manipulated.

The algorithms developed in this thesis emphasize efficient parallel computation and are evaluated using both simulated and real experiments. All implementations are made publicly available as open-source libraries.