Spatial-Semantic 3D Robot Perception with Computational Symmetries

Abstract

Robot localization and mapping is a process where a robot constructs a global spatial model of a scene based on multiple local observations. It is a fundamental problem supporting other components of robot autonomy, such as planning and navigation. In this thesis, we are concerned with building a robust and semantic-aware framework of robot perception. With the help of modern vision sensors, recent perception techniques are capable of producing fast and accurate estimations for robot positions and 3D geometric structures under feature-rich environments. In perceptually degraded situations, mainstream algorithms' dependencies on feature quality and scene structure are no longer valid. Besides, the advances of deep neural networks provide ways of learning point-wise semantic information. However, these data-driven methods require an enormous amount of labeled data to capture the underlying distribution of the practical real world. Furthermore, effective strategies for fusing the available predictions from upstream machine learning algorithms into a consistent and efficient spatial-semantic environment model remain an ongoing challenge. To address these problems, we propose leveraging the tools from the reproducing kernel Hilbert space and the equivariance learning theory based on the symmetry of input data to construct a continuous and functional representation of the sensor observations. This representation enables a novel formulation of the localization problem that is robust towards sensor noise and outliers. Furthermore, this representation can tightly couple the spatial-semantic relations for different perception tasks.

Specifically, in Chapter 4, we detail the functional and equivariant formulation that integrates geometric and semantic measurements and then introduce a frame-to-frame sensor registration framework. It outperforms the existing frame-to-frame registration methods in accuracy and robustness. Chapter 5 extends the two-frame registration to a multi-frame bundle adjustment formulation. It demonstrates the potential in local and large-scale robot trajectory estimations, with robust performances in feature-less and outlier-rich scenarios. Finally, Chapter 6 presents an unsupervised equivariance learning framework for learning the semantic features of the above sensor representation, which excels in environments with limited ground truth data. It offers robust adaptation and improved resilience against outliers or mismatches in simulated and unseen real-world datasets. The aforementioned contributions enhance the data efficiency, generalization, semantic understanding, and robustness of robot perception techniques within the demanding context of perceptually degraded applications.