TOUCH and GO: A Real-World Multisensory Dataset

Abstract

Unlike how humans perceive the world from associations between senses and through a series of inanimate objects, contemporary research on robot perception problem mainly rely on vision units or visual inputs to teach the robots interact with the world. We identify that this is due to the lack of real-world multisenory rich object dataset. To tackle this challenge, we present TOUCH and GO, a multisensory dataset containing real-world synchronized high-quality video and tactile data containing 12600 object instances over 37800 touches and 30 hours of video captured from egocentric viewpoint, greatly exceeding the size of existing real-world multisensory datasets. All objects in our dataset are originated from real environments with fine-grained textures retained. We propose and apply our dataset on two novel tasks, tactile-guided image stylization and multi-modal video prediction on tactile images.