Analysis of Situated Interactive Non-Expert Instruction of A Hierarchical Task to a Learning Robot

Abstract

Interactive Task Learning (ITL) is an approach to designing robots that can learn tasks on the fly from human instruction and demonstration in a shared environment. ITL approaches until now have focused on extending a robot's capabilities so that it can efficiently learn a wide variety of task components as a part of hierarchical tasks from an expert human instructor. However, the typical instructor is likely to be a non-expert who does not have a good mental model of the robot's physical and mental capabilities and is therefore likely to face challenges during the teaching process. Our hypothesis is that we need to focus on building better robot learners that actively interact to make the teaching process more accessible and efficient for the non-expert teacher.

Towards this goal, we conducted a human participant think-aloud study (N=14) where we asked participants to teach a multi-step hierarchical task of baking a pizza to an ITL robot in a simulated environment. We built a templated instruction interface, which participants could use to provide two types of action and two types of goal instructions to the robot. In this dissertation, we present a qualitative analysis of the data collected from this study. We identify the different types of knowledge that non-expert teachers leveraged during instruction, namely knowledge about the task, environment, interface, and robot. Participants had access to this knowledge as a result of their prior experiences as well as through interaction with the robot using the interface and the environment. We also present our characterization of different aspects of the teaching processes used by non-expert teachers. We observed that participants developed complex strategies to teach the task to the robot. They actively evaluated the robot's task progress and became effective at teaching the task. However, we found that participants encountered challenges during the teaching process, which included difficulty in providing desired instructions and encountering failure situations. Participants also found the knowledge about the task, environment, interface, and robot to be insufficient at times, which led them to have incomplete and incorrect mental models. However, participants were motivated to continue teaching even after encountering these challenges and 13 out of 14 participants successfully finished teaching the task to the robot.

To address challenges faced by non-expert teachers, we propose extensions to existing robot interaction approaches that would allow the robot to use its knowledge of the various aspects of the task and itself to help the instructor teach better. To enable the non-expert instructor to provide their desired instructions, we propose extensions to the instruction interface to allow for more complex instructions and accommodate free-form instructions when necessary. To improve the instructor's mental models, we propose that the robot can leverage its access to various sources of knowledge to provide relevant updates and knowledge to the instructor. To determine how the robot can help the non-expert instructor recover from these failures, we recommend the need for more research to understand more about the different failures that can occur during teaching and to conduct studies to evaluate how non-experts can effectively resolve them. In the end, we emphasize the value of iterative development in this dissertation towards creating robots that can successfully interact with non-experts in the real world.