Real-time Human Workload Estimation and Its Application in Adaptive Haptic Shared Control

Abstract

Automated vehicles (AVs) are promising to have the potential to reduce driving-related injuries and deaths. However, autonomous driving technology is currently limited in its scope and reliability, giving rise to the semi-autonomous driving model, where the autonomy and the human share the control of the vehicle. Workload, despite being an important human factor, has not yet been considered when designing adaptive shared control. Recently, researchers have started to apply machine learning techniques to classify mental workload into different levels. However, most of these studies have adopted either a single-model-single-feature approach or a single-model-all-features approach. However different machine learning models are suitable for different features, how to leverage different models for different features is critical.

To address these shortcomings and research gaps, the goals of this dissertation were to (1) examine whether and to what extent haptic shared control performance can be improved by incorporating operators' workload; (2) develop a computational model for workload estimation, and the model should be able to leverage different machine learning models that work best for different features; and (3) investigate the generalizability of the workload estimation model. To address these research goals, this dissertation was composed of four research phases with two pilot studies and four human subject experiments.

(1) Collaborating with Yifan Weng, Dr. Tulga Ersal, and Prof. Jeffrey Stein from the Department of Mechanical Engineering at the University of Michigan, we developed a teleoperated dual-task shared control simulation platform where the human shared control of a ground vehicle with autonomy while performing a surveillance task simultaneously. In addition, we developed a real-time eye-tracking system based on Tobii Pro Glasses 2 to measure the human gaze points in a world frame and pupil sizes.

(2) We proposed a workload-adaptive haptic shared control scheme together with our collaborators. We conducted two human subject experiments during this phase. The results indicated that the proposed workload-adaptive haptic shared control scheme can reduce human workload, increase human trust in the system, increase driving performance, and reduce human effort without sacrificing surveillance task performance.

(3) We proposed a Bayesian inference model for workload estimation that can leverage the different machine learning models that work best for different features. Specifically, we used support-vector machines (SVMs) for pupil size change, the Hidden Markov Model (HMM) for gaze trajectory, SVMs for fixation feature, and Gaussian Mixture Models (GMMs) for fixation trajectory. The empirical results indicated that our proposed model achieved a 0.82 F1 score for workload imposed by varying surveillance task urgency.

(4) We investigated the generalizability of our proposed Bayesian inference model for workload estimation by conducting two human subject experiments with 24 participants and using different factors to impose human workload, i.e., obstacle headway and driving speed. The results indicated that our proposed model achieved a 0.68 F1 score for the workload imposed by obstacle avoidance and the personalized version of our proposed model can distinguish the workload imposed by different driving speeds under high surveillance task urgency.