Visual Query Optimization: Algorithms and Software Systems

Abstract

The development of inexpensive digital cameras and availability of local and cloud data storage has lead to incredible amounts of digital video production. Meanwhile, the amount of compute used for frontier machine learning workloads is growing even faster than Moore's Law.

In this thesis, I focus on the problem of visual analytics for dashboard camera video. Novel algorithms and systems which make large-scale video search tractable are highly important for autonomous vehicle development, which requires curating a library of video of the vehicle performing in real-world conditions. I first show and evaluate techniques to reconstruct, classify, and index vehicle maneuvers for dashboard camera video, and then evaluate search techniques for visual queries which are independent of these algorithms.

First, I present our work describing a novel approach for detecting vehicle maneuvers from monocular dash-cam video building by leveraging deep learning techniques to estimate frame-accurate ego-vehicle movement and techniques to search video efficiently for disjoint computer vision operations. We show a technique to process these motion estimates into a vehicle trajectory, and compare trajectory segments to the reference maneuvers. We then extend the idea of motion estimation for maneuver detection to lane changes, highway events, and acceleration. We discuss and evaluate a ``min pool'' approach to allow each classifier to select the closest match between several reference maneuvers, which we find outperforms a single reference maneuver classification approach.

We then build a classifier pipeline to operate the individual classifiers and prioritize detected maneuvers, and show statistical results. Then, I present work on an end-to-end software system to process human-semantic queries for dashboard camera video among arbitrary visual processing techniques and models. We adapt computer vision models for use in an existing distributed video processing framework by writing API functionality for object detection, semantic segmentation, depth, and optical flow. We design and demonstrate a query parser to dynamically parse, plan, and execute visual queries. This infrastructure then allows us to focus on how to do cost planning for these visual analytics queries by adapting database cost planning metrics to the use of computer vision techniques.