Frame-Skipping and Interpolation Algorithm for Efficient Data Processing

Abstract

When processing image and video data for machine learning algorithms, a major obstacle that many research teams face is having to manually label the collected data before training can begin. This process can be incredibly slow, taking away valuable time and resources away from other necessary tasks, such as developing and tuning the machine learning algorithm or collecting a larger, more diverse set of data for testing. This, in turn, can lead to poor model performance during training and testing. In addition, much of this effort is often spent on labeling consecutive, nearly identical frames, which is redundant and adds little new information for the model to work with. The goal of this project is to expedite the labeling process by using key features within a video to detect and extract unique, high-information frames for manual labelling and propagating those labels to the rest of the dataset through interpolation. At a high level, the algorithm described here is comprised of two components. The first, the frame-skipping component, iterates through the frames of a given video, detecting and saving the unique frames to an output directory. Once labeled, using the unique frames and their labels, the interpolation component estimates the position of the labeled objects throughout the remaining frames in the video. Through extensive experimentation using a set of 50 videos, I have found that the algorithm generally performs quite quickly, with a relatively small error due to interpolation. In addition, the frame-skipping and interpolation algorithm, on average, only requires half of the frames to be labeled in order to achieve optimal accuracy performance. As such, the algorithm could reduce the time needed to label a given video by 50%. Currently, I am conducting tests to determine how the interpolation error affects the performance of machine learning models trained on the interpolated data.