Exploring Training Provenance for Clues of Data Poisoning in Machine Learning

Abstract

Machine Learning today plays a vital role in a wide range of critical applications. To ensure ML models are consistently able to produce correct output, such models must be retrained as new input samples become available to avoid performance degradation as a result of data drift. During retraining, such models are left vulnerable to possible injection of poisonous samples via data poisoning attacks, where adversaries manipulate the training data to have the ML model misbehave to serve adversarial goals. Inspired by previous works that leverage data provenance for detecting and filtering out poisonous data from the training set, we adapt the definition of provenance for what we define as training provenance, the history of training metrics captured over training time-frame. We then build a framework that allows us to capture both key performance metrics for the overall dataset and per-class metrics for each label at every epoch as training provenance. Through exploratory analysis of captured training provenance we aim to find clues that standout to serve as strong signals for making a call on training data poisoning. We evaluate our proposed framework on two benchmark image classification datasets: MNIST and CIFAR-10. For MNIST, we observed promising signal(s) for establishing a per-epoch poisoning detection threshold based on captured metrics at the overall dataset level. For CIFAR-10, captured overall dataset metrics as training provenance for clean and poisoned training data were not as effective as compared to our observations for MNIST experiments. As for captured per-class metrics, we discovered that these metrics provided little insight as a result of the nondeterministic nature of machine learning. Overall, we observe that this is a promising direction that invites further exploration with more poisoning attacks and diverse datasets.