Anomaly Detection and Sequential Filtering with Partial Observations

Abstract

With the rise of “big data” where any and all data is collected, comes a series of new challenges involving the computation and analysis of such massive data sets. Nowadays, data is continuously collected leading to questions of at which point should analysis begin and how to incorporate new data into the analysis. And, within the massive amounts of data collected, there can be other complications in addition to the noise. The features of interest may not be directly observable to a user, and thus are modeled as latent variables. There may be only a very small subset of the data with certain properties that are of interest to the user. Or, there could be data that is only partially labeled due to the costs of user labeled data or simply a lack of information. In this thesis, we develop methods that deal with data containing partial labels, latent variables, and anomalies. Many of the models in our frameworks are extendable to an online or streaming scenario where the data is continuously being collected and discarded. We also illustrate some real world applications of our proposed models using datasets from cyber security, transportation, and weather systems. The contributions of this thesis are that we have developed: 1. Penalized ensemble Kalman filter that is designed for superior performance in nonlinear high dimensional systems. 2. Framework to generate and update regression and classification models, which can be used to build an optimal non-linear filter and also an approximation to it that is computationally efficient. 3. Recursive versions of supervised and semi-supervised maximum margin classifiers. 4. Method for detecting anomalous points that are partially labeled high utility by a domain expert. 5. Framework and probabilistic model for detecting anomalous activity in the traffic rates of sparse networks.