Topics on Anomaly Detection, High Dimensional Testing and Spectral Inference for Functional Data

Abstract

This dissertation consists of two major parts. The first part is concerned with high-dimensional testing and involves both a methodological and theoretical results. The methodology portion is centered around the detection of anomalous Internet traffic. It is motivated by and applied to network telescopes or ``Darknet'' type data, which is Internet traffic obtained by monitoring a large number of streams corresponding to ``unused'' Internet address space. We propose an algorithm for the synchronous online detection of abnormal Internet traffic, based on recent theoretical developments, and evaluate its performance both in the detection and the identification aspects. The remainder of the first part involves theoretical contributions which solve an open problem in probability on the rates of convergence of maxima for dependent Gaussian triangular arrays. These technical results on the rates allow us to establish that the concentration of maxima phenomenon holds in more general, not necessarily Gaussian, models. The latter phenomenon is the key to an important phase transition result that characterizes the statistical limits in the exact support recovery problem for a sparse high-dimensional signal observed in additive, light-tailed noise. Thus, our theoretical results make direct contributions to high-dimensional statistics. The second part of this dissertation is focused on the non-parametric estimation of the spectral density of space-time random field processes taking values in a separable Hilbert space. The estimator relies on kernel smoothing and is applicable to spatial sampling schemes where data are not necessarily observed at regular spatial locations. In a mixed-domain asymptotic setting and under general conditions, rates for the bias and variance of the estimator are obtained which lead to rates for its consistency. Considering practical applications, where complete functional data are usually unavailable, our asymptotic results are specialized to the case of discretely-sampled functional data taking values in a reproducing kernel Hilbert space. Further, it is shown that when the data are observed on a regular spatial grid, the optimal rate of the estimator matches the minimax rate for the class of covariance functions that decay according to a power law. Finally, the asymptotic normality of the spectral density estimator is also established under general conditions for Gaussian Hilbert-space valued processes.