Efficient Regularized Solution Path Algorithms with Applications in Machine Learning and Data Mining.

Abstract

The topic of this dissertation is based on regularization methods and efficient solution path algorithms for machine learning and data mining.

The first essay proposes the doubly regularized support vector machine (DrSVM) for classification. The DrSVM uses the elastic-net penalty, a mixture of the L2-norm and the L1-norm penalties. By doing so, the DrSVM performs automatic variable selection in a way similar to the L1-norm SVM. In addition, the DrSVM encourages highly correlated variables to be selected (or removed) together, which is called the grouping effect. It also develops efficient algorithms to compute the whole solution paths of the DrSVM.

Based on the DrSVM, the second essay proposes the hybrid huberized support vector machine (HHSVM). The HHSVM uses the elastic-net penalty and the huberized hinge loss function. Similar to the DrSVM, the HHSVM performs automatic variable selection and has the grouping effect. However its computational cost is significantly reduced due to its loss function.

The third essay proposes two models for image denoising. In this essay, the L1-norm of the pixel updates is used as the penalty. The L1-norm penalty has the advantage of changing only the noisy pixels, while leaving the non-noisy pixels untouched. Efficient algorithms are designed to compute entire solution paths of these L1-norm penalized models, which facilitate the selection of a balance between the "loss" and the "penalty."

The last essay proposes a two-step kernel learning method based on the support vector regression (SVR) for financial time series forecasting. Given a number of candidate kernels, our method learns a sparse linear combination of these kernels so that the resulting kernel can be used to predict well on future data. The L1-norm regularization approach is used to achieve kernel learning. Since the regularization parameter must be carefully selected, to facilitate parameter tuning, we develop an efficient solution path algorithm that solves the optimal solutions for all possible values of the regularization parameter. Our kernel learning method is applied to forecast the S&P500 and the NASDAQ market indices and shows promising results.