Dimensionality Reduction via Sparse Support Vector Machines

Kristin Bennett, Jinbo Bi, Mark Embrechts, Curt Breneman and Minghu Song

Departments of Mathematics, DSES, and Chemistry
Rensselaer Polytechnic Institute


Abstract
We describe a methodology for performing variable ranking and selection using support vector machines (SVMs). The method constructs a series of sparse linear SVMs to generate linear models that can generalize well, and uses a subset of nonzero weighted variables found by the linear models to produce a final nonlinear model. The method exploits the fact that a linear SVM (no kernels) with $\ell_1$-norm regularization inherently performs variable selection as a side-effect of minimizing capacity of the SVM model. The distribution of the linear model weights provides a mechanism for ranking and interpreting the effects of variables. Starplots are used to visualize the magnitude and variance of the weights for each variable. We illustrate the effectiveness of the methodology on synthetic data, benchmark problems and challenging regression problems in drug design. This method can dramatically reduce the number of variables, and outperforms SVMs trained using all attributes and using the attributes selected according to correlation coefficients. The visualization of the resulting models is useful for understanding the role of underlying variables.