Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data

Abstract

We consider the problem of estimating the parameters of a Gaussian or binary distribution in such a way that the resulting undirected graphical model is sparse. Our approach is to solve a maximum likelihood problem with an added l₁-norm penalty term. The problem as formulated is convex but the memory requirements and complexity of existing interior point methods are prohibitive for problems with more than tens of nodes. We present two new algorithms for solving problems with at least a thousand nodes in the Gaussian case. Our first algorithm uses block coordinate descent, and can be interpreted as recursive l₁-norm penalized regression. Our second algorithm, based on Nesterov's first order method, yields a complexity estimate with a better dependence on problem size than existing interior point methods. Using a log determinant relaxation of the log partition function (Wainwright and Jordan, 2006), we show that these same algorithms can be used to solve an approximate sparse maximum likelihood problem for the binary case. We test our algorithms on synthetic data, as well as on gene expression and senate voting records data.

References

1. M. Ashburner, C. A. Ball, J. A. Blake, D. Botstein, H. Butler, J. M. Cherry, A. P. Davis, K. Dolinski, S. S. Dwight, J. T. Eppig, M. A. Harris, D. P. Hill, L. Issel-Tarver, A. Kasarskis, S. Lewis, J. C. Matese, J. E. Richardson, M. Ringwald, G. M. Rubin, and G. Sherlock. Gene ontology: Tool for the unification of biology. Nature Genet., 25:25-29, 2000.
2. D. Bertsekas. Nonlinear Programming. Athena Scientific, 1998.
3. P. J. Bickel and E. Levina. Regularized estimation of large covariance matrices. Annals of Statistics, 36(1):199-227, 2008.
4. J. Dahl, V. Roychowdhury, and L. Vandenberghe. Covariance selection for non-chordal graphs via chordal embedding. To appear in Optimization Methods and Software, Revised 2007.
5. A. d'Aspremont, L. El Ghaoui, M.I. Jordan, and G. R. G. Lanckriet. A direct formulation for sparse PCA using semidefinite programming. Advances in Neural Information Processing Systems, 17, 2004.
6. A. Dobra and M. West. Bayesian covariance selection. Technical Report, 2004.
7. J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 2007.
8. T. R. Hughes, M. J. Marton, A. R. Jones, C. J. Roberts, R. Stoughton, C. D. Armour, H. A. Bennett, E. Coffey, H. Dai, Y. D. He, M. J. Kidd, A. M. King, M. R. Meyer, D. Slade, P. Y. Lum, S. B. Stepaniants, D. D. Shoemaker, D. Gachotte, K. Chakraburtty, J. Simon, M. Bard, and S. H. Friend. Functional discovery via a compendium of expression profiles. Cell, 102(1):109-126, 2000.
9. S. Lauritzen. Graphical Models. Springer Verlag, 1996.
10. H. Li and J. Gui. Gradient directed regularization for sparse gaussian concentration graphs, with applications to inference of genetic networks. University of Pennsylvania Technical Report, 2005.
11. Z. Q. Luo and P. Tseng. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72(1):7-35, 1992.
12. N. Meinshausen and P. Bühlmann. High dimensional graphs and variable selection with the lasso. Annals of statistics, 34:1436-1462, 2006.
13. G. Natsoulis, L. El Ghaoui, G. Lanckriet, A. Tolley, F. Leroy, S. Dunlea, B. Eynon, C. Pearson, S. Tugendreich, and K. Jarnagin. Classification of a large microarray data set: algorithm comparison and analysis of drug signatures. Genome Research, 15:724 -736, 2005.
14. Y. Nesterov. Smooth minimization of non-smooth functions. Math. Prog., 103(1):127-152, 2005.
15. M. R. Osborne, B. Presnell, and B. A. Turlach. On the lasso and its dual. Journal of Computational and Graphical Statistics, 9(2):319-337, 2000.
16. T. P. Speed and H. T. Kiiveri. Gaussian markov distributions over finite graphs. Annals of Statistics, 14(1):138-150, 1986.
17. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal statistical society, series B, 58(267-288), 1996.
18. L. Vandenberghe, S. Boyd, and S.-P. Wu. Determinant maximization with linear matrix inequality constraints. SIAM Journal on Matrix Analysis and Applications, 19(4):499-533, 1998.
19. M. Wainwright and M. Jordan. Log-determinant relaxation for approximate inference in discrete markov random fields. IEEE Transactions on Signal Processing, 2006.
20. M. Wainwright, P. Ravikumar, and J. D. Lafferty. High-dimensional graphical model selection using l ₁-regularized logistic regression. Proceedings of Advances in Neural Information Processing Systems, 2006.

Index Terms

• Mathematics of computing

  • Discrete mathematics

    • Graph theory

      • Graph algorithms

  • Probability and statistics

    • Distribution functions

    • Multivariate statistics

    • Probabilistic algorithms

    • Probabilistic reasoning algorithms

      • Markov-chain Monte Carlo methods

      • Sequential Monte Carlo methods

• Theory of computation

  • Design and analysis of algorithms