Semi-supervised classification with hybrid generative/discriminative methods

ABSTRACT

We compare two recently proposed frameworks for combining generative and discriminative probabilistic classifiers and apply them to semi-supervised classification. In both cases we explore the tradeoff between maximizing a discriminative likelihood of labeled data and a generative likelihood of labeled and unlabeled data. While prominent semi-supervised learning methods assume low density regions between classes or are subject to generative modeling assumptions, we conjecture that hybrid generative/discriminative methods allow semi-supervised learning in the presence of strongly overlapping classes and reduce the risk of modeling structure in the unlabeled data that is irrelevant for the specific classification task of interest. We apply both hybrid approaches within naively structured Markov random field models and provide a thorough empirical comparison with two well-known semi-supervised learning methods on six text classification tasks. A semi-supervised hybrid generative/discriminative method provides the best accuracy in 75% of the experiments, and the multi-conditional learning hybrid approach achieves the highest overall mean accuracy across all tasks.

References

1. M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric framework for learning from examples. Technical Report TR-2004-06, University of Chicago, 2004.
2. A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In COLT: Proceedings of the Workshop on Computational Learning Theory, 1998.
3. G. Bouchard and B. Triggs. The tradeoff between generative and discriminative classifiers. In J. Antoch, editor, Proceedings in Computational Statistics, 16th Symposium of IASC, volume 16, Prague. Physica-Verlag.
4. U. Brefeld, T. Gaertner, T. Scheffer, and S. Wrobel. Efficient co-regularized least squares regression. In ICML06, 23rd International Conference on Machine Learning, Pittsburgh, USA, 2006.
5. R. Caruana. Multitask learning. Machine Learning, 28(1):41--75, 1997.
6. O. Chapelle, V. Sindhwani, and S. S. Keerthi. Branch and bound for semi-supervised support vector machines. In Advances in Neural Information Processing Systems (NIPS), 2006.
7. R. Collobert, F. Sinz, J. Weston, and L. Bottou. Large scale transductive SVMs. The Journal of Machine Learning Research, 7(Aug):1687--1712, 2006.
8. T. Joachims. Transductive inference for text classification using support vector machines. In Proc. 16th International Conf. on Machine Learning, pages 200--209. Morgan Kaufmann, San Francisco, CA, 1999.
9. C. Kang and J. Tian. A hybrid generative/discriminative bayesian classifier. In Proceedings of the 19th International FLAIRS Conference, 2006.
10. B. M. Kelm, C. Pal, and A. McCallum. Combining generative and discriminative methods for pixel classification with multi-conditional learning. ICPR, 2:828--832, 2006.
11. J. A. Lasserre, C. M. Bishop, and T. P. Minka. Principled hybrids of generative and discriminative models. In CVPR '06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 87--94, Washington, DC, USA, 2006. IEEE Computer Society.
12. R. Malouf. A comparison of algorithms for maximum entropy parameter estimation. In In Sixth Conf. on Natural Language Learning, pages 49--55, 2002.
13. A. McCallum, C. Pal, G. Druck, and X. Wang. Multi-conditional learning: Generative/discriminative training for clustering and classification. In AAAI '06: American Association for Artificial Intelligence National Conference on Artificial Intelligence, 2006.
14. A. K. McCallum. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu, 2002.
15. A. Ng and M. Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Advances in Neural Information Processing Systems (NIPS), 2002.
16. K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification, 1999.
17. K. Nigam, A. McCallum, S. Thrun, and T. M. Mitchell. Learning to classify text from labeled and unlabeled documents. In AAAI/IAAI, page 792, 1998.
18. R. Raina, Y. Shen, A. Y. Ng, and A. McCallum. Classification with hybrid generative/discriminative models. In NIPS, 2003.
19. J. Schler, M. Koppel, S. Argamon, and J. Pennebaker. Effects of age and gender on blogging. In 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs, 2006.
20. V. Sindhwani, P. Niyogi, and M. Belkin. A co-regularized approach to semi-supervised learning with multiple views. In Proc. of the 22nd ICML Workshop on Learning with Multiple Views, August 2005.
21. D.-Q. Zhang and S.-F. Chang. A generative-discriminative hybrid method for multi-view object detection. CVPR, 2:2017--2024, 2006.
22. X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning using Gaussian fields and harmonic functions. In ICML-03, 20th International Conference on Machine Learning, 2003.

Index Terms

• Computing methodologies

  • Machine learning