Compositional noisy-logical learning

ABSTRACT

We describe a new method for learning the conditional probability distribution of a binary-valued variable from labelled training examples. Our proposed Compositional Noisy-Logical Learning (CNLL) approach learns a noisy-logical distribution in a compositional manner. CNLL is an alternative to the well-known AdaBoost algorithm which performs coordinate descent on an alternative error measure. We describe two CNLL algorithms and test their performance compared to AdaBoost on two types of problem: (i) noisy-logical data (such as noisy exclusive-or), and (ii) four standard datasets from the UCI repository. Our results show that we outperform AdaBoost while using significantly fewer weak classifiers, thereby giving a more transparent classifier suitable for knowledge extraction.

References

1. Alpaydin, E. (2004), Introduction to Machine Learning, MIT Press.
2. Cheng, P. W. (1997) From covariation to causation: A causal power theory. Psychological Review, 104, 367--405.
3. Freund, Y., & Schapire, R. (1996). Experiments with a new boosting algorithm. Proceedings of 13th International Conference on Machine Learning, 148--156.
4. Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive Logistic Regression: a Statistical View of Boosting. Annal of Statistics 28, 337--407.
5. Griffiths, T. L., & Tenenbaum, J. B.(2005). Structure and strength in causal induction. Cognitive Psychology, 51, 334--384.
6. Meyer, Y. (2001). Oscillating Pattern in Image Processing and Nonlinear Evolution Equations. University Lecture Series Vol 22. American Mathematical Society.
7. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann.
8. Reyzin, L., & Schapire, R. (2006). How Boosting the Margin can also boost classifier complexity. 23rd International Conference on Machine Learning, 753--760.
9. Yuille, A. L., & Lu H. J. (2008). The Noisy-Logical Distribution. Advances in Neural Information Processing Systems.

Index Terms

• Computing methodologies

  • Machine learning

  • Modeling and simulation

    • Model development and analysis

      • Model verification and validation

      • Modeling methodologies

• Mathematics of computing

  • Probability and statistics

    • Probabilistic algorithms

    • Probabilistic reasoning algorithms

      • Markov-chain Monte Carlo methods

      • Sequential Monte Carlo methods