Understanding belief propagation and its generalizations

ABSTRACT

"Inference" problems arise in statistical physics, computer vision, error-correcting coding theory, and AI. We explain the principles behind the belief propagation (BP) algorithm, which is an efficient way to solve inference problems based on passing local messages. We develop a unified approach, with examples, notation, and graphical models borrowed from the relevant disciplines.We explain the close connection between the BP algorithm and the Bethe approximation of statistical physics. In particular, we show that BP can only converge to a fixed point that is also a stationary point of the Bethe approximation to the free energy. This result helps explaining the successes of the BP algorithm and enables connections to be made with variational approaches to approximate inference.The connection of BP with the Bethe approximation also suggests a way to construct new message-passing algorithms based on improvements to Bethe's approximation introduced Kikuchi and others. The new generalized belief propagation (GBP) algorithms are significantly more accurate than ordinary BP for some problems. We illustrate how to construct GBP algorithms with a detailed example.

References

1. Aji, S. M., and McEliece, R. J. (2000). The generalized distributive law. In IEEE Transactions on Information Theory 46, 325-343.
2. Aji, S. M., and McEliece, R. J. (2001). The generalized distributive law and free energy minimization. Proceedings of the 39th Annual Allerton Conference on Communication, Control, and Computing. Forthcoming.
3. Baxter, R. I. (1982). Exactly Solved Models in Statistical Mechanics. New York: Academic Press.
4. Berrou, C., A. Glavieux, and P. Thitimajshima (1993). Near Shannon limit error-correcting coding and decoding: Turbo-codes. In Proceedings of the IEEE International Conference on Communications, Geneva, 1064-1070.
5. Boykov, Y., O. Veksler, and R. Zabih (2001). Fast approximate energy minimization via graph cuts. In IEEE Transactions in Pattern Analysis and Machine Intelligence 23(11), 1222-1239.
6. Cover, T. M., and J. A. Thomas (1991). Elements of Information Theory. New York: John Wiley & Sons.
7. Cowell, R. (1998). Introduction to inference for Bayesian networks. In Learning in Graphical Models, ed. M. Jordan. Cambridge, MA: MIT Press.
8. Freeman, W. T., E. C. Pasztor, and O. T. Canrtichael (2000). Learning low-level vision. International Journal of Computer Vision 40(1), 25-47.
9. Frey, B. J. (1998). Graphical Models for Machine Learning and Digital Communication. Cambridge, MA: MIT Press.
10. Frey, B. J., and D. J. C. Mackay (1998). A revolution: Belief propagation in graphs with cycles. In Advances in Neural Information Processing Systems, Vol. 10, ed. M. Jordan, M. S. Kearns, and S. A. Solla. Cambridge, MA: MIT Press.
11. Gallager, R. G. (1963). Low-Density Parity, Check Codes. Cambridge, MA: MIT Press.
12. Gallager, R. G. (1968). Information Theory and Reliable Communication. New York: John Wiley & Sons.
13. Geman, S., and D. Geman (1984). Stochastic Relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions in Pattern Analysis and Machine Intelligence 6(6), 721-741.
14. Jaakkola, T. (2000). Tutorial on variational approximation methods. Available online at www.ai.mit.edu/people/tommi/papers.html.
15. Jensen, F. (1996). An Introduction to Bayesian Networks. Berlin: Springer-Verlag.
16. Jordan, M. I., Z. Ghahramani, T. Jaakkola, and L. Saul(1998). An introduction to variational methods for graphical models. In Learning in Graphical Models, ed. M Jordan. Cambridge, MA: MIT Press.
17. Kikuchi, R. (1951). A theory of cooperative phenomena. Phys. Rev. 81,988-1003.
18. Kikuchi, R. (1994). Special issue in honor of R. Kikuchi. Progr. Theor. Phys. Suppl., 115.
19. Kschischang, F. R., B. J. Frey, and H.-A. Loeliger (2001). Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory 47, 498-519.
20. Lauritzen, S. L., and D. J. Spiegelhalter (1988). Local computations with probabilities on graphical structures and their application to expert systems (with discussion). Journal of the Royal Statistical Society, Series B 50, 157-224.
21. Mackay, D. J. C. (1999). Good error-correcting codes based on very sparse matrices. IEEE Transactions Information Theory 45, 399-431
22. McEliece, R. J., D. J. C. MacKay, and J. F. Cheng (1998). Turbo decoding as an instance of Pearl's "belief propagation" algorithm. In IEEE Journal on Selected Areas in Communications 16(2), 140-152.
23. Mezard, M., G. Parisi, and M. A. Virasoro (1987). Spin Glass Theory and Beyond. Singapore: World Scientific.
24. Minka, T.P. (2001). Expectation propagation for approximate Bayesian inference. In Proceedings of the Annual Conference on Uncertainty in Artificial Intelligence, Seattle, WA, 362-369.
25. Murphy, K., Y. Weiss, and M. Jordan (1999). Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the Annual Conference on Uncertainty in Artificial Intelligence, Stockholm, 467-475.
26. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco: Morgan Kaufmann Publishers.
27. Shafer G. R., and Shenoy P. P. (1990). Probability propagation. Annals of Mathematics and Artifical Intelligence 2, 327-352.
28. Tanner, R. M. (1981). A recursive approach to low complexity codes. In IEEE Transaction on Information Theory IT-27, 533-547.
29. Wainright, M. J., T. Jaakkola, and A. S. Willsky (2001). Tree-based reparameterization framework for approximate estimation in graphs with cycles. LIDS Technical Report P-2510. Available online at ssg.mit.edu/group/mjwain/mjwain.shtml.
30. Welling, M., and Y. W. Teh (2001). Belief optimization for binary networks: A stable alternative to loopy belief propagation. In Proceedings of the Annual Conference on Uncertainty in Artificial Intelligence, Seattle, WA, 554-561.
31. Yedidia, J. S. (2001). An idiosyncratic journey beyond mean field theory. In Advance Mean Field Methods-Theory and Practice, ed. D. Saad and M. Opper. Cambridge, MA: MIT Press.
32. Yedidia, J. S., W. T. Freeman, and Y. Weiss (2001a). Bethe free energies, Kikuchi approximations, and belief propagation algorithms. Available online at www.merl.com/reports/TR2001-16/index. html.
33. Yedidia, J. S., W. T. Freeman, and Y. Weiss (2001b). Characterizing belief propagation and its generalizations. Available online at www.merl.com/reports/TR2001-15/index.html.
34. Yuille, A. L. (2001). A double-loop algorithm to minimize the Bethe and Kikuchi free energies. Unpublished.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Knowledge representation and reasoning

  • Machine learning

    • Machine learning approaches

      • Rule learning

• Theory of computation

  • Theory and algorithms for application domains

    • Machine learning theory

      • Inductive inference