Exact Bayesian Structure Discovery in Bayesian Networks

Abstract

Learning a Bayesian network structure from data is a well-motivated but computationally hard task. We present an algorithm that computes the exact posterior probability of a subnetwork, e.g., a directed edge; a modified version of the algorithm finds one of the most probable network structures. This algorithm runs in time O(n 2ⁿ + n^k+1C(m)), where n is the number of network variables, k is a constant maximum in-degree, and C(m) is the cost of computing a single local marginal conditional likelihood for m data instances. This is the first algorithm with less than super-exponential complexity with respect to n. Exact computation allows us to tackle complex cases where existing Monte Carlo methods and local search procedures potentially fail. We show that also in domains with a large number of variables, exact computation is feasible, given suitable a priori restrictions on the structures; combining exact and inexact methods is also possible. We demonstrate the applicability of the presented algorithm on four synthetic data sets with 17, 22, 37, and 100 variables.

References

1. S. Acid and L. de Campos. Searching for Bayesian network structures in the space of restricted acyclic partially directed graphs. Journal of Artificial Intelligence Research, 18:445-490, 2003.
2. S. M. Aji and R. J. McEliece. The generalized distributive law. IEEE Transactions on Information Theory, 46(2):325-343, 2000.
3. I. Beinlich, G. Suermondt, R. Chavez, and G. F. Cooper. The ALARM monitoring system. In J. Hunter, editor, Proceedings of the Second European Conference on Artificial Intelligence and Medicine, pages 247-256. Springer-Verlag, Berlin, 1989.
4. R. Bellman. Dynamic programming treatment of the travelling salesman problem. Journal of the ACM, 9:61-63, 1962.
5. C. L. Blake and C. J. Merz. UCI repository of machine learning databases, 1998.
6. R. R. Bouckaert. Properties of Bayesian belief network learning algorithms. In R. Lopez de Mantaras and D. Poole, editors, Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pages 102-109, Seattle, WA, 1994. Morgan Kaufmann, San Francisco, CA.
7. W. Buntine. Theory refinement on Bayesian networks. In B. D'Ambrosio, P. Smets, and P. Bonissone, editors, Proceedings of the Seventh Conference on Uncertainty in Artificial Intelligence, pages 52-60, Los Angeles, CA, 1991. Morgan Kaufmann, San Mateo, CA.
8. R. Castelo and T. Kočka. On inclusion-driven learning of Bayesian networks. Journal of Machine Learning Research, 4:527-574, 2003.
9. D. M. Chickering. Optimal structure indentification with greedy search. Journal of Machine Learning Research, 3:507-554, 2002.
10. D. M. Chickering, D. Geiger, and D. Heckerman. Learning Bayesian networks: Search methods and experimental results. In Proceedings of the Fifth Conference on Artificial Intelligence and Statistics, pages 112-128. Society for Artificial Intelligence and Statistics, Ft. Lauderdale, 1995.
11. D. M. Chickering, D. Heckerman, and C. Meek. A Bayesian approach to learning Bayesian networks with local structure. In D. Geiger and P. Shenoy, editors, Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pages 80-89, Providence, RI, 1997. Morgan Kaufmann, San Francisco, CA.
12. C. Chow and C. Liu. Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory, 14:462-467, 1968.
13. G. F. Cooper and E. Herskovits. A Bayesian method for the induction of probabilistic networks from data. Machine Learning, 9:309-347, 1992.
14. R. Dechter. Bucket elimination: A unifying framework for reasoning. Artificial Intelligence, 113 (1-2):41-85, 1999.
15. N. Friedman and D. Koller. Being Bayesian about network structure: A Bayesian approach to structure discovery in Bayesian networks. Machine Learning, 50(1-2):95-125, 2003.
16. A. E. Gelfand and A. F. M. Smith. Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association, 85:398-409, 1990.
17. D. Heckerman, D. Geiger, and D. M. Chickering. Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20:197-243, 1995a.
18. D. Heckerman, A. Mamdani, and M. P. Wellman. Real-world applications of Bayesian networks. Communications of the ACM, 38(3):24-30, 1995b.
19. D. Heckerman, C. Meek, and G. F. Cooper. A Bayesian approach to causal discovery. In C. Glymour and G. F. Cooper, editors, Computation, Causation, Discovery, pages 141-165. MIT Press, Cambridge, 1999.
20. E. Herskovits. Computer-Based Probabilistic Network Construction. PhD thesis, Medical Information Sciences, Stanford University, 1991.
21. K.-U. Höffgen. Learning and robust learning of product distributions. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, pages 77-83, Santa Cruz, CA, USA, 1993. ACM Press.
22. R. Kennes and P. Smets. Computational aspects of the Möbius transformation. In P. B. Bonissone, M. Henrion, L. N. Kanal, and J. F. Lemmer, editors, Uncertainty in Artificial Intelligence 6, pages 401-416. North-Holland, Amsterdam, 1991.
23. P. Larrañaga, C. Kuijpers, R. Murga, and Y. Yurramendi. Learning Bayesian network structures by searching for the best ordering with genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics, 26(4):487-493, 1996.
24. S. L. Lauritzen and F. V. Jensen. Local computation with valuations from a commutative semigroup. Annals of Mathematics and Artificial Intelligence, 21(1):51-69, 1997.
25. D. Madigan and J. York. Bayesian graphical models for discrete data. International Statistical Review, 63:215-232, 1995.
26. P. Myllymäki, T. Silander, H. Tirri, and P. Uronen. B-Course: A web-based tool for Bayesian and causal data analysis. International Journal on Artificial Intelligence Tools, 11(3):369-387, 2002.
27. J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.
28. R. E. Stearns and H. B. Hunt III. An algebraic model for combinatorial problems. SIAM Journal on Computing, 25(2):448-476, 1996.

Index Terms

• Mathematics of computing

  • Discrete mathematics

    • Graph theory

      • Graph algorithms

  • Probability and statistics

    • Probabilistic algorithms

    • Probabilistic reasoning algorithms

      • Markov-chain Monte Carlo methods

      • Sequential Monte Carlo methods