sgen1: A generator of small but difficult satisfiability benchmarks

Abstract

The satisfiability problem is known to be NP-Complete; therefore, there should be relatively small problem instances that take a very long time to solve. However, most of the smaller benchmarks that were once thought challenging, especially the satisfiable ones, can be processed quickly by modern SAT-solvers. We describe and make available a generator that produces both unsatisfiable and, more significantly, satisfiable formulae that take longer to solve than any others known. At the two most recent international SAT Competitions, the smallest unsolved benchmarks were created by this generator. We analyze the results of all solvers in the most recent competition when applied to these benchmarks and also present our own more focused experiments.

References

1. Aloul, F. A., Ramani, A., Markov, I., and Sakallah, K. 2003. Solving difficult instances of Boolean satisfiability in the presence of symmetry. IEEE Trans. CAD 22, 9, 1117--1137.
2. Batory, D. 2005. Feature models, grammars, and propositional formulas. In Proceedings of the 9th International Conference on Software Product Lines. Springer, Berlin, 7--20.
3. clasp. 2009. A conflict-driven no good learning answer set solver. http://www.cs.uni-potsdam.de/clasp/.
4. Cook, S. A. 1971. The complexity of theorem-proving procedures. In Proceedings of the 3rd Annual ACM Symposium on Theory of Computing. ACM, New York, 151--158.
5. Crawford, J. M., Kearns, M. J., and Shapire, R. E. 1994. The minimal disagreement parity problem as a hard satisfiability problem. Tech. rep., Computational Intelligence Research Laboratory and AT&T Bell Labs.
6. Davis, M., Logemann, G., and Loveland, D. 1962. A machine program for theorem-proving. Commun. ACM 5, 7, 394--397.
7. Davis, M. and Putnam, H. 1960. A computing procedure for quantification theory. J. ACM 7, 3, 201--215.
8. DIMACS. 1993. Satisfiability suggested format. ftp://dimacs.rutgers.edu/pub/challenge/satisfiability/doc/satformat.tex.
9. Eén, N. and Sörensson, N. 2003. An extensible sat-solver. In Proceedings of the 6th Annual Conference on Theory and Applications of Satisfiability Testing. Springer, Berlin, 502--518.
10. Fortnow, L. 2009. The status of the P versus NP problem. Commun. ACM 52, 9, 78--86.
11. Haanpää, H., Järvisalo, M., Kaski, P., and Niemelä, I. 2006. Hard satisfiable clause sets for benchmarking equivalence reasoning techniques. J. Satisfiability Boolean Model. Comput. 2, 1-4, 27--46.
12. Haixia, J., Moore, C., and Selman, B. 2005. From spin glasses to hard satisfiable formulas. In Proceedings of the 7th International Conference on Theory and Applications of Satisfiability Testing. Springer, Berlin, 199--210.
13. Hirsch, E. 2002. Random generator hgen2 of satisfiable formulas in 3-CNF. http://logic.pdmi.ras.ru/~hirsch/benchmarks/.
14. Hoos, H. H. 2009. The international conferences on theory and applications of satisfiability testing (sat). http://www.satisfiability.org.
15. Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. 1983. Optimization by simulated annealing. Science 220, 4598, 671--680.
16. Krishnamachari, B., Xi Xie, A. B. S., and Wicker, S. 2000. Analysis of random noise and random walk algorithms for satisfiability testing. In Proceedings of the 6th Annual Conference on Principles and Practice of Constraint Programming. Springer, Berlin, 278--290.
17. le Berre, D. 2009. The sat competitions. http://www.satcompetition.org.
18. Li, C. M. and Huang, W. Q. 2005. Diversification and determinism in local search for satisfiability. In Proceedings of the 8th International Conference on Theory and Applications of Satisfiability Testing. Springer, Berlin, 158--172.
19. Pham, D. N., Thornton, J. R., Gretton, C., and Sattar, A. 2007. Advances in local search for satisfiability. In Proceedings of the 20th Australian Joint Conference on Artificial Intelligence. Springer, Berlin, 213--222.
20. Rintanen, J., Heljanko, K., and Niemel, I. 2006. Planning as satisfiability: Parallel plans and algorithms for plan search. Artif. Intell. 170, 12-13, 1031--1080.
21. Safarpour, S. A., Baeckler, G., Yuan, R., and Veneris, A. 2006. Efficient sat-based Boolean matching for FPGA technology mapping. In Proceedings 43rd Design Automation Conference. ACM, New York, 466--471.
22. Sinz, C. 2008. SAT-Race 2008. http://baldur.iti.uka.de/sat-race-2008.
23. Smith, A., Veneris, A. G., Ali, M. F., and Viglas, A. 2003. Fault diagnosis and logic debugging using Boolean satisfiability. IEEE Trans. CAD 24, 10, 1606--1621.
24. Spence, I. 2008. tts: A SAT-solver for small, difficult instances. J. Satisfiability Boolean Model. Comput. 4, 173--190.
25. Whitesitt, J. E. 1995. Boolean Algebra and Its Applications. Dover Publications, Mineola, NY.
26. Xu, K., Boussemart, F., Hemery, F., and Lecoutre, C. 2007. Random constraint satisfaction: Easy generation of hard (satisfiable) instances. Artif. Intell. 171, 514--534.
27. Zhang, L. and Malik, S. 2002. The quest for efficient Boolean satisfiability solvers. In Proceedings of the 14th International Conference on Computer-Aided Verification. Springer-Verlag, Berlin, 17--36.

Index Terms

• Theory of computation

  • Computational complexity and cryptography

    • Interactive proof systems

    • Proof complexity

  • Logic

    • Proof theory

  • Models of computation

    • Computability

  • Randomness, geometry and discrete structures