SIGACT News Complexity Theory Column 76: an atypical survey of typical-case heuristic algorithms

Abstract

Heuristic approaches often do so well that they seem to pretty much always give the right answer. How close can heuristic algorithms get to always giving the right answer, without inducing seismic complexity-theoretic consequences? This article first discusses how a series of results by Berman, Buhrman, Hartmanis, Homer, Longpré, Ogiwara, Schöning, and Watanabe, from the early 1970s through the early 1990s, explicitly or implicitly limited how well heuristic algorithms can do on NP-hard problems. In particular, many desirable levels of heuristic success cannot be obtained unless severe, highly unlikely complexity class collapses occur. Second, we survey work initiated by Goldreich and Wigderson, who showed how under plausible assumptions deterministic heuristics for randomized computation can achieve a very high frequency of correctness. Finally, we consider formal ways in which theory can help explain the effectiveness of heuristics that solve NP-hard problems in practice.

References

1. M. Alekhnovich and E. Ben-Sasson. Linear upper bounds for random walk on small density random 3-CNFs. SIAM Journal on Computing, 36(5):1248--1263, 2007.
2. S. Arora, D. Steurer, and A. Wigderson. Towards a study of low-complexity graphs.In Proceedings (Part I) of the 36th International Colloquium on Automata, Languages, and Programming, pages 119--131. Springer-Verlag Lecture Notes in Computer Science #5555, July 2009.
3. B. Barak. Truth vs. proof in computational complexity. Bulletin of the EATCS, 108, October 2012. To appear.
4. L. Berman and J. Hartmanis. On isomorphisms and density of NP and other complete sets. SIAM Journal on Computing, 6(2):305--322, 1977.
5. H. Buhrman and S. Homer. Superpolynomial circuits, almost sparse oracles, and the exponential hierarchy. In Proceedings of the 12th Conference on Foundations of Software Technology and Theoretical Computer Science, pages 116--127. Springer-Verlag Lecture Notes in Computer Science #652, December 1992.
6. H. Buhrman and J. Hitchcock. NP-hard sets are exponentially dense unless coNP⊆ NP/poly. In Proceedings of the 23rd Annual IEEE Conference on Computational Complexity, pages 1--7. IEEE Computer Society Press, June 2008.
7. P. Beame, R. Karp, T. Pitassi, and M. Saks. The efficiency of resolution and Davis-Putnam procedures. SIAM Journal on Computing, 31(4):1048--1075, 2002.
8. A. Bogdanov and M. Safra. Hardness amplification for errorless heuristics. In Proceedings of the 49th IEEE Symposium on Foundations of Computer Science, pages 418--426. IEEE Computer Society Press, October 2007.
9. A. Bogdanov and L. Trevisan. Average-Case Complexity. Now Publishers, 2006.
10. E. Ben-Sasson and A. Wigderson. Short proofs are narrow---Resolution made simple. Journal of the ACM, 48(2):149--169, 2001.
11. J. Cai. Sp2 ⊆ ZPPNP. In Proceedings of the 42nd IEEE Symposium on Foundations of Computer Science, pages 620--629. IEEE Computer Society Press, October 2001.
12. J. Cai, V. Chakaravarthy, L. Hemaspaandra, and M. Ogihara. Competing provers yield improved Karp--Lipton collapse results. Information and Computation, 198(1):1--23, 2005.
13. C. Cadar, V. Ganesh, P. Pawlowski, D. Dill, and D. Engler. EXE: Automatically generating inputs of death. ACM Transactions on Information and System Security, 12(2), 2008.
14. B. Dilkina, C. Gomes, Y. Malitsky, A. Sabharwal, and M. Sellmann. Backdoors to combinatorial optimization: Feasibility and optimality. In Proceedings of the 6th International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, pages 56--70. Springer-Verlag Lecture Notes in Computer Science #5547, May 2009.
15. B. Dilkina, C. Gomes, and A. Sabharwal. Tradeoffs in the complexity of backdoor detection. In Proceedings of the 13th International Conference on Principles and Practice of Constraint Programming, pages 256--270. Springer-Verlag Lecture Notes in Computer Science #4741, September 2007.
16. B. Dilkina, C. Gomes, and A. Sabharwal. Backdoors in the context of learning. In Proceedings of the 12th International Conference on Theory and Applications of Satisfiability Testing, pages 73--79. Springer-Verlag Lecture Notes in Computer Science #5584, June/July 2009.
17. G. Erdélyi, L. Hemaspaandra, J. Rothe, and H. Spakowski. Generalized juntas and NP-hard sets. Theoretical Computer Science, 410(38--40):3995--4000, 2009.
18. U. Feige. Relations between average case complexity and approximation complexity. In Proceedings of the 34th ACM Symposium on Theory of Computing, pages 534--543. ACM Press, May 2002.
19. P. Faliszewski, E. Hemaspaandra, and L. Hemaspaandra. The complexity of manipulative attacks in nearly single-peaked electorates. In Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge, pages 228--237. ACM Digital Library, July 2011.
20. M. Furst, J. Saxe, and M. Sipser. Parity, circuits, and the polynomial-time hierarchy.Mathematical Systems Theory, 17(1):13--27, 1984.
21. C. Glaser and L. Hemaspaandra. A moment of perfect clarity II: Consequences of sparse sets hard for NP with respect to weak reductions. SIGACT News, 31(4):39--51, 2000.
22. J. Gill. Computational complexity of probabilistic Turing machines. SIAM Journal on Computing, 6(4):675--695, 1977.
23. M. Garey and D. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman and Company, 1979.
24. C. Gomes, H. Kautz, A. Sabharwal, and B. Selman. Satisfiability solvers. In F. van Harmelen, V. Lifschitz, and B. Porter, editors, Handbook of Knowledge Representation. Elsevier, 2008.
25. S. Gaspers and S. Szeider. Backdoors to satisfaction. In The Multivariate Algorithmic Revolution and Beyond, pages 287--317. Springer-Verlag Lecture Notes in Computer Science #7370, 2012.
26. C. Gomes, B. Selman, and H. Kautz. Boosting combinatorial search through randomization. In Proceedings of the 15th National Conference on Artificial Intelligence, pages 431--437, July 1998.
27. O. Goldreich and A. Wigderson. Derandomization that is rarely wrong from short advice that is typically good. In Proceedings of the 6th International Workshop on Randomization and Approximation Techniques, pages 209--223, September 2002.
28. A. Haken. The intractability of resolution. Theoretical Computer Science, 39(2--3):297--308, 1985.
29. E. Hemaspaandra, L. Hemaspaandra, and C. Menton. Search versus decision for election manipulation problems. Technical Report arXiv:1202.6641 {cs.GT}, Computing Research Repository, arXiv.org/corr/, February 2012. Revised, March 2012.
30. J. Hartmanis, N. Immerman, and V. Sewelson. Sparse sets in NP?P: EXPTIME versus NEXPTIME. Information and Control, 65(2--3):159--181, 1985.
31. S. Homer and L. Longpré. On reductions of NP sets to sparse sets. Journal of Computer and System Sciences, 48(2):324--336, 1994.
32. J. Hartmanis and S. Mahaney. An essay about research on sparse NP complete sets. In Proceedings of the 9th Symposium on Mathematical Foundations of Computer Science, pages 40--57. Springer-Verlag Lecture Notes in Computer Science #88, September 1980.
33. L. Hemachandra, M. Ogiwara, and O. Watanabe. How hard are sparse sets? In Proceedings of the 7th Structure in Complexity Theory Conference, pages 222--238. IEEE Computer Society Press, June 1992.
34. R. Impagliazzo. A personal view of average-case complexity. In Proceedings of the 10th Structure in Complexity Theory Conference, pages 134--147. IEEE Computer Society Press, June 1995.
35. R. Impagliazzo. Relativized separations of worst-case and average-case complexities for NP. In Proceedings of the 26th Annual IEEE Conference on Computational Complexity, pages 104--114. IEEE Computer Society Press, June 2011.
36. Y. Interian. Backdoor sets for random 3-SAT. In Sixth International Conference on Theory and Applications of Satisfiability Testing (informal proceedings), pages 231--238, 2003.
37. R. Kaivola, R. Ghughal, N. Narasimhan, A. Telfer, J. Whittemore, S. Pandav, A. Slobodová, C. Taylor, V. Frolov, E. Reeber, and A. Naik. Replacing testing with formal verification in Intel Core i7 processor execution engine validation. In Proceedings of the 21st International Conference on Computer Aided Verification, pages 414--429. Springer-Verlag Lecture Notes in Computer Science #5643, June/July 2009.
38. V. Kabanets and R. Impagliazzo. Derandomizing polynomial identity tests means proving circuit lower bounds. Computational Complexity, 13(1--2):1--46, 2004.
39. R. Karp and R. Lipton. Some connections between nonuniform and uniform complexity classes. In Proceedings of the 12th ACM Symposium on Theory of Computing, pages 302--309. ACM Press, April 1980. An extended version has also appeared as: Turing machines that take advice, L'Enseignement Mathématique, 2nd series, 28:191--209, 1982.
40. J. Kinne, D. van Melkebeek, and R. Shaltiel. Pseudorandom generators, typicallycorrect derandomization, and circuit lower bounds. Computational Complexity, 21(1):3--61, 2012.
41. L. Levin. Average case complete problems. SIAM Journal on Computing, 15(1):285--286, 1986.
42. R. Ladner, N. Lynch, and A. Selman. A comparison of polynomial time reducibilities. Theoretical Computer Science, 1(2):103--124, 1975.
43. S. Mahaney. Sparse complete sets for NP: Solution of a conjecture of Berman and Hartmanis. Journal of Computer and System Sciences, 25(2):130--143, 1982.
44. S. Mahaney. Sparse sets and reducibilities. In R. Book, editor, Studies in Complexity Theory, pages 63--118. John Wiley and Sons, 1986.
45. D. van Melkebeek and R. Santhanam. Holographic proofs and derandomization. SIAM Journal on Computing, 35(1):59--90, 2005.
46. M. Ogiwara and O. Watanabe. On polynomial-time bounded truth-table reducibility of NP sets to sparse sets. SIAM Journal on Computing, 20(3):471--483, June 1991.
47. C. Papadimitriou. Computational Complexity. Addison Wesley, 1994.
48. A. Procaccia and J. Rosenschein. Junta distributions and the average-case complexity of manipulating elections. Journal of Artificial Intelligence Research, 28:157--181, 2007.
49. R. Raz. Resolution lower bounds for the weak pigeonhole principle. Journal of the ACM, 51(2):115--138, 2004.
50. O. Reingold. Undirected connectivity in log-space. Journal of the ACM, 55(4), 2008.
51. U. Schöning. Complete sets and closeness to complexity classes. Mathematical Systems Theory, 19(1):29--42, 1986.
52. N. Segerlind. The complexity of propositional proofs. Bulletin of Symbolic Logic, 13(4):417--481, 2007.
53. R. Shaltiel. Typically-correct derandomization. SIGACT News, 41(2):57--72, 2010.
54. M. Samer and S. Szeider. Backdoor trees. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pages 363--368. AAAI Press, July 2008.
55. L. Trevisan. Lecture notes on computational complexity. www.cs.berkeley.edu/¿luca/notes/complexitynotes02.pdf (Lecture 12), 2002.
56. R. Vanderbei. Linear Programming: Foundations and Extensions. Springer, 2001.
57. R. Williams, C. Gomes, and B. Selman. Backdoors to typical case complexity. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, pages 1173--1178. Morgan Kaufmann, August 2003.
58. L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown. SATzilla: Portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 32:565--606, 2008.
59. Y. Yesha. On certain polynomial-time truth-table reducibilities of complete sets to sparse sets. SIAM Journal on Computing, 12(3):411--425, 1983.
60. P. Young. How reductions to sparse sets collapse the polynomial-time hierarchy: A primer. SIGACT News, 23, 1992. Part I (#3, pages 107--117), Part II (#4, pages 83--94), and Corrigendum to Part I (#4, page 94).
61. M. Zimand. Exposure-resilient extractors and the derandomization of probabilistic sublinear time. Computational Complexity, 17(2):220--253, 2008.

Index Terms

• Theory of computation

  • Computational complexity and cryptography

    • Complexity classes