Abstract
We present an algorithm for synthesizing a context-free grammar encoding the language of valid program inputs from a set of input examples and blackbox access to the program. Our algorithm addresses shortcomings of existing grammar inference algorithms, which both severely overgeneralize and are prohibitively slow. Our implementation, GLADE, leverages the grammar synthesized by our algorithm to fuzz test programs with structured inputs. We show that GLADE substantially increases the incremental coverage on valid inputs compared to two baseline fuzzers.
- A. Albarghouthi, S. Gulwani, and Z. Kincaid. Recursive program synthesis. In Computer Aided Verification, pages 934–950. Springer, 2013.Google Scholar
- S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, et al. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 86(8):1978–2001, 2013. Google Scholar
Digital Library
- D. Angluin. Learning regular sets from queries and counterexamples. Information and computation, 75(2):87–106, 1987. Google Scholar
Digital Library
- S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst. Finding bugs in dynamic web applications. In Proceedings of the 2008 international symposium on Software testing and analysis, pages 261–272. ACM, 2008. Google Scholar
Digital Library
- B. Bollig, J.-P. Katoen, C. Kern, M. Leucker, D. Neider, and D. R. Piegdon. libalf: The automata learning framework. In International Conference on Computer Aided Verification, pages 360–364. Springer, 2010. Google Scholar
Digital Library
- M. Botinˇcan and D. Babi´c. Sigma*: Symbolic learning of input-output specifications. In Proceedings of the 40th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 443–456, 2013. Google Scholar
Digital Library
- A. S. Boujarwah and K. Saleh. Compiler test case generation methods: a survey and assessment. Information and software technology, 39(9):617–625, 1997.Google Scholar
- J. Caballero, H. Yin, Z. Liang, and D. Song. Polyglot: Automatic extraction of protocol message format using dynamic binary analysis. In Proceedings of the 14th ACM conference on Computer and communications security, pages 317–329. ACM, 2007. Google Scholar
Digital Library
- C. Cadar and K. Sen. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82– 90, 2013. Google Scholar
Digital Library
- C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209–224, 2008. Google Scholar
Digital Library
- C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. Exe: automatically generating inputs of death. ACM Transactions on Information and System Security (TISSEC), 12(2):10, 2008. Google Scholar
Digital Library
- C. Y. Cho, D. Babic, P. Poosankam, K. Z. Chen, E. X. Wu, and D. Song. Mace: Model-inference-assisted concolic exploration for protocol and vulnerability discovery. In USENIX Security Symposium, pages 139–154, 2011. Google Scholar
Digital Library
- W. Choi, G. Necula, and K. Sen. Guided gui testing of android apps with minimal restart and approximate learning. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, pages 623–640, 2013. Google Scholar
Digital Library
- C. De la Higuera. Grammatical inference: learning automata and grammars. Cambridge University Press, 2010. Google Scholar
Digital Library
- ECMA International. Standard ECMA-262: ECMA 2015 Language Specification. 6 edition, June 2015.Google Scholar
- J. K. Feser, S. Chaudhuri, and I. Dillig. Synthesizing data structure transformations from input-output examples. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 229–239. ACM, 2015. Google Scholar
Digital Library
- J. E. Forrester and B. P. Miller. An empirical study of the robustness of windows nt applications using random testing. In Proceedings of the 4th USENIX Windows System Symposium, pages 59–68. Seattle, 2000. Google Scholar
Digital Library
- V. Ganesh, T. Leek, and M. Rinard. Taint-based directed whitebox fuzzing. In Proceedings of the 31st International Conference on Software Engineering, pages 474–484. IEEE Computer Society, 2009. Google Scholar
Digital Library
- D. Giannakopoulou, Z. Rakamari´c, and V. Raman. Symbolic learning of component interfaces. In International Static Analysis Symposium, pages 248–264. Springer, 2012. Google Scholar
Digital Library
- GNU. Gnu bison. https://www.gnu.org/software/ bison, 2014.Google Scholar
- GNU Grep. https://www.gnu.org/software/grep/ manual, 2016.Google Scholar
- P. Godefroid, N. Klarlund, and K. Sen. Dart: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 213–223. ACM, 2005. Google Scholar
Digital Library
- P. Godefroid, A. Kiezun, and M. Y. Levin. Grammar-based whitebox fuzzing. In Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 206–215, 2008. Google Scholar
Digital Library
- P. Godefroid, M. Y. Levin, D. A. Molnar, et al. Automated whitebox fuzz testing. In NDSS, volume 8, pages 151–166, 2008.Google Scholar
- E. M. Gold. Language identification in the limit. Information and control, 10(5):447–474, 1967.Google Scholar
Cross Ref
- S. Gulwani. Automating string processing in spreadsheets using input-output examples. In Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 317–330, 2011. Google Scholar
Digital Library
- W. R. Harris and S. Gulwani. Spreadsheet table transformations from examples. In Proceedings of the 32Nd ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 317–328, 2011. Google Scholar
Digital Library
- C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In Presented as part of the 21st USENIX Security Symposium (USENIX Security 12), pages 445–458, 2012. Google Scholar
Digital Library
- M. Höschele and A. Zeller. Mining input grammars from dynamic taints. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pages 720–725. ACM, 2016. Google Scholar
Digital Library
- L. Huang, J. Jia, B. Yu, B.-G. Chun, P. Maniatis, and M. Naik. Predicting execution time of computer programs using sparse polynomial regression. In Advances in Neural Information Processing Systems, pages 883–891, 2010. Google Scholar
Digital Library
- H. Ishizaka. Polynomial time learnability of simple deterministic languages. Machine Learning, 5(2):151–164, 1990. Google Scholar
Digital Library
- B. Knobe and K. Knobe. A method for inferring context-free grammars. Information and Control, 31(2):129–146, 1976.Google Scholar
Cross Ref
- L. Lee. Learning of context-free languages: A survey of the literature. Techn. Rep. TR-12-96, Harvard University, 1996.Google Scholar
- A. Leung, J. Sarracino, and S. Lerner. Interactive parser synthesis by example. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 565–574. ACM, 2015. Google Scholar
Digital Library
- Z. Lin and X. Zhang. Deriving input syntactic structure from execution. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, pages 83–93. ACM, 2008. Google Scholar
Digital Library
- Z. Lin, X. Zhang, and D. Xu. Reverse engineering input syntactic structure from program execution and its applications. Software Engineering, IEEE Transactions on, 36(5):688–703, 2010. Google Scholar
Digital Library
- C. Lindig. Random testing of c calling conventions. In Proceedings of the sixth international symposium on Automated analysis-driven debugging, pages 3–12. ACM, 2005. Google Scholar
Digital Library
- R. Majumdar and R.-G. Xu. Directed test generation using symbolic grammars. In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, pages 134–143. ACM, 2007. Google Scholar
Digital Library
- B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32–44, 1990. Google Scholar
Digital Library
- B. P. Miller, G. Cooksey, and F. Moore. An empirical study of the robustness of macos applications using random testing. In Proceedings of the 1st international workshop on Random testing, pages 46–54. ACM, 2006. Google Scholar
Digital Library
- M. Naik, H. Yang, G. Castelnuovo, and M. Sagiv. Abstractions from tests. pages 373–386, 2012. Google Scholar
Digital Library
- N. Nethercote and J. Seward. Valgrind: A framework for heavyweight dynamic binary instrumentation. In Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 89–100, 2007. Google Scholar
Digital Library
- P. Norvig. http://norvig.com/lispy.html, 2010.Google Scholar
- J. Oncina and P. Garc´ıa. Identifying regular languages in polynomial time. Advances in Structural and Syntactic Pattern Recognition, 5(99-108):15–20.Google Scholar
- Oracle America, Inc. The Java TM Virtual Machine Specification. 7 edition, July 2011.Google Scholar
- D. Perelman, S. Gulwani, D. Grossman, and P. Provost. Testdriven synthesis. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 408–418, 2014. Google Scholar
Digital Library
- O. Polozov and S. Gulwani. Flashmeta: A framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 107–126. ACM, 2015. Google Scholar
Digital Library
- P. Purdom. A sentence generator for testing parsers. BIT Numerical Mathematics, 12(3):366–375, 1972.Google Scholar
Digital Library
- M. Rinard. Acceptability-oriented computing. pages 221– 239, 2003. Google Scholar
Digital Library
- M. C. Rinard. Living in the comfort zone. pages 611–622, 2007. Google Scholar
Digital Library
- R. L. Sauder. A general test data generator for cobol. In Proceedings of the May 1-3, 1962, spring joint computer conference, pages 317–323. ACM, 1962. Google Scholar
Digital Library
- K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C, volume 30. ACM, 2005. Google Scholar
Digital Library
- R. Singh and S. Gulwani. Synthesizing number transformations from input-output examples. In Computer Aided Verification, pages 634–651. Springer, 2012. Google Scholar
Digital Library
- R. J. Solomonoff. A new method for discovering the grammars of phrase structure languages. In Information Processing. Unesco, Paris, 1960.Google Scholar
- Stack Overflow. http:// stackoverflow.com/questions/3809401/ what-is-a-good-regular-expression-to-match-a-url, 2010.Google Scholar
- A. Stolcke. Bayesian learning of probabilistic language models. PhD thesis. Google Scholar
Digital Library
- A. Stolcke and S. Omohundro. Inducing probabilistic grammars by bayesian model merging. Grammatical inference and applications, pages 106–118, 1994. Google Scholar
Digital Library
- Z. Su and G. Wassermann. The essence of command injection attacks in web applications. In Conference Record of the 33rd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 372–382, 2006. Google Scholar
Digital Library
- M. Sutton and A. Greene. The art of file format fuzzing. In Blackhat USA conference, 2005.Google Scholar
- M. Sutton, A. Greene, and P. Amini. Fuzzing: brute force vulnerability discovery. Pearson Education, 2007. Google Scholar
Digital Library
- The Flex Project. Flex: The fast lexical analyzer. http: //flex.sourceforge.net, 2008.Google Scholar
- A. Vardhan, K. Sen, M. Viswanathan, and G. Agha. Learning to verify safety properties. In International Conference on Formal Engineering Methods, pages 274–289. Springer, 2004.Google Scholar
- J. Viide, A. Helin, M. Laakso, P. Pietikäinen, M. Seppänen, K. Halunen, R. Puuperä, and J. Röning. Experiences with model inference assisted fuzzing. In WOOT, 2008. Google Scholar
Digital Library
- W3C. https://www.w3.org/TR/2008/ REC-xml-20081126, 2008.Google Scholar
- T. Wang, T. Wei, G. Gu, and W. Zou. Taintscope: A checksumaware directed fuzzing tool for automatic software vulnerability detection. In Security and privacy (SP), 2010 IEEE symposium on, pages 497–512. IEEE, 2010. Google Scholar
Digital Library
- G. Wondracek, P. M. Comparetti, C. Kruegel, E. Kirda, and S. S. S. Anna. Automatic network protocol analysis. In NDSS, volume 8, pages 1–14, 2008.Google Scholar
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in c compilers. In Proceedings of the 32Nd ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 283–294, 2011. Google Scholar
Digital Library
- M. Zalewski. American fuzzy lop. http://lcamtuf. coredump.cx/afl, 2015.Google Scholar
Index Terms
Synthesizing program input grammars
Recommendations
Synthesizing program input grammars
PLDI 2017: Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and ImplementationWe present an algorithm for synthesizing a context-free grammar encoding the language of valid program inputs from a set of input examples and blackbox access to the program. Our algorithm addresses shortcomings of existing grammar inference algorithms,...
REINAM: reinforcement learning for input-grammar inference
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringProgram input grammars (i.e., grammars encoding the language of valid program inputs) facilitate a wide range of applications in software engineering such as symbolic execution and delta debugging. Grammars synthesized by existing approaches can cover ...
Mining input grammars from dynamic control flow
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringOne of the key properties of a program is its input specification. Having a formal input specification can be critical in fields such as vulnerability analysis, reverse engineering, software testing, clone detection, or refactoring. Unfortunately, ...






Comments