Abstract
The reliability of compilers, interpreters, and development environments for programming languages is essential for effective software development and maintenance. They are often tested only as an afterthought. Languages with a smaller scope, such as domain-specific languages, often remain untested. General-purpose testing techniques and test case generation methods fall short in providing a low-threshold solution for test-driven language development. In this paper we introduce the notion of a language-parametric testing language (LPTL) that provides a reusable, generic basis for declaratively specifying language definition tests. We integrate the syntax, semantics, and editor services of a language under test into the LPTL for writing test inputs. This paper describes the design of an LPTL and the tool support provided for it, shows use cases using examples, and describes our implementation in the form of the Spoofax testing language.
- K. Beck. Test-driven development: by example. Addison-Wesley Professional, 2003. Google Scholar
Digital Library
- B. Beizer. Software testing techniques. Dreamtech Press, 2002.Google Scholar
- A. Boujarwah and K. Saleh. Compiler test case generation methods: a survey and assessment. Information and Software Technology, 39(9):617--625, 1997.Google Scholar
Cross Ref
- J. Bovet and T. Parr. ANTLRWorks: an ANTLR grammar development environment. Software: Practice and Experience, 38(12):1305--1332, 2008. Google Scholar
Digital Library
- M. Bravenboer, K. T. Kalleberg, R. Vermaas, and E. Visser. Stratego/XT 0.17. A language and toolset for program transformation. Science of Computer Programming, 72(1-2):52-- 70, 2008. Google Scholar
Digital Library
- M. Bravenboer and E. Visser. Concrete syntax for objects: domain-specific language embedding and assimilation without restrictions. In J. M. Vlissides and D. C. Schmidt, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2004, pages 365--383. ACM, 2004. Google Scholar
Digital Library
- P. Charles, R. M. Fuhrer, S. M. Sutton, Jr., E. Duesterwald, and J. Vinju. Accelerating the creation of customized, language-specific IDEs in Eclipse. In Proceeding of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications, OOPSLA '09, pages 191--206. ACM, 2009. Google Scholar
Digital Library
- B. Daniel, D. Dig, K. Garcia, and D. Marinov. Automated testing of refactoring engines. In I. Crnkovic and A. Bertolino, editors, 6th joint meeting of the European Software Engineering Conference and the Int. Symposium on Foundations of Software Engineering, pages 185--194. ACM, 2007. Google Scholar
Digital Library
- M. de Jonge, E. Nilsson-Nyman, L. C. L. Kats, and E. Visser. Natural and flexible error recovery for generated parsers. In M. van den Brand, D. Gasevic, and J. Gray, editors, Software Language Engineering, SLE 2009, volume 5969 of LNCS, pages 204--223. Springer, 2009. Google Scholar
Digital Library
- P. Degano and C. Priami. Comparison of syntactic error handling in LR parsers. Software -- Practice and Experience, 25(6):657--679, 1995. Google Scholar
Digital Library
- S. Efftinge and M. Völter. oAW xText - a framework for textual DSLs. In Modeling Symposium, Eclipse Summit, 2006.Google Scholar
- M. Fowler. Refactoring: Improving the design of existing code. In D. Wells and L. A. Williams, editors, Extreme Programming and Agile Methods - XP/Agile Universe 2002, Second XP Universe and First Agile Universe Conference, volume 2418 of LNCS, page 256. Springer, 2002. Google Scholar
Digital Library
- M. Fowler. Language workbenches: The killer-app for domain specific languages? http://martinfowler.com/articles/languageWorkbench.html, 2005.Google Scholar
- J. D. Frens and A. Meneely. Fifteen compilers in fifteen days. In D. Baldwin, P. T. Tymann, S. M. Haller, and I. Russell, editors, 39th Technical Symposium on Computer Science Education, 2006, pages 92--96. ACM, 2006. Google Scholar
Digital Library
- E. Gamma and K. Beck. Test infected: Programmers love writing tests. Java Report, 3(7):37--50, 1998.Google Scholar
- J. B. Goodenough. The Ada compiler validation capability. In Proceedings of the ACM-SIGPLAN symposium on Ada programming language, SIGPLAN '80, pages 1--8. ACM, 1980. Google Scholar
Digital Library
- gUnit - grammar unit testing. http://www.antlr.org/wiki/display/ANTLR3/gUnit+-+Grammar+ Unit+Testing.Google Scholar
- R. Gómez, J. C. Augusto, and A. Galton. Testing an event specification language. In Software Engineering & Knowledge Engineering, SEKE 2001, pages 341--345, 2001.Google Scholar
- P. Hamill. Unit Test Frameworks, chapter. Chapter 3: The xUnit Family of Unit Test Frameworks. O'Reilly, 2004.Google Scholar
- Z. Hemel and E. Visser. Declaratively programming the mobile web with mobl. In Proceedings of the 26th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2011, Portland, Oregon, USA, 2011. ACM. Google Scholar
Digital Library
- P. Hudak. Modular domain specific languages and tools. In Software Reuse, ICSR '98. Computer Society, jun 1998. Google Scholar
Digital Library
- Jacks (Jacks is an Automated Compiler Killing Suite). http://sources.redhat.com/mauve/jacks. html.Google Scholar
- L. C. L. Kats, M. de Jonge, E. Nilsson-Nyman, and E. Visser. Providing rapid feedback in generated modular language environments: adding error recovery to scannerless generalized- LR parsing. In S. Arora and G. T. Leavens, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2009, pages 445--464. ACM, 2009. Google Scholar
Digital Library
- L. C. L. Kats, K. T. Kalleberg, and E. Visser. Interactive disambiguation of meta programs with concrete object syntax. In M. van den Brand, B. Malloy, and S. Staab, editors, Software Language Engineering, SLE 2010, LNCS. Springer, 2011. Google Scholar
Digital Library
- L. C. L. Kats and E. Visser. The Spoofax language workbench: rules for declarative specification of languages and IDEs. In W. R. Cook, S. Clarke, and M. C. Rinard, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2010, pages 444--463. ACM, 2010. Google Scholar
Digital Library
- L. C. L. Kats, E. Visser, and G. Wachsmuth. Pure and declarative syntax definition: paradise lost and regained. In W. R. Cook, S. Clarke, and M. C. Rinard, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2010, pages 918--932. ACM, 2010. Google Scholar
Digital Library
- A. Kossatchev and M. Posypkin. Survey of compiler testing methods. Programming and Computer Software, 31(1):10-- 19, 2005. Google Scholar
Digital Library
- H. Krahn, B. Rumpe, and S. Völkel. Monticore: Modular development of textual domain specific languages. In R. F. Paige and B. Meyer, editors, Objects, Components, Models and Patterns, TOOLS EUROPE 2008, volume 11 of Lecture Notes in Business Information Processing, pages 297--315. Springer, 2008.Google Scholar
- R. Lämmel. Grammar testing. In H. Hußmann, editor, Fundamental Approaches to Software Engineering, FASE 2001, volume 2029 of LNCS, pages 201--216. Springer, 2001. Google Scholar
Digital Library
- B. A. Malloy, J. F. Power, and J. T. Waldron. Applying software engineering techniques to parser design: the development of a C# parser. In SAICSIT '02: 2002 research conference of the South African institute of computer scientists and information technologists on Enablement through technology. South African Institute for Computer Scientists and Information Technologists, 2002. Google Scholar
Digital Library
- G. Myers. The art of software testing, 2nd edition. WileyIndia, 2008. Google Scholar
Digital Library
- R. W. Selby, editor. Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research. Wiley-Computer Society Press, 2007. Google Scholar
Digital Library
- M. Strembeck and U. Zdun. An approach for the systematic development of domain-specific languages. Software: Practice and Experience, 39(15):1253--1292, 2009. Google Scholar
Digital Library
- L. Tratt. Domain specific language implementation via compile-time meta-programming. Trans. Program. Lang. Syst., 30(6), 2008. Google Scholar
Digital Library
- E. Visser. Syntax Definition for Language Prototyping. PhD thesis, University of Amsterdam, September 1997.Google Scholar
- E. Visser. WebDSL: A case study in domain-specific language engineering. In R. Lämmel, J. Visser, and J. Saraiva, editors, Generative and Transformational Techniques in Software Engineering II, Int. Summer School, GTTSE 2007, volume 5235 of LNCS, pages 291--373. Springer, 2007. Google Scholar
Digital Library
- M. Voelter and K. Solomatov. Language modularization and composition with projectional language workbenches illustrated with MPS. In M. van den Brand, B. Malloy, and S. Staab, editors, Software Language Engineering, SLE 2010, LNCS. Springer, 2010.Google Scholar
- B. Wichmann and Z. Ciechanowicz. Pascal compiler validation. John Wiley & Sons, Inc. New York, NY, USA, 1983. Google Scholar
Digital Library
- H. Wu, J. Gray, and M. Mernik. Grammar-driven generation of domain-specific language debuggers. Software: Practice and Experience, 38(10):1073--1103, 2008. Google Scholar
Digital Library
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in C compilers. In Conference on Programming Language Design and Implementation, PLDI 2011. Press, June 2011. Google Scholar
Digital Library
Index Terms
Integrated language definition testing: enabling test-driven language development
Recommendations
The spoofax language workbench: rules for declarative specification of languages and IDEs
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applicationsSpoofax is a language workbench for efficient, agile development of textual domain-specific languages with state-of-the-art IDE support. Spoofax integrates language processing techniques for parser generation, meta-programming, and IDE development into ...
The spoofax language workbench: rules for declarative specification of languages and IDEs
OOPSLA '10Spoofax is a language workbench for efficient, agile development of textual domain-specific languages with state-of-the-art IDE support. Spoofax integrates language processing techniques for parser generation, meta-programming, and IDE development into ...
Pure and declarative syntax definition: paradise lost and regained
OOPSLA '10Syntax definitions are pervasive in modern software systems, and serve as the basis for language processing tools like parsers and compilers. Mainstream parser generators pose restrictions on syntax definitions that follow from their implementation ...







Comments