skip to main content
research-article
Open Access

Automated testing of graphics shader compilers

Published:12 October 2017Publication History
Skip Abstract Section

Abstract

We present an automated technique for finding defects in compilers for graphics shading languages. key challenge in compiler testing is the lack of an oracle that classifies an output as correct or incorrect; this is particularly pertinent in graphics shader compilers where the output is a rendered image that is typically under-specified. Our method builds on recent successful techniques for compiler validation based on metamorphic testing, and leverages existing high-value graphics shaders to create sets of transformed shaders that should be semantically equivalent. Rendering mismatches are then indicative of shader compilation bugs. Deviant shaders are automatically minimized to identify, in each case, a minimal change to an original high-value shader that induces a shader compiler bug. We have implemented the approach as a tool, GLFuzz, targeting the OpenGL shading language, GLSL. Our experiments over a set of 17 GPU and driver configurations, spanning the main 7 GPU designers, have led to us finding and reporting more than 60 distinct bugs, covering all tested configurations. As well as defective rendering, these issues identify security-critical vulnerabilities that affect WebGL, including a significant remote information leak security bug where a malicious web page can capture the contents of other browser tabs, and a bug whereby visiting a malicious web page can lead to a ``blue screen of death'' under Windows 10. Our findings show that shader compiler defects are prevalent, and that metamorphic testing provides an effective means for detecting them automatically.

References

  1. AMD. 2016. Graphics Core Next Architecture, Generation 3. (2016). http://gpuopen.com/compute-product/ amd-gcn3-isa-architecture-manual/ .Google ScholarGoogle Scholar
  2. Android Community. 2017. OpenGL ES Testing. (2017). http://source.android.com/devices/graphics/testing.html .Google ScholarGoogle Scholar
  3. Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Trans. Software Eng. 41, 5 (2015), 507–525. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. A.S. Boujarwah and K. Saleh. 1997. Compiler test case generation methods: a survey and assessment. Information and Software Technology 39, 9 (1997), 617 – 625. Google ScholarGoogle ScholarCross RefCross Ref
  5. C.J. Burgess and M. Saidi. 1996. The automatic generation of test cases for optimizing Fortran compilers. Information and Software Technology 38, 2 (1996), 111 – 119. Google ScholarGoogle Scholar
  6. Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Bing Xie. 2017. Learning to prioritize test programs for compiler testing. In Proc. International Conference on Software Engineering, Sebastián Uchitel, Alessandro Orso, and Martin P. Robillard (Eds.). IEEE / ACM, 700–711. http://dl.acm.org/citation.cfm?id=3097451Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An empirical comparison of compiler testing techniques. In Proc. International Conference on Software Engineering. ACM, 180–190. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. T.Y. Chen, S.C. Cheung, and S.M. Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01. Hong Kong University of Science and Technology.Google ScholarGoogle Scholar
  9. Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming compiler fuzzers. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 197–208. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Context. 2011. WebGL: More WebGL Security Flaws. (2011). https://www.contextis.com/resources/blog/ webgl-more-webgl-security-flaws/ .Google ScholarGoogle Scholar
  11. Pascal Cuoq, Benjamin Monate, Anne Pacalet, Virgile Prevosto, John Regehr, Boris Yakobowski, and Xuejun Yang. 2012. Testing Static Analyzers with Randomly Generated Programs. In Proc. NASA Formal Methods Symposium. Springer, 120–125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Alastair F. Donaldson. 2016. Crashes, Hangs and Crazy Images by Adding Zero: Fuzzing OpenGL Shader Compilers. (2016). https://medium.com/@afd_icl/crashes-hangs-and-crazy-images-by-adding-zero-689d15ce922b .Google ScholarGoogle Scholar
  13. Alastair F. Donaldson and Andrei Lascu. 2016. Metamorphic Testing for (Graphics) Compilers [Short Paper]. In Proc. International Workshop on Metamorphic Testing . ACM, 44–47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Alastair F. Donaldson and Paul Thomson. 2017. Automated Testing of Graphics Shader Compiler: Video Illustration of Security Bugs. (2017). https://youtu.be/d3CNfMoP2t8 .Google ScholarGoogle Scholar
  15. Google. 2017. ANGLE: Almost Native Graphics Layer Engine. (2017). https://chromium.googlesource.com/angle/angle .Google ScholarGoogle Scholar
  16. Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm testing. In Proc. International Symposium on Software Testing and Analysis . ACM, 78–88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Ralph Guderlei and Johannes Mayer. 2007. Towards Automatic Testing of Imaging Software by Means of Random and Metamorphic Testing. International Journal of Software Engineering and Knowledge Engineering 17, 6 (2007), 757–781. Google ScholarGoogle ScholarCross RefCross Ref
  18. Nicolai Hähnle. 2017. Piglit - OpenGL driver testing framework. (2017). https://people.freedesktop.org/~nh/piglit/ .Google ScholarGoogle Scholar
  19. Tahir Jameel, Mengxiang Lin, and Liu Chao. 2016. Metamorphic Relations Based Test Oracles for Image Processing Applications. International Journal of Software Innovation 4, 1 (2016), 16–30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. John Kessenich, Dave Baldwin, and Randi Rost. 2016a. The OpenGL Shading Language, Language Version 4.50. (2016). https://www.opengl.org/registry/doc/GLSLangSpec.4.50.pdf .Google ScholarGoogle Scholar
  21. John Kessenich, Boaz Ouriel, and Raun Krisch. 2016b. SPIR-V Specification (Provisional). (2016). https://www.khronos.org/ registry/spir-v/specs/1.1/SPIRV.pdf .Google ScholarGoogle Scholar
  22. John Kessenich, Graham Sellers, and Dave Shreiner. 2016c. OpenGL Programming Guide: The Official Guide to Learning OpenGL, Version 4.5 with SPIR-V (9 ed.). Addison-Wesley.Google ScholarGoogle Scholar
  23. Khronos Group. 2014. WebGL Specification, Version 1.0.3. (2014). https://www.khronos.org/registry/webgl/specs/1.0/ .Google ScholarGoogle Scholar
  24. Khronos Group. 2015. Khronos Invites Industry Participation to Create Safety Critical Graphics and Compute Standards, https://www.khronos.org/news/press/ . (August 2015).Google ScholarGoogle Scholar
  25. Khronos Group. 2016. Vulkan 1.0.38 – A Specification. (2016). https://www.khronos.org/registry/vulkan/specs/1.0/pdf/ vkspec.pdf .Google ScholarGoogle Scholar
  26. Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation . ACM, 216–226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In Proc. ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications . ACM, 386–399. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Sangho Lee, Youngsok Kim, Jangwoo Kim, and Jong Kim. 2014. Stealing Webpages Rendered on Your Browser by Exploiting GPU Vulnerabilities. In Proc. IEEE Symposium on Security and Privacy. IEEE, 19–33. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation . ACM, 65–76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100–107.Google ScholarGoogle Scholar
  31. Microsoft. 2017a. Direct3D 12 Progrmming Guide. (2017). https://msdn.microsoft.com/en-us/library/windows/desktop/ dn899121(v=vs.85).aspx .Google ScholarGoogle Scholar
  32. Microsoft. 2017b. HLSL. (2017). https://msdn.microsoft.com/en-us/library/windows/desktop/bb509561(v=vs.85).aspx .Google ScholarGoogle Scholar
  33. Multicore Programming Group. 2017. A collection of shader compiler bugs. (2017). http://github.com/mc-imperial/ shader-compiler-bugs .Google ScholarGoogle Scholar
  34. Moritz Pflanzer, Alastair F. Donaldson, and Andrei Lascu. 2016. Automatic Test Case Reduction for OpenCL. In Proc. International Workshop on OpenCL . ACM, 1:1–1:12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 335–346. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. R.S. Scowen and Z.J. Ciechanowicz. 1983. Compiler validation—a survey. In PASCAL Compiler Validation, B.A. Wichmann and Z.J. Ciechanowicz (Eds.). Wiley-Blackwell, Chapter 13, 90–144.Google ScholarGoogle Scholar
  37. SecurityWeek. 2016. Code Execution Flaw Plagues Intel Graphics Driver. (2016). http://www.securityweek.com/ code-execution-flaw-plagues-intel-graphics-driver .Google ScholarGoogle Scholar
  38. Sergio Segura, Gordon Fraser, Ana B. Sánchez, and Antonio Ruiz Cortés. 2016. A Survey on Metamorphic Testing. IEEE Trans. Software Eng. 42, 9 (2016), 805–824. Google ScholarGoogle ScholarCross RefCross Ref
  39. Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding compiler bugs via live code mutation. In Proc. ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications . ACM, 849–863. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique. In Proc. Asia Pacific Software Engineering Conference. IEEE, 270–279. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Ilja van Sprundel. 2014. Windows Kernel Graphics Driver Attack Surface. (2014). https://www.blackhat.com/docs/us-14/ materials/us-14-vanSprundel-Windows-Kernel-Graphics-Driver-Attack-Surface.pdf .Google ScholarGoogle Scholar
  42. Elaine J. Weyuker. 1982. On Testing Non-Testable Programs. Comput. J. 25, 4 (1982), 465–470. Google ScholarGoogle ScholarCross RefCross Ref
  43. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation . ACM, 283–294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and Isolating Failure-Inducing Input. IEEE Trans. Software Eng. 28, 2 (2002), 183–200. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Automated testing of graphics shader compilers

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image Proceedings of the ACM on Programming Languages
        Proceedings of the ACM on Programming Languages  Volume 1, Issue OOPSLA
        October 2017
        1786 pages
        EISSN:2475-1421
        DOI:10.1145/3152284
        Issue’s Table of Contents

        Copyright © 2017 Owner/Author

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 12 October 2017
        Published in pacmpl Volume 1, Issue OOPSLA

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!