Abstract
We present an automated technique for finding defects in compilers for graphics shading languages. key challenge in compiler testing is the lack of an oracle that classifies an output as correct or incorrect; this is particularly pertinent in graphics shader compilers where the output is a rendered image that is typically under-specified. Our method builds on recent successful techniques for compiler validation based on metamorphic testing, and leverages existing high-value graphics shaders to create sets of transformed shaders that should be semantically equivalent. Rendering mismatches are then indicative of shader compilation bugs. Deviant shaders are automatically minimized to identify, in each case, a minimal change to an original high-value shader that induces a shader compiler bug. We have implemented the approach as a tool, GLFuzz, targeting the OpenGL shading language, GLSL. Our experiments over a set of 17 GPU and driver configurations, spanning the main 7 GPU designers, have led to us finding and reporting more than 60 distinct bugs, covering all tested configurations. As well as defective rendering, these issues identify security-critical vulnerabilities that affect WebGL, including a significant remote information leak security bug where a malicious web page can capture the contents of other browser tabs, and a bug whereby visiting a malicious web page can lead to a ``blue screen of death'' under Windows 10. Our findings show that shader compiler defects are prevalent, and that metamorphic testing provides an effective means for detecting them automatically.
- AMD. 2016. Graphics Core Next Architecture, Generation 3. (2016). http://gpuopen.com/compute-product/ amd-gcn3-isa-architecture-manual/ .Google Scholar
- Android Community. 2017. OpenGL ES Testing. (2017). http://source.android.com/devices/graphics/testing.html .Google Scholar
- Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Trans. Software Eng. 41, 5 (2015), 507–525. Google Scholar
Digital Library
- A.S. Boujarwah and K. Saleh. 1997. Compiler test case generation methods: a survey and assessment. Information and Software Technology 39, 9 (1997), 617 – 625. Google Scholar
Cross Ref
- C.J. Burgess and M. Saidi. 1996. The automatic generation of test cases for optimizing Fortran compilers. Information and Software Technology 38, 2 (1996), 111 – 119. Google Scholar
- Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Bing Xie. 2017. Learning to prioritize test programs for compiler testing. In Proc. International Conference on Software Engineering, Sebastián Uchitel, Alessandro Orso, and Martin P. Robillard (Eds.). IEEE / ACM, 700–711. http://dl.acm.org/citation.cfm?id=3097451Google Scholar
Digital Library
- Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An empirical comparison of compiler testing techniques. In Proc. International Conference on Software Engineering. ACM, 180–190. Google Scholar
Digital Library
- T.Y. Chen, S.C. Cheung, and S.M. Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01. Hong Kong University of Science and Technology.Google Scholar
- Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming compiler fuzzers. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 197–208. Google Scholar
Digital Library
- Context. 2011. WebGL: More WebGL Security Flaws. (2011). https://www.contextis.com/resources/blog/ webgl-more-webgl-security-flaws/ .Google Scholar
- Pascal Cuoq, Benjamin Monate, Anne Pacalet, Virgile Prevosto, John Regehr, Boris Yakobowski, and Xuejun Yang. 2012. Testing Static Analyzers with Randomly Generated Programs. In Proc. NASA Formal Methods Symposium. Springer, 120–125. Google Scholar
Digital Library
- Alastair F. Donaldson. 2016. Crashes, Hangs and Crazy Images by Adding Zero: Fuzzing OpenGL Shader Compilers. (2016). https://medium.com/@afd_icl/crashes-hangs-and-crazy-images-by-adding-zero-689d15ce922b .Google Scholar
- Alastair F. Donaldson and Andrei Lascu. 2016. Metamorphic Testing for (Graphics) Compilers [Short Paper]. In Proc. International Workshop on Metamorphic Testing . ACM, 44–47. Google Scholar
Digital Library
- Alastair F. Donaldson and Paul Thomson. 2017. Automated Testing of Graphics Shader Compiler: Video Illustration of Security Bugs. (2017). https://youtu.be/d3CNfMoP2t8 .Google Scholar
- Google. 2017. ANGLE: Almost Native Graphics Layer Engine. (2017). https://chromium.googlesource.com/angle/angle .Google Scholar
- Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm testing. In Proc. International Symposium on Software Testing and Analysis . ACM, 78–88. Google Scholar
Digital Library
- Ralph Guderlei and Johannes Mayer. 2007. Towards Automatic Testing of Imaging Software by Means of Random and Metamorphic Testing. International Journal of Software Engineering and Knowledge Engineering 17, 6 (2007), 757–781. Google Scholar
Cross Ref
- Nicolai Hähnle. 2017. Piglit - OpenGL driver testing framework. (2017). https://people.freedesktop.org/~nh/piglit/ .Google Scholar
- Tahir Jameel, Mengxiang Lin, and Liu Chao. 2016. Metamorphic Relations Based Test Oracles for Image Processing Applications. International Journal of Software Innovation 4, 1 (2016), 16–30. Google Scholar
Digital Library
- John Kessenich, Dave Baldwin, and Randi Rost. 2016a. The OpenGL Shading Language, Language Version 4.50. (2016). https://www.opengl.org/registry/doc/GLSLangSpec.4.50.pdf .Google Scholar
- John Kessenich, Boaz Ouriel, and Raun Krisch. 2016b. SPIR-V Specification (Provisional). (2016). https://www.khronos.org/ registry/spir-v/specs/1.1/SPIRV.pdf .Google Scholar
- John Kessenich, Graham Sellers, and Dave Shreiner. 2016c. OpenGL Programming Guide: The Official Guide to Learning OpenGL, Version 4.5 with SPIR-V (9 ed.). Addison-Wesley.Google Scholar
- Khronos Group. 2014. WebGL Specification, Version 1.0.3. (2014). https://www.khronos.org/registry/webgl/specs/1.0/ .Google Scholar
- Khronos Group. 2015. Khronos Invites Industry Participation to Create Safety Critical Graphics and Compute Standards, https://www.khronos.org/news/press/ . (August 2015).Google Scholar
- Khronos Group. 2016. Vulkan 1.0.38 – A Specification. (2016). https://www.khronos.org/registry/vulkan/specs/1.0/pdf/ vkspec.pdf .Google Scholar
- Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation . ACM, 216–226. Google Scholar
Digital Library
- Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In Proc. ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications . ACM, 386–399. Google Scholar
Digital Library
- Sangho Lee, Youngsok Kim, Jangwoo Kim, and Jong Kim. 2014. Stealing Webpages Rendered on Your Browser by Exploiting GPU Vulnerabilities. In Proc. IEEE Symposium on Security and Privacy. IEEE, 19–33. Google Scholar
Digital Library
- Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation . ACM, 65–76. Google Scholar
Digital Library
- William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100–107.Google Scholar
- Microsoft. 2017a. Direct3D 12 Progrmming Guide. (2017). https://msdn.microsoft.com/en-us/library/windows/desktop/ dn899121(v=vs.85).aspx .Google Scholar
- Microsoft. 2017b. HLSL. (2017). https://msdn.microsoft.com/en-us/library/windows/desktop/bb509561(v=vs.85).aspx .Google Scholar
- Multicore Programming Group. 2017. A collection of shader compiler bugs. (2017). http://github.com/mc-imperial/ shader-compiler-bugs .Google Scholar
- Moritz Pflanzer, Alastair F. Donaldson, and Andrei Lascu. 2016. Automatic Test Case Reduction for OpenCL. In Proc. International Workshop on OpenCL . ACM, 1:1–1:12. Google Scholar
Digital Library
- John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 335–346. Google Scholar
Digital Library
- R.S. Scowen and Z.J. Ciechanowicz. 1983. Compiler validation—a survey. In PASCAL Compiler Validation, B.A. Wichmann and Z.J. Ciechanowicz (Eds.). Wiley-Blackwell, Chapter 13, 90–144.Google Scholar
- SecurityWeek. 2016. Code Execution Flaw Plagues Intel Graphics Driver. (2016). http://www.securityweek.com/ code-execution-flaw-plagues-intel-graphics-driver .Google Scholar
- Sergio Segura, Gordon Fraser, Ana B. Sánchez, and Antonio Ruiz Cortés. 2016. A Survey on Metamorphic Testing. IEEE Trans. Software Eng. 42, 9 (2016), 805–824. Google Scholar
Cross Ref
- Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding compiler bugs via live code mutation. In Proc. ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications . ACM, 849–863. Google Scholar
Digital Library
- Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique. In Proc. Asia Pacific Software Engineering Conference. IEEE, 270–279. Google Scholar
Digital Library
- Ilja van Sprundel. 2014. Windows Kernel Graphics Driver Attack Surface. (2014). https://www.blackhat.com/docs/us-14/ materials/us-14-vanSprundel-Windows-Kernel-Graphics-Driver-Attack-Surface.pdf .Google Scholar
- Elaine J. Weyuker. 1982. On Testing Non-Testable Programs. Comput. J. 25, 4 (1982), 465–470. Google Scholar
Cross Ref
- Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation . ACM, 283–294. Google Scholar
Digital Library
- Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and Isolating Failure-Inducing Input. IEEE Trans. Software Eng. 28, 2 (2002), 183–200. Google Scholar
Digital Library
Index Terms
Automated testing of graphics shader compilers
Recommendations
Metamorphic testing for (graphics) compilers
MET '16: Proceedings of the 1st International Workshop on Metamorphic TestingWe present strategies for metamorphic testing of compilers using opaque value injection, and experiences using the method to test compilers for the OpenGL shading language.
A hybrid GPU rasterized and ray traced rendering pipeline for real time rendering of per pixel effects
ICEC'12: Proceedings of the 11th international conference on Entertainment ComputingRendering in 3D games typically uses rasterization approaches in order to guarantee interactive frame rates, since ray tracing, a superior method for rendering photorealistic images, has greater computational cost. With the advent of massively parallel ...
Simplified photon mapping for real-time caustics rendering
The objective of this paper is to adapt photon mapping for real-time simulation of caustics. High-performance algorithm adapted for the GPU and implemented on the basis of cross-platform OpenGL and OpenCL APIs is proposed. For effective rendering of ...






Comments