Abstract
Microarchitectural code analyzers, i.e., tools that estimate the throughput of machine code basic blocks, are important utensils in the tool belt of performance engineers. Recent tools like llvm-mca, uiCA, and Ithemal use a variety of techniques and different models for their throughput predictions. When put to the test, it is common to see these state-of-the-art tools give very different results. These inconsistencies are either errors, or they point to different and rarely documented assumptions made by the tool designers.
In this paper, we present AnICA, a tool taking inspiration from differential testing and abstract interpretation to systematically analyze inconsistencies among these code analyzers. Our evaluation shows that AnICA can summarize thousands of inconsistencies in a few dozen descriptions that directly lead to high-level insights into the different behavior of the tools. In several case studies, we further demonstrate how AnICA automatically finds and characterizes known and unknown bugs in llvm-mca, as well as a quirk in AMD's Zen microarchitectures.
- Andreas Abel. 2020. Automatic Generation of Models of Microarchitectures. Ph. D. Dissertation. Universität des Saarlandes. https://d-nb.info/1212853466/34
Google Scholar
- Andreas Abel. 2022. DiffTune Revisited: A Simple Baseline for Evaluating Learned llvm-mca Parameters. In Machine Learning for Computer Architecture and Systems 2022. https://openreview.net/forum?id=dw4evoj6AE
Google Scholar
- Andreas Abel and Jan Reineke. 2019. uops.info: Characterizing Latency, Throughput, and Port Usage of Instructions on Intel Microarchitectures. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’19). ACM, New York, NY, USA. 673–686. isbn:978-1-4503-6240-5 https://doi.org/10.1145/3297858.3304062
Google Scholar
Digital Library
- Andreas Abel and Jan Reineke. 2020. nanoBench: A Low-Overhead Tool for Running Microbenchmarks on x86 Systems. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 34–46. https://doi.org/10.1109/ISPASS48437.2020.00014
Google Scholar
Cross Ref
- Andreas Abel and Jan Reineke. 2022. uiCA: Accurate Throughput Prediction of Basic Blocks on Recent Intel Microarchitectures. In ICS ’22: 2022 International Conference on Supercomputing, Virtual Event, June 28 - 30, 2022, Lawrence Rauchwerger, Kirk W. Cameron, Dimitrios S. Nikolopoulos, and Dionisios N. Pnevmatikatos (Eds.). ACM, 33:1–33:14. https://doi.org/10.1145/3524059.3532396
Google Scholar
Digital Library
- AMD. 2017. Software Optimization Guide for AMD Family 17h Processors. AMD.
Google Scholar
- Nathan L. Binkert, Bradford M. Beckmann, Gabriel Black, Steven K. Reinhardt, Ali G. Saidi, Arkaprava Basu, Joel Hestness, Derek Hower, Tushar Krishna, Somayeh Sardashti, Rathijit Sen, Korey Sewell, Muhammad Shoaib Bin Altaf, Nilay Vaish, Mark D. Hill, and David A. Wood. 2011. The gem5 Simulator. SIGARCH Comput. Archit. News, 39, 2 (2011), 1–7. https://doi.org/10.1145/2024716.2024718
Google Scholar
Digital Library
- Igor Böhm, Björn Franke, and Nigel P. Topham. 2010. Cycle-Accurate Performance Modelling in an Ultra-Fast Just-in-Time Dynamic Binary Translation Instruction Set Simulator. In Proceedings of the 2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation (IC-SAMOS 2010), Samos, Greece, July 19-22, 2010, Fadi J. Kurdahi and Jarmo Takala (Eds.). IEEE, 1–10. https://doi.org/10.1109/ICSAMOS.2010.5642102
Google Scholar
Cross Ref
- Chad Brubaker, Suman Jana, Baishakhi Ray, Sarfraz Khurshid, and Vitaly Shmatikov. 2014. Using Frankencerts for Automated Adversarial Testing of Certificate Validation in SSL/TLS Implementations. In 2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21, 2014. IEEE Computer Society, 114–129. https://doi.org/10.1109/SP.2014.15
Google Scholar
Digital Library
- Gregory J. Chaitin, Marc A. Auslander, Ashok K. Chandra, John Cocke, Martin E. Hopkins, and Peter W. Markstein. 1981. Register Allocation Via Coloring. Comput. Lang., 6, 1 (1981), 47–57. https://doi.org/10.1016/0096-0551(81)90048-5
Google Scholar
Cross Ref
- Guillaume Chatelet. 2018. llvm-exegesis: Automatic Measurement of Instruction Latency/Uops. https://lists.llvm.org/pipermail/llvm-dev/2018-March/121814.html Accessed: 2021-07-22
Google Scholar
- Yishen Chen, Ajay Brahmakshatriya, Charith Mendis, Alex Renda, Eric Atkinson, Ondrej Sỳkora, Saman Amarasinghe, and Michael Carbin. 2019. BHive: A Benchmark Suite and Measurement Framework for Validating x86-64 Basic Block Performance Models. In 2019 IEEE international symposium on workload characterization (IISWC). IEEE. https://doi.org/10.1109/IISWC47752.2019.9042166
Google Scholar
Cross Ref
- Yuting Chen and Zhendong Su. 2015. Guided Differential Testing of Certificate Validation in SSL/TLS implementations. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (Eds.). ACM, 793–804. https://doi.org/10.1145/2786805.2786835
Google Scholar
Digital Library
- Patrick Cousot and Radhia Cousot. 1977. Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In Conference Record of the Fourth ACM Symposium on Principles of Programming Languages, Los Angeles, California, USA, January 1977, Robert M. Graham, Michael A. Harrison, and Ravi Sethi (Eds.). ACM, 238–252. https://doi.org/10.1145/512950.512973
Google Scholar
Digital Library
- Nicolas Derumigny, Théophile Bastian, Fabian Gruber, Guillaume Iooss, Christophe Guillon, Louis-Noël Pouchet, and Fabrice Rastello. 2022. PALMED: Throughput Characterization for Superscalar Architectures. In IEEE/ACM International Symposium on Code Generation and Optimization, CGO 2022, Seoul, Korea, Republic of, April 2-6, 2022, Jae W. Lee, Sebastian Hack, and Tatiana Shpeisman (Eds.). IEEE, 106–117. https://doi.org/10.1109/CGO53902.2022.9741289
Google Scholar
Digital Library
- Andrea Di Biagio. 2018. llvm-mca: A Static Performance Analysis Tool. http://lists.llvm.org/pipermail/llvm-dev/2018-March/121490.html
Google Scholar
- Rafael Dutra, Jonathan Bachrach, and Koushik Sen. 2019. GUIDEDSAMPLER: Coverage-guided Sampling of SMT Solutions. In 2019 Formal Methods in Computer Aided Design, FMCAD 2019, San Jose, CA, USA, October 22-25, 2019, Clark W. Barrett and Jin Yang (Eds.). IEEE, 203–211. https://doi.org/10.23919/FMCAD.2019.8894251
Google Scholar
Cross Ref
- Google. 2018. EXEgesis: Automatic Measurement of Instruction Latency/Uops. https://github.com/google/EXEgesis Accessed: 2021-07-22
Google Scholar
- Intel. 2012. Intel Architecture Code Analyzer. https://software.intel.com/en-us/articles/intel-architecture-code-analyzer
Google Scholar
- Nathan Jay and Barton P. Miller. 2018. Structured Random Differential Testing of Instruction Decoders. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). 84–94. https://doi.org/10.1109/SANER.2018.8330199
Google Scholar
Cross Ref
- Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proceedings of the 2004 International Symposium on Code Generation and Optimization (CGO’04). Palo Alto, California. https://doi.org/10.1109/CGO.2004.1281665
Google Scholar
Cross Ref
- Jan Laukemann, Julian Hammer, Johannes Hofmann, Georg Hager, and Gerhard Wellein. 2018. Automated Instruction Stream Throughput Prediction for Intel and AMD Microarchitectures. In 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). 121–131. https://doi.org/10.1109/PMBS.2018.8641578
Google Scholar
Cross Ref
- LLVM. 2021. LLVM 13 Scheduling Model for AMD Zen/Zen+ CPUs. https://github.com/llvm/llvm-project/blob/release/13.x/llvm/lib/Target/X86/X86ScheduleZnver1.td Accessed: 2022-04-07
Google Scholar
- LLVM. 2021. llvm-exegesis - LLVM Machine Instruction Benchmark. https://llvm.org/docs/CommandGuide/llvm-exegesis.html Accessed: 2021-07-22
Google Scholar
- LLVM. 2021. llvm-mca - LLVM Machine Code Analyzer. https://llvm.org/docs/CommandGuide/llvm-mca.html Accessed: 2021-11-15
Google Scholar
- Valentin Jean Marie Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering, 1–1. https://doi.org/10.1109/TSE.2019.2946563
Google Scholar
Cross Ref
- William M. McKeeman. 1998. Differential Testing for Software. Digit. Tech. J., 10, 1 (1998), 100–107. http://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf
Google Scholar
- Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin. 2019. Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.) (Proceedings of Machine Learning Research, Vol. 97). PMLR, Long Beach, California, USA. 4505–4515. http://proceedings.mlr.press/v97/mendis19a.html
Google Scholar
- Georg Ofenbeck, Ruedi Steinmann, Victoria Caparrós Cabezas, Daniele G. Spampinato, and Markus Püschel. 2014. Applying the Roofline Model. In 2014 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2014, Monterey, CA, USA, March 23-25, 2014. IEEE Computer Society, 76–85. https://doi.org/10.1109/ISPASS.2014.6844463
Google Scholar
Cross Ref
- Oleksii Oleksenko, Christof Fetzer, Boris Köpf, and Mark Silberstein. 2021. Revizor: Fuzzing for Leaks in Black-box CPUs. CoRR, abs/2105.06872 (2021), arXiv:2105.06872. arxiv:2105.06872
Google Scholar
- Roberto Paleari, Lorenzo Martignoni, Giampaolo Fresi Roglia, and Danilo Bruschi. 2010. N-Version Disassembly: Differential Testing of X86 Disassemblers. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA ’10). Association for Computing Machinery, New York, NY, USA. 265–274. isbn:9781605588230 https://doi.org/10.1145/1831708.1831741
Google Scholar
Digital Library
- Chris Parnin and Alessandro Orso. 2011. Are Automated Debugging Techniques Actually Helping Programmers? In Proceedings of the 20th International Symposium on Software Testing and Analysis, ISSTA 2011, Toronto, ON, Canada, July 17-21, 2011, Matthew B. Dwyer and Frank Tip (Eds.). ACM, 199–209. https://doi.org/10.1145/2001420.2001445
Google Scholar
Digital Library
- Theofilos Petsios, Adrian Tang, Salvatore Stolfo, Angelos D. Keromytis, and Suman Jana. 2017. NEZHA: Efficient Domain-Independent Differential Testing. In 2017 IEEE Symposium on Security and Privacy (SP). 615–632. https://doi.org/10.1109/SP.2017.27
Google Scholar
Cross Ref
- Alex Renda, Yishen Chen, Charith Mendis, and Michael Carbin. 2020. DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 442–455. https://doi.org/10.1109/MICRO50266.2020.00045
Google Scholar
Cross Ref
- Fabian Ritter and Sebastian Hack. 2020. PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization. In Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2020, London, UK, June 15-20, 2020, Alastair F. Donaldson and Emina Torlak (Eds.). ACM, 608–622. https://doi.org/10.1145/3385412.3385995
Google Scholar
Digital Library
- Fabian Ritter and Sebastian Hack. 2022. AnICA: Analyzing Inconsistencies in Microarchitectural Code Analyzers (Artifact). https://doi.org/10.5281/zenodo.6818171
Google Scholar
Digital Library
- Andres Charif Rubial, Emmanuel Oseret, Jose Noudohouenou, William Jalby, and Ghislain Lartigue. 2014. CQA: A Code Quality Analyzer Tool at Binary Level. In 21st International Conference on High Performance Computing, HiPC 2014, Goa, India, December 17-20, 2014. IEEE Computer Society, 1–10. https://doi.org/10.1109/HiPC.2014.7116904
Google Scholar
Cross Ref
- Samuel Williams, Andrew Waterman, and David A. Patterson. 2009. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Commun. ACM, 52, 4 (2009), 65–76. https://doi.org/10.1145/1498765.1498785
Google Scholar
Digital Library
- William Woodruff, Niki Carroll, and Sebastiaan Peters. 2021. Differential Analysis of x86-64 Instruction Decoders. 152–161. https://doi.org/10.1109/SPW53761.2021.00029
Google Scholar
Cross Ref
Index Terms
AnICA: analyzing inconsistencies in microarchitectural code analyzers
Recommendations
Hunting for bugs in code coverage tools via randomized differential testing
ICSE '19: Proceedings of the 41st International Conference on Software EngineeringReliable code coverage tools are critically important as it is heavily used to facilitate many quality assurance activities, such as software testing, fuzzing, and debugging. However, little attention has been devoted to assessing the reliability of ...
A comparison of code similarity analysers
Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications ...
Energy-efficient and high-performance instruction fetch using a block-aware ISA
ISLPED '05: Proceedings of the 2005 international symposium on Low power electronics and designThe front-end in superscalar processors must deliver high application performance in an energy-effective manner. Impediments such as multi-cycle instruction accesses, instruction-cache misses, and mispredictions reduce performance by 48% and increase ...






Comments