skip to main content

AnICA: analyzing inconsistencies in microarchitectural code analyzers

Published:31 October 2022Publication History
Skip Abstract Section

Abstract

Microarchitectural code analyzers, i.e., tools that estimate the throughput of machine code basic blocks, are important utensils in the tool belt of performance engineers. Recent tools like llvm-mca, uiCA, and Ithemal use a variety of techniques and different models for their throughput predictions. When put to the test, it is common to see these state-of-the-art tools give very different results. These inconsistencies are either errors, or they point to different and rarely documented assumptions made by the tool designers.

In this paper, we present AnICA, a tool taking inspiration from differential testing and abstract interpretation to systematically analyze inconsistencies among these code analyzers. Our evaluation shows that AnICA can summarize thousands of inconsistencies in a few dozen descriptions that directly lead to high-level insights into the different behavior of the tools. In several case studies, we further demonstrate how AnICA automatically finds and characterizes known and unknown bugs in llvm-mca, as well as a quirk in AMD's Zen microarchitectures.

References

  1. Andreas Abel. 2020. Automatic Generation of Models of Microarchitectures. Ph. D. Dissertation. Universität des Saarlandes. https://d-nb.info/1212853466/34 Google ScholarGoogle Scholar
  2. Andreas Abel. 2022. DiffTune Revisited: A Simple Baseline for Evaluating Learned llvm-mca Parameters. In Machine Learning for Computer Architecture and Systems 2022. https://openreview.net/forum?id=dw4evoj6AE Google ScholarGoogle Scholar
  3. Andreas Abel and Jan Reineke. 2019. uops.info: Characterizing Latency, Throughput, and Port Usage of Instructions on Intel Microarchitectures. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’19). ACM, New York, NY, USA. 673–686. isbn:978-1-4503-6240-5 https://doi.org/10.1145/3297858.3304062 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Andreas Abel and Jan Reineke. 2020. nanoBench: A Low-Overhead Tool for Running Microbenchmarks on x86 Systems. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 34–46. https://doi.org/10.1109/ISPASS48437.2020.00014 Google ScholarGoogle ScholarCross RefCross Ref
  5. Andreas Abel and Jan Reineke. 2022. uiCA: Accurate Throughput Prediction of Basic Blocks on Recent Intel Microarchitectures. In ICS ’22: 2022 International Conference on Supercomputing, Virtual Event, June 28 - 30, 2022, Lawrence Rauchwerger, Kirk W. Cameron, Dimitrios S. Nikolopoulos, and Dionisios N. Pnevmatikatos (Eds.). ACM, 33:1–33:14. https://doi.org/10.1145/3524059.3532396 Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. AMD. 2017. Software Optimization Guide for AMD Family 17h Processors. AMD. Google ScholarGoogle Scholar
  7. Nathan L. Binkert, Bradford M. Beckmann, Gabriel Black, Steven K. Reinhardt, Ali G. Saidi, Arkaprava Basu, Joel Hestness, Derek Hower, Tushar Krishna, Somayeh Sardashti, Rathijit Sen, Korey Sewell, Muhammad Shoaib Bin Altaf, Nilay Vaish, Mark D. Hill, and David A. Wood. 2011. The gem5 Simulator. SIGARCH Comput. Archit. News, 39, 2 (2011), 1–7. https://doi.org/10.1145/2024716.2024718 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Igor Böhm, Björn Franke, and Nigel P. Topham. 2010. Cycle-Accurate Performance Modelling in an Ultra-Fast Just-in-Time Dynamic Binary Translation Instruction Set Simulator. In Proceedings of the 2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation (IC-SAMOS 2010), Samos, Greece, July 19-22, 2010, Fadi J. Kurdahi and Jarmo Takala (Eds.). IEEE, 1–10. https://doi.org/10.1109/ICSAMOS.2010.5642102 Google ScholarGoogle ScholarCross RefCross Ref
  9. Chad Brubaker, Suman Jana, Baishakhi Ray, Sarfraz Khurshid, and Vitaly Shmatikov. 2014. Using Frankencerts for Automated Adversarial Testing of Certificate Validation in SSL/TLS Implementations. In 2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21, 2014. IEEE Computer Society, 114–129. https://doi.org/10.1109/SP.2014.15 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Gregory J. Chaitin, Marc A. Auslander, Ashok K. Chandra, John Cocke, Martin E. Hopkins, and Peter W. Markstein. 1981. Register Allocation Via Coloring. Comput. Lang., 6, 1 (1981), 47–57. https://doi.org/10.1016/0096-0551(81)90048-5 Google ScholarGoogle ScholarCross RefCross Ref
  11. Guillaume Chatelet. 2018. llvm-exegesis: Automatic Measurement of Instruction Latency/Uops. https://lists.llvm.org/pipermail/llvm-dev/2018-March/121814.html Accessed: 2021-07-22 Google ScholarGoogle Scholar
  12. Yishen Chen, Ajay Brahmakshatriya, Charith Mendis, Alex Renda, Eric Atkinson, Ondrej Sỳkora, Saman Amarasinghe, and Michael Carbin. 2019. BHive: A Benchmark Suite and Measurement Framework for Validating x86-64 Basic Block Performance Models. In 2019 IEEE international symposium on workload characterization (IISWC). IEEE. https://doi.org/10.1109/IISWC47752.2019.9042166 Google ScholarGoogle ScholarCross RefCross Ref
  13. Yuting Chen and Zhendong Su. 2015. Guided Differential Testing of Certificate Validation in SSL/TLS implementations. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (Eds.). ACM, 793–804. https://doi.org/10.1145/2786805.2786835 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Patrick Cousot and Radhia Cousot. 1977. Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In Conference Record of the Fourth ACM Symposium on Principles of Programming Languages, Los Angeles, California, USA, January 1977, Robert M. Graham, Michael A. Harrison, and Ravi Sethi (Eds.). ACM, 238–252. https://doi.org/10.1145/512950.512973 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Nicolas Derumigny, Théophile Bastian, Fabian Gruber, Guillaume Iooss, Christophe Guillon, Louis-Noël Pouchet, and Fabrice Rastello. 2022. PALMED: Throughput Characterization for Superscalar Architectures. In IEEE/ACM International Symposium on Code Generation and Optimization, CGO 2022, Seoul, Korea, Republic of, April 2-6, 2022, Jae W. Lee, Sebastian Hack, and Tatiana Shpeisman (Eds.). IEEE, 106–117. https://doi.org/10.1109/CGO53902.2022.9741289 Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Andrea Di Biagio. 2018. llvm-mca: A Static Performance Analysis Tool. http://lists.llvm.org/pipermail/llvm-dev/2018-March/121490.html Google ScholarGoogle Scholar
  17. Rafael Dutra, Jonathan Bachrach, and Koushik Sen. 2019. GUIDEDSAMPLER: Coverage-guided Sampling of SMT Solutions. In 2019 Formal Methods in Computer Aided Design, FMCAD 2019, San Jose, CA, USA, October 22-25, 2019, Clark W. Barrett and Jin Yang (Eds.). IEEE, 203–211. https://doi.org/10.23919/FMCAD.2019.8894251 Google ScholarGoogle ScholarCross RefCross Ref
  18. Google. 2018. EXEgesis: Automatic Measurement of Instruction Latency/Uops. https://github.com/google/EXEgesis Accessed: 2021-07-22 Google ScholarGoogle Scholar
  19. Intel. 2012. Intel Architecture Code Analyzer. https://software.intel.com/en-us/articles/intel-architecture-code-analyzer Google ScholarGoogle Scholar
  20. Nathan Jay and Barton P. Miller. 2018. Structured Random Differential Testing of Instruction Decoders. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). 84–94. https://doi.org/10.1109/SANER.2018.8330199 Google ScholarGoogle ScholarCross RefCross Ref
  21. Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proceedings of the 2004 International Symposium on Code Generation and Optimization (CGO’04). Palo Alto, California. https://doi.org/10.1109/CGO.2004.1281665 Google ScholarGoogle ScholarCross RefCross Ref
  22. Jan Laukemann, Julian Hammer, Johannes Hofmann, Georg Hager, and Gerhard Wellein. 2018. Automated Instruction Stream Throughput Prediction for Intel and AMD Microarchitectures. In 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). 121–131. https://doi.org/10.1109/PMBS.2018.8641578 Google ScholarGoogle ScholarCross RefCross Ref
  23. LLVM. 2021. LLVM 13 Scheduling Model for AMD Zen/Zen+ CPUs. https://github.com/llvm/llvm-project/blob/release/13.x/llvm/lib/Target/X86/X86ScheduleZnver1.td Accessed: 2022-04-07 Google ScholarGoogle Scholar
  24. LLVM. 2021. llvm-exegesis - LLVM Machine Instruction Benchmark. https://llvm.org/docs/CommandGuide/llvm-exegesis.html Accessed: 2021-07-22 Google ScholarGoogle Scholar
  25. LLVM. 2021. llvm-mca - LLVM Machine Code Analyzer. https://llvm.org/docs/CommandGuide/llvm-mca.html Accessed: 2021-11-15 Google ScholarGoogle Scholar
  26. Valentin Jean Marie Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering, 1–1. https://doi.org/10.1109/TSE.2019.2946563 Google ScholarGoogle ScholarCross RefCross Ref
  27. William M. McKeeman. 1998. Differential Testing for Software. Digit. Tech. J., 10, 1 (1998), 100–107. http://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf Google ScholarGoogle Scholar
  28. Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin. 2019. Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.) (Proceedings of Machine Learning Research, Vol. 97). PMLR, Long Beach, California, USA. 4505–4515. http://proceedings.mlr.press/v97/mendis19a.html Google ScholarGoogle Scholar
  29. Georg Ofenbeck, Ruedi Steinmann, Victoria Caparrós Cabezas, Daniele G. Spampinato, and Markus Püschel. 2014. Applying the Roofline Model. In 2014 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2014, Monterey, CA, USA, March 23-25, 2014. IEEE Computer Society, 76–85. https://doi.org/10.1109/ISPASS.2014.6844463 Google ScholarGoogle ScholarCross RefCross Ref
  30. Oleksii Oleksenko, Christof Fetzer, Boris Köpf, and Mark Silberstein. 2021. Revizor: Fuzzing for Leaks in Black-box CPUs. CoRR, abs/2105.06872 (2021), arXiv:2105.06872. arxiv:2105.06872 Google ScholarGoogle Scholar
  31. Roberto Paleari, Lorenzo Martignoni, Giampaolo Fresi Roglia, and Danilo Bruschi. 2010. N-Version Disassembly: Differential Testing of X86 Disassemblers. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA ’10). Association for Computing Machinery, New York, NY, USA. 265–274. isbn:9781605588230 https://doi.org/10.1145/1831708.1831741 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Chris Parnin and Alessandro Orso. 2011. Are Automated Debugging Techniques Actually Helping Programmers? In Proceedings of the 20th International Symposium on Software Testing and Analysis, ISSTA 2011, Toronto, ON, Canada, July 17-21, 2011, Matthew B. Dwyer and Frank Tip (Eds.). ACM, 199–209. https://doi.org/10.1145/2001420.2001445 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Theofilos Petsios, Adrian Tang, Salvatore Stolfo, Angelos D. Keromytis, and Suman Jana. 2017. NEZHA: Efficient Domain-Independent Differential Testing. In 2017 IEEE Symposium on Security and Privacy (SP). 615–632. https://doi.org/10.1109/SP.2017.27 Google ScholarGoogle ScholarCross RefCross Ref
  34. Alex Renda, Yishen Chen, Charith Mendis, and Michael Carbin. 2020. DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 442–455. https://doi.org/10.1109/MICRO50266.2020.00045 Google ScholarGoogle ScholarCross RefCross Ref
  35. Fabian Ritter and Sebastian Hack. 2020. PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization. In Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2020, London, UK, June 15-20, 2020, Alastair F. Donaldson and Emina Torlak (Eds.). ACM, 608–622. https://doi.org/10.1145/3385412.3385995 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Fabian Ritter and Sebastian Hack. 2022. AnICA: Analyzing Inconsistencies in Microarchitectural Code Analyzers (Artifact). https://doi.org/10.5281/zenodo.6818171 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Andres Charif Rubial, Emmanuel Oseret, Jose Noudohouenou, William Jalby, and Ghislain Lartigue. 2014. CQA: A Code Quality Analyzer Tool at Binary Level. In 21st International Conference on High Performance Computing, HiPC 2014, Goa, India, December 17-20, 2014. IEEE Computer Society, 1–10. https://doi.org/10.1109/HiPC.2014.7116904 Google ScholarGoogle ScholarCross RefCross Ref
  38. Samuel Williams, Andrew Waterman, and David A. Patterson. 2009. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Commun. ACM, 52, 4 (2009), 65–76. https://doi.org/10.1145/1498765.1498785 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. William Woodruff, Niki Carroll, and Sebastiaan Peters. 2021. Differential Analysis of x86-64 Instruction Decoders. 152–161. https://doi.org/10.1109/SPW53761.2021.00029 Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. AnICA: analyzing inconsistencies in microarchitectural code analyzers

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Article Metrics

          • Downloads (Last 12 months)164
          • Downloads (Last 6 weeks)22

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!