skip to main content
research-article
Public Access

A Systematic Framework to Identify Violations of Scenario-dependent Driving Rules in Autonomous Vehicle Software

Published:04 June 2021Publication History
Skip Abstract Section

Abstract

Safety compliance is paramount to the safe deployment of autonomous vehicle (AV) technologies in real-world transportation systems. As AVs will share road infrastructures with human drivers and pedestrians, it is an important requirement for AVs to obey standard driving rules. Existing AV software testing methods, including simulation and road testing, only check fundamental safety rules such as collision avoidance and safety distance. Scenario-dependent driving rules, including crosswalk and intersection rules, are more complicated because the expected driving behavior heavily depends on the surrounding circumstances. However, a testing framework is missing for checking scenario-dependent driving rules on various AV software.

In this paper, we design and implement a systematic framework AVChecker for identifying violations of scenario-dependent driving rules in AV software using formal methods. AVChecker represents both the code logic of AV software and driving rules in proposed formal specifications and leverages satisfiability modulo theory (SMT) solvers to identify driving rule violations. To improve the automation of systematic rule-based checking, AVChecker provides a powerful user interface for writing driving rule specifications and applies static code analysis to extract rule-related code logic from the AV software codebase. Evaluations on two open-source AV software platforms, Baidu Apollo and Autoware, uncover 19 true violations out of 28 real-world driving rules covering crosswalks, traffic lights, stop signs, and intersections. Seven of the violations can lead to severe risks of a collision with pedestrians or blocking traffic.

References

  1. 2017. Automated Driving Systems 2.0: A Vision for Safety. https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/13069a-ads2.0_090617_v9a_tag.pdf.Google ScholarGoogle Scholar
  2. 2019. A Matter of Trust Ford's Approach to Developing Self-driving Vehicles. https://media.ford.com/content/dam/fordmedia/pdf/Ford_AV_LLC_FINAL_HR_2.pdf.Google ScholarGoogle Scholar
  3. 2019. Apollo Auto: An open autonomous driving platform. https://github.com/ApolloAuto/apollo.Google ScholarGoogle Scholar
  4. 2019. General Motors 2018 Self-Driving Safety Report. https://www.gm.com/content/dam/company/docs/us/en/gmcom/gmsafetyreport.pdf .Google ScholarGoogle Scholar
  5. 2020. 2010 Georgia Code Title 40 - Motor Vehicles and Traffic. https://law.justia.com/codes/georgia/2010/title-40/chapter-6/article-5/40--6--91.Google ScholarGoogle Scholar
  6. 2020. Autoware: Open-source software for self-driving vehicles. https://gitlab.com/autowarefoundation/autoware.ai.Google ScholarGoogle Scholar
  7. 2020. Baidu Apollo's fuzzing support. https://github.com/ApolloAuto/apollo/commit/7aca63966211ceada44981d96b35a1252f9f1729.Google ScholarGoogle Scholar
  8. 2020. Louisiana DMV Handbook. https://driving-tests.org/wp-content/uploads/2018/03/LA_Guide-2017.pdf .Google ScholarGoogle Scholar
  9. 2020. New York State DMV: Driver's Manual and Practice Tests. https://dmv.ny.gov/driver-license/drivers-manual-practice-tests.Google ScholarGoogle Scholar
  10. 2020. ROS.org | Powering the world's robots. https://www.ros.org/.Google ScholarGoogle Scholar
  11. 2020. State of California DMV: California Driver Handbook. https://www.dmv.ca.gov/web/eng_pdf/dl600.pdf.Google ScholarGoogle Scholar
  12. 2020. State of Michigan DMV: What Every Driver Must Know. https://www.michigan.gov/documents/wedmk_16312_7.pdf.Google ScholarGoogle Scholar
  13. 2020. Waymo Safety Report. https://waymo.com/safety.Google ScholarGoogle Scholar
  14. Matthias Althoff and John M Dolan. 2011. Set-based computation of vehicle behaviors for the online verification of autonomous vehicles. In 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1162--1167.Google ScholarGoogle ScholarCross RefCross Ref
  15. Matthias Althoff and Sebastian Lutz. 2018. Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1326--1333.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Matthias Althoff, Olaf Stursberg, and Martin Buss. 2009. Model-based probabilistic collision detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems10, 2 (2009), 299--310.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, and Daniela Rus. 2020. Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation. IEEE Robotics and Automation Letters 5, 2 (2020), 1143--1150.Google ScholarGoogle ScholarCross RefCross Ref
  18. Mathieu Barbier, Alessandro Renzaglia, Jean Quilbeuf, Lukas Rummelhard, Anshul Paigwar, Christian Laugier, Axel Legay, Javier Ibañez-Guzmán, and Olivier Simonin. 2019. Validation of perception and decision-making systems for autonomous driving via statistical model checking. In 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE, 252--259.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Dirk Beyer and Matthias Dangl. 2016. SMT-based software model checking: an experimental comparison of four algorithms. In Working Conference on Verified Software: Theories, Tools, and Experiments. Springer, 181--198.Google ScholarGoogle ScholarCross RefCross Ref
  20. Dirk Beyer, Matthias Dangl, and Philipp Wendler. 2018. A unifying view on SMT-based software verification. Journal of automated reasoning60, 3 (2018), 299--335.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Nikolaj Bjørner. 2010. Linear quantifier elimination as an abstract decision procedure. In International Joint Conference on Automated Reasoning. Springer, 316--330.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. NTS Board. 2018. Preliminary report, highway, hwy18mh010.National Transpotation Safety Board, https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf, accessed(2018), 11--15.Google ScholarGoogle Scholar
  23. Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In 8th USENIX Symposium on Operating Systems Design and Implementation(OSDI 08). 209--224.Google ScholarGoogle Scholar
  24. Silviu S Craciunas and Ramon Serna Oliver. 2014. SMT-based task-and network-level static schedule generation for time-triggered networked systems. In Proceedings of the 22nd international conference on real-time networks and systems.45--54.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems.Google ScholarGoogle ScholarCross RefCross Ref
  26. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. arXiv preprint arXiv:1711.03938(2017).Google ScholarGoogle Scholar
  27. Hassan Eldib, Chao Wang, and Patrick Schaumont. 2014. SMT-based verification of software countermeasures against side-channel attacks. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 62--77.Google ScholarGoogle ScholarCross RefCross Ref
  28. Haoyang Fan, Fan Zhu, Changchun Liu, Liangliang Zhang, Li Zhuang, Dong Li, Weicheng Zhu, Jiangtao Hu, Hongye Li, and Qi Kong. 2018. Baidu apollo em motion planner.arXiv preprint arXiv:1807.08048(2018).Google ScholarGoogle Scholar
  29. Daniel S Fowler, Jeremy Bryans, Siraj Ahmed Shaikh, and Paul Wooderson. 2018. Fuzz testing for automotive cyber-security. In 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops(DSN-W). IEEE, 239--246.Google ScholarGoogle ScholarCross RefCross Ref
  30. David Ke Hong, Qi Alfred Chen, and Z. Morley Mao. 2017. An Initial Investigation of Protocol Customization. In Proceedings of the 2017 Workshop on Forming an Ecosystem Around Software Transformation.Google ScholarGoogle Scholar
  31. David Ke Hong, John Kloosterman, Yuqi Jin, Yulong Cao, Qi Alfred Chen, Scott Mahlke, and Z Morley Mao. 2020. AVGuardian: Detecting and Mitigating Publish-Subscribe Overprivilege for Autonomous Vehicle Systems. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P). 445--459.Google ScholarGoogle Scholar
  32. Ke Hong. 2019. Performance, Security, and Safety Requirements Testing for Smart Systems Through Systematic Software Analysis. Ph.D. Dissertation. University of Michigan, Ann Arbor, MI.Google ScholarGoogle Scholar
  33. Eunkyoung Jee, Donghwan Shin, Sungdeok Cha, Jang-Soo Lee, and Doo-Hwan Bae. 2014. Automated test case generation for FBD programs implementing reactor protection system software.Software Testing, Verification andReliability24, 8 (2014), 608--628.Google ScholarGoogle Scholar
  34. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solverfor verifying deep neural networks. In International Conference on Computer Aided Verification. Springer, 97--117.Google ScholarGoogle ScholarCross RefCross Ref
  35. BaekGyu Kim, Akshay Jarandikar, Jonathan Shum, Shinichi Shiraishi, and Masahiro Yamaura. 2016. The SMT-based automatic road network generation in vehicle simulation environment. In2016 International Conference on Embedded Software (EMSOFT). IEEE, 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Philip Koopman and Michael Wagner. 2016. Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety(2016).Google ScholarGoogle Scholar
  37. Philip Koopman and Michael Wagner. 2017. Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intelligent Transportation Systems Magazine(2017).Google ScholarGoogle Scholar
  38. Chris Lattner and Vikram Adve. 2004. LLVM: A compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004. IEEE, 75--86.Google ScholarGoogle ScholarCross RefCross Ref
  39. John Leonard, Jonathan How, Seth Teller, Mitch Berger, Stefan Campbell, Gaston Fiore, Luke Fletcher, Emilio Frazzoli, Albert Huang, Sertac Karaman, Olivier Koch, Yoshiaki Kuwata, David Moore, Edwin Olson, Steve Peters, Justin Teo,Robert Truax, Matthew Walter, David Barrett, Alexander Epstein, Keoni Maheloni, Katy Moyer, Troy Jones, Ryan Buckley, Matthew Antone, Robert Galejs, Siddhartha Krishnamurthy, and Jonathan Williams. 2008. A Perception-Driven Autonomous Urban Vehicle. Journal of Field Robotic(2008).Google ScholarGoogle Scholar
  40. Guodong Li and Ganesh Gopalakrishnan. 2010. Scalable SMT-based verification of GPU kernel functions. In Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. 187--196.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Guanpeng Li, Yiran Li, Saurabh Jha, Timothy Tsai, Siva Hari, Michael Sullivan, Zbigniew T. Kalbarczyk, and Ravis-hankar K. Iyer. 2020. AV-FUZZER: Finding safety violations in autonomous driving systems. ISSRE'20, Proceedings of the IEEE International Conference on Software Reliability Engineering(Nov 2020).Google ScholarGoogle Scholar
  42. Nan Li, Dave W Oyler, Mengxuan Zhang, Yildiray Yildiz, Ilya Kolmanovsky, and Anouck R Girard. 2017. Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems. IEEE Transactions on control systems technology 26, 5 (2017), 1782--1797.Google ScholarGoogle ScholarCross RefCross Ref
  43. Satoshi Masuda, Hiroaki Nakamura, and Kohichi Kajitani. 2018. Rule-based searching for collision test cases of autonomous vehicles simulation. IET Intelligent Transport Systems 12, 9 (2018), 1088--1095.Google ScholarGoogle ScholarCross RefCross Ref
  44. M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, D. Dolgov, S. Ettinger, D. Haehnel, T. Hilden, G. Hoffmann, B.Huhnke, D. Johnston, S. Klumpp, D. Langer, A. Levandowski, J. Levinson, J. Marcil, D. Orenstein, J. Paefgen, I. Penny,A. Petrovskaya, M. Pflueger, G. Stanek, D. Stavens, A. Vogt, and S. Thrun. 2008. Junior: The Stanford Entry in the Urban Challenge. Journal of Field Robotics, Special Issue on the 2007 DARPA Urban Challenge, Part II(2008).Google ScholarGoogle Scholar
  45. Luke Nelson, Helgi Sigurbjarnarson, Kaiyuan Zhang, Dylan Johnson, James Bornholt, Emina Torlak, and Xi Wang.2017. Hyperkernel: Push-button verification of an OS kernel. In Proceedings of the 26th Symposium on Operating Systems Principles. 252--269.Google ScholarGoogle Scholar
  46. Andres Nötzli, Jehandad Khan, Andy Fingerhut, Clark Barrett, and Peter Athanas. 2018. P4pktgen: Automated testcase generation for P4 programs. In Proceedings of the Symposium on SDN Research. 1--7.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C Duchi. 2018. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Advances in Neural Information Processing Systems. 9827--9838.Google ScholarGoogle Scholar
  48. Anshul Paigwar, Eduard Baranov, Alessandro Renzaglia, Christian Laugier, and Axel Legay. 2020. Probabilistic Collision Risk Estimation for Autonomous Driving: Validation via Statistical Model Checking. In 31st IEEE Intelligent Vehicles Symposium.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Christian Pek, Peter Zahn, and Matthias Althoff. 2017. Verifying the safety of lane change maneuvers of self-driving vehicles based on formalized traffic rules. In 2017 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1477--1483.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Michael Pradel, Ciera Jaspan, Jonathan Aldrich, and Thomas R Gross. 2012. Statically checking API protocol conformance with mined multi-object specifications. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 925--935.Google ScholarGoogle ScholarCross RefCross Ref
  51. Albert Rizaldi, Fabian Immler, and Matthias Althoff. 2016. A formally verified checker of the safe distance traffic rulesfor autonomous vehicles. In NASA Formal Methods Symposium. Springer, 175--190.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Guodong Rong, Byung Hyun Shin, Hadi Tabatabaee, Qiang Lu, Steve Lemke, Martins Mozeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, et al.[n.d.]. LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving.([n. d.]).Google ScholarGoogle Scholar
  53. Tao B Schardl, Tyler Denniston, Damon Doucet, Bradley C Kuszmaul, I-Ting Angelina Lee, and Charles E Leiserson. 2017. The CSI framework for compiler-inserted program instrumentation. Proceedings of the ACM on Measurement and Analysis of Computing Systems 1, 2 (2017), 1--25.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. 2017. On a Formal Model of Safe and Scalable Self-driving Cars. CoRR(2017).Google ScholarGoogle Scholar
  55. Helgi Sigurbjarnarson, James Bornholt, Emina Torlak, and Xi Wang. 2016. Push-Button Verification of File Systems via Crash Refinement. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16).Google ScholarGoogle Scholar
  56. Yulei Sui and Jingling Xue. 2016. SVF: interprocedural static value-flow analysis in LLVM. In Proceedings of the 25th international conference on compiler construction. 265--266.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering. 303--314.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher Baker, Robert Bittner, M. N. Clark, John Dolan, Dave Duggins, Tugrul Galatali, Chris Geyer, Michele Gittleman, Sam Harbaugh, Martial Hebert, Thomas M. Howard, Sascha Kolski, Alonzo Kelly, Maxim Likhachev, Matt McNaughton, Nick Miller, Kevin Peterson, Brian Pilnick, Raj Rajkumar, Paul Rybski, Bryan Salesky, Young-Woo Seo, Sanjiv Singh, Jarrod Snider, Anthony Stentz, William "Red" Whittaker, Ziv Wolkowicki, Jason Ziglar, Hong Bae, Thomas Brown, Daniel Demitrish, Bakhtiar Litkouhi, Jim Nickolaou, Varsha Sadekar, Wende Zhang, Joshua Struble, Michael Taylor, Michael Darms, and Dave Ferguson. 2008. Autonomous Driving in Urban Environments: Boss and the Urban Challenge. Journal of Field Robotics(2008).Google ScholarGoogle Scholar
  59. Niki Vazou, Anish Tondwalkar, Vikraman Choudhury, Ryan G Scott, Ryan R Newton, Philip Wadler, and Ranjit Jhala.2017. Refinement reflection: complete verification with SMT. Proceedings of the ACM on Programming Languages 2, POPL (2017), 1--31.Google ScholarGoogle Scholar
  60. J. Wei, J. M. Snider, T. Gu, J. M. Dolan, and B. Litkouhi. 2014. A behavioral planning framework for autonomous driving. In 2014 IEEE Intelligent Vehicles Symposium Proceedings.Google ScholarGoogle Scholar
  61. Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 132--142.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Marc René Zofka, Sebastian Klemm, Florian Kuhnt, Thomas Schamm, and J Marius Zöllner. 2016. Testing and validating high level components for automated driving: simulation framework for traffic scenarios. In 2016 IEEE Intelligent Vehicles Symposium (IV). IEEE, 144--150.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A Systematic Framework to Identify Violations of Scenario-dependent Driving Rules in Autonomous Vehicle Software

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!