skip to main content
research-article

Modeling and Analysis of Explanation for Secure Industrial Control Systems

Published:15 December 2022Publication History
Skip Abstract Section

Abstract

Many self-adaptive systems benefit from human involvement and oversight, where a human operator can provide expertise not available to the system and detect problems that the system is unaware of. One way of achieving this synergy is by placing the human operator on the loop—i.e., providing supervisory oversight and intervening in the case of questionable adaptation decisions. To make such interaction effective, an explanation can play an important role in allowing the human operator to understand why the system is making certain decisions and improve the level of knowledge that the operator has about the system. This, in turn, may improve the operator’s capability to intervene and, if necessary, override the decisions being made by the system. However, explanations may incur costs, in terms of delay in actions and the possibility that a human may make a bad judgment. Hence, it is not always obvious whether an explanation will improve overall utility and, if so, then what kind of explanation should be provided to the operator. In this work, we define a formal framework for reasoning about explanations of adaptive system behaviors and the conditions under which they are warranted. Specifically, we characterize explanations in terms of explanation content, effect, and cost. We then present a dynamic system adaptation approach that leverages a probabilistic reasoning technique to determine when an explanation should be used to improve overall system utility. We evaluate our explanation framework in the context of a realistic industrial control system with adaptive behaviors.

REFERENCES

  1. [1] Cheng Betty H. C.al. et2009. Software engineering for self-adaptive systems: A research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 126.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Lemos Rogério deal. et2010. Software engineering for self-adaptive systems: A second research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 132.Google ScholarGoogle Scholar
  3. [3] Sukkerd Roykrong, Garlan David, and Simmons Reid G.. 2015. Task planning of cyber-human systems. In 13th International Conference on Software Engineering and Formal Methods. 293309.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Cámara Javier, Moreno Gabriel A., and Garlan David. 2015. Reasoning about human participation in self-adaptive systems. In 10th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 146156.Google ScholarGoogle Scholar
  5. [5] Lloyd Eric, Huang Shihong, and Tognoli Emmanuelle. 2017. Improving human-in-the-loop adaptive systems using brain-computer interaction. In 12th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 163174.Google ScholarGoogle Scholar
  6. [6] Fischer Joel E., Greenhalgh Chris, Jiang Wenchao, Ramchurn Sarvapali D., Wu Feng, and Rodden Tom. 2021. In-the-loop or on-the-loop? Interactional arrangements to support team coordination with a planning agent. Concurr. Computat.: Pract. Exper. 33, 8 (2021), e4082.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Department of Homeland Security. 2022. ICS-CERT Advisories. Retrieved from https://ics-cert.us-cert.gov/advisories.Google ScholarGoogle Scholar
  8. [8] Biran Or and Cotton Courtenay. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI), Vol. 8. 813.Google ScholarGoogle Scholar
  9. [9] Nomura Tatsuya and Kawakami Kayoko. 2011. Relationships between robot’s self-disclosures and human’s anxiety toward robots. In IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology. IEEE Computer Society, 6669.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Li Nianyu, Adepu Sridhar, Kang Eunsuk, and Garlan David. Explanations for human-on-the-loop: A probabilistic model checking approach. In 15th International Symposium on Software Engineering for Adaptive and Self-managing Systems (SEAMS).Google ScholarGoogle Scholar
  11. [11] Kwiatkowska Marta, Norman Gethin, and Parker David. 2018. Probabilistic Model Checking: Advances and Applications. Springer International Publishing, Cham, 73121.Google ScholarGoogle Scholar
  12. [12] Design Singapore University of Technology and. 2022. Secure Water Treatment (SWaT). Retrieved from https://itrust.sutd.edu.sg/itrust-labs-home/itrust-labs_swat/.Google ScholarGoogle Scholar
  13. [13] Miller Tim. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell. 267 (2019), 138.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Chandrasekaran B., Tanner Michael C., and Josephson John R.. 1989. Explaining control strategies in problem solving. IEEE Expert 4, 1 (1989), 924.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Fennel T. R. and Johannes James D.. 1990. An architecture for rule based system explanation. In Fifth Conference on Artificial Intelligence for Space Applications, vol. 3073. NASA, 113.Google ScholarGoogle Scholar
  16. [16] Paris Cécile L.. 1991. Generation and explanation: Building an explanation facility for the explainable expert systems framework. In Natural Language Generation in Artificial Intelligence and Computational Linguistics. Springer, 4982.Google ScholarGoogle Scholar
  17. [17] Hayes Bradley and Shah Julie A.. 2017. Improving robot controller transparency through autonomous policy explanation. In 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 303312.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Wohleber Ryan W., Stowers Kimberly, Chen Jessie Y. C., and Barnes Michael. 2017. Effects of agent transparency and communication framing on human-agent teaming. In IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 34273432.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Molineaux Matthew, Dannenhauer Dustin, and Aha David W.. 2018. Towards explainable NPCs: A relational exploration learning agent. In Workshops at the 32nd AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  20. [20] Harbers Maaike, Bosch Karel Van Den, and Meyer John-Jules. 2009. A methodology for developing self-explaining agents for virtual training. In International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems. Springer, 168182.Google ScholarGoogle Scholar
  21. [21] Kaptein Frank, Broekens Joost, Hindriks Koen, and Neerincx Mark. 2017. The role of emotion in self-explanations by cognitive agents. In 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 8893.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Kulesza Todd, Stumpf Simone, Burnett Margaret, Yang Sherry, Kwan Irwin, and Wong Weng-Keen. 2013. Too much, too little, or just right? Ways explanations impact end users’ mental models. In IEEE Symposium on Visual Languages and Human Centric Computing. IEEE, 310.Google ScholarGoogle Scholar
  23. [23] Carey. Peter2018. Data Protection: A Practical Guide to UK and EU Law.Oxford University Press, Inc.Google ScholarGoogle Scholar
  24. [24] Neerincx Mark A., Waa Jasper van der, Kaptein Frank, and Diggelen Jurriaan van. 2018. Using perceptual and cognitive explanations for enhanced human-agent team performance. In 15th International Conference on Engineering Psychology and Cognitive Ergonomics. 204214.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Sukkerd Roykrong. 2018. Improving Transparency and Understandability of Multi-objective Probabilistic Planning. Thesis Proposal. School of Computer Science Institute for Software Research Software Engineering, Carnegie Mellon University.Google ScholarGoogle Scholar
  26. [26] Hellström Thomas and Bensch Suna. 2018. Understandable robotswhat, why, and how. Paladyn, J. Behav. Robot. 9, 1 (2018), 110123.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Bethel Cindy L.. 2009. Robots without faces: Non-verbal social human-robot Interaction. Doctoral dissertation, dissertation/Ph. D.’s thesis. University of South Florida.Google ScholarGoogle Scholar
  28. [28] Broekens Joost, Harbers Maaike, Hindriks Koen, Bosch Karel Van Den, Jonker Catholijn, and Meyer John-Jules. 2010. Do you get it? User-evaluated explainable BDI agents. In German Conference on Multiagent System Technologies. Springer, 2839.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Lim Brian Y., Dey Anind K., and Avrahami Daniel. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In 27th International Conference on Human Factors in Computing Systems. 21192128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Chakraborti Tathagata, Sreedharan Sarath, Zhang Yu, and Kambhampati Subbarao. 2017. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In 26th International Joint Conference on Artificial Intelligence. 156163.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Elizalde Francisco, Sucar L. Enrique, Luque Manuel, Diez J., and Reyes Alberto. 2008. Policy explanation in factored Markov decision processes. In European Workshop on Probabilistic Graphical Models (PGM). 97104.Google ScholarGoogle Scholar
  32. [32] Khan Omar Zia, Poupart Pascal, and Black James P.. 2009. Minimal sufficient explanations for factored Markov decision processes. In 19th International Conference on Automated Planning and Scheduling.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Sukkerd Roykrong, Simmons Reid G., and Garlan David. 2018. Towards explainable multi-objective probabilistic planning. In 4th International Workshop on Software Engineering for Smart Cyber-physical Systems. 1925.Google ScholarGoogle Scholar
  34. [34] Cárdenas Alvaro A., Amin Saurabh, Lin Zong-Syun, Huang Yu-Lun, Huang Chi-Yen, and Sastry Shankar. 2011. Attacks against process control systems: Risk assessment, detection, and response. In 6th ACM Symposium on Information, Computer and Communications Security. 355366.Google ScholarGoogle Scholar
  35. [35] Adepu Sridhar and Mathur Aditya. 2018. Assessing the effectiveness of attack detection at a hackfest on industrial control systems. IEEE Trans. Sustain. Comput. 6, 2 (2018), 231–244.Google ScholarGoogle Scholar
  36. [36] Liu Yao, Ning Peng, and Reiter Michael K.. 2011. False data injection attacks against state estimation in electric power grids. ACM Trans. Inf. Syst. Secur. 14, 1 (2011), 133.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Trcka Nikola, Moulin Mark, Bopardikar Shaunak, and Speranzon Alberto. 2014. A formal verification approach to revealing stealth attacks on networked control systems. In 3rd International Conference on High Confidence Networked Systems. 6776.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] Adepu Sridhar, Kandasamy Nandha Kumar, and Mathur Aditya. 2018. EPIC: An electric power testbed for research and training in cyber physical systems security. In Computer Security. Springer, 3752.Google ScholarGoogle Scholar
  39. [39] Lun Yuriy Zacchia, D’Innocenzo Alessandro, Smarra Francesco, Malavolta Ivano, and Benedetto Maria Domenica Di. 2019. State of the art of cyber-physical systems security: An automatic control perspective. J. Syst. Softw. 149 (2019), 174216.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Sabaliauskaite Giedre and Adepu Sridhar. 2017. Integrating six-step model with information flow diagrams for comprehensive analysis of cyber-physical system safety and security. In IEEE 18th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 4148.Google ScholarGoogle Scholar
  41. [41] Adepu Sridhar, Kang Eunsuk, and Mathur Aditya P.. 2019. Challenges in secure engineering of critical infrastructure systems. In 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). IEEE, 6164.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Demertzis Konstantinos, Iliadis Lazaros S., and Anezakis Vardis-Dimitrios. 2018. An innovative soft computing system for smart energy grids cybersecurity. Adv. Build. Energ. Res. 12, 1 (2018), 324.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Khan Muhammad Taimoor, Serpanos Dimitrios, and Shrobe Howard. 2017. ARMET: Behavior-based secure and resilient industrial control systems. Proc. IEEE 106, 1 (2017), 129143.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Zhang Xin, Cai Xiaobo, Wang Chaogang, Han Ke, and Zhang Shujuan. 2019. A dynamic security control architecture for industrial cyber-physical system. In IEEE International Conference on Industrial Internet (ICII). IEEE, 148151.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Greenyer Joel, Lochau Malte, and Vogel Thomas. 2019. Explainable software for cyber-physical systems (ES4CPS): Report from the GI Dagstuhl Seminar 19023, January 6-11 2019, Schloss Dagstuhl. arXiv preprint arXiv:1904.11851.Google ScholarGoogle Scholar
  46. [46] Tesla’s Trouble with Semi Trucks & Another Shakeup of the Autopilot Team – Is There a Connection? Retrieved 10 Oct., 2021 from https://cleantechnica.com/2019/05/21/teslas-trouble-with-trucks-and-another-shakeup-of-the-autopilot-team-is-there-a-connection/.Google ScholarGoogle Scholar
  47. [47] Deng Yao, Zhang Tiehua, Lou Guannan, Zheng Xi, Jin Jiong, and Han Qing-Long. 2021. Deep learning-based autonomous driving systems: A survey of attacks and defenses. IEEE Trans. Industr. Inf. 17, 12 (2021), 78977912.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Combéfis Sébastien, Giannakopoulou Dimitra, Pecheur Charles, and Feary Michael. 2011. Learning system abstractions for human operators. In International Workshop on Machine Learning Technologies in Software Engineering. 310.Google ScholarGoogle Scholar
  49. [49] Palmer E.. 1996. Oops, it didn’t arm.—A case study of two automation surprises. In 8th International Symposium on Aviation Psychology. 227232.Google ScholarGoogle Scholar
  50. [50] Moreno Gabriel A., Cámara Javier, Garlan David, and Schmerl Bradley R.. 2015. Proactive self-adaptation under uncertainty: A probabilistic model checking approach. In 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE. 112.Google ScholarGoogle Scholar
  51. [51] Bianco Andrea and Alfaro Luca de. 1995. Model checking of probabilistic and nondeterministic systems. In Foundations of Software Technology and Theoretical Computer Science, Thiagarajan P. S. (Ed.). Springer Berlin.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Kwiatkowska Marta Z., Norman Gethin, and Parker David. 2011. PRISM 4.0: Verification of probabilistic real-time systems. In 23rd International Conference on Computer Aided Verification. 585591.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Puterman Martin L.. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Norman Gethin, Parker David, and Sproston Jeremy. 2013. Model checking for probabilistic timed automata. Form. Meth. Syst. Des. 43, 2 (2013), 164190.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. [55] Soetaert Karline. 2018. plot3D: Tools for Plotting 3-D and 2-D Data. Retrieved from https://cran.r-project.org/web/packages/plot3D/vignettes/plot3D.pdf.Google ScholarGoogle Scholar
  56. [56] Mathur A. P. and Tippenhauer N. O.. 2016. SWaT: A water treatment testbed for research and training on ICS security. In International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater). 3136.Google ScholarGoogle Scholar
  57. [57] Kang Eunsuk, Adepu Sridhar, Jackson Daniel, and Mathur Aditya P.. 2016. Model-based security analysis of a water treatment system. In IEEE/ACM 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS). IEEE, 2228.Google ScholarGoogle Scholar
  58. [58] Adepu Sridhar and Mathur Aditya. 2016. An investigation into the response of a water treatment system to cyber attacks. In IEEE 17th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 141148.Google ScholarGoogle Scholar
  59. [59] Maw Aung, Adepu Sridhar, and Mathur Aditya. 2019. ICS-BlockOpS: Blockchain for operational data security in industrial control system. Pervas. Mob. Comput. 59 (2019), 101048.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. [60] Chen Yuqi, Poskitt Christopher M., Sun Jun, Adepu Sridhar, and Zhang Fan. 2019. Learning-guided network fuzzing for testing cyber-physical system defences. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 962973.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. [61] Adepu Sridhar, Brasser Ferdinand, Garcia Luis, Rodler Michael, Davi Lucas, Sadeghi Ahmad-Reza, and Zonouz Saman. 2020. Control behavior integrity for distributed cyber-physical systems. In ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 3040.Google ScholarGoogle ScholarCross RefCross Ref
  62. [62] Das Tanmoy Kanti, Adepu Sridhar, and Zhou Jianying. 2020. Anomaly detection in industrial control systems using logical analysis of data. Comput. Secur. 96 (2020), 101935.Google ScholarGoogle ScholarCross RefCross Ref
  63. [63] Umer Muhammad Azmi, Mathur Aditya, Junejo Khurum Nazir, and Adepu Sridhar. 2017. Integrating design and data centric approaches to generate invariants for distributed attack detection. In Workshop on Cyber-Physical Systems Security and PrivaCy. 131136.Google ScholarGoogle Scholar
  64. [64] Mahdavi-Hezavehi Sara, Durelli Vinicius H. S., Weyns Danny, and Avgeriou Paris. 2017. A systematic literature review on methods that handle multiple quality attributes in architecture-based self-adaptive systems. Inf. Softw. Technol. 90 (2017), 126.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Modeling and Analysis of Explanation for Secure Industrial Control Systems

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Autonomous and Adaptive Systems
        ACM Transactions on Autonomous and Adaptive Systems  Volume 17, Issue 3-4
        December 2022
        49 pages
        ISSN:1556-4665
        EISSN:1556-4703
        DOI:10.1145/3561963
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 15 December 2022
        • Online AM: 17 August 2022
        • Accepted: 29 June 2022
        • Revised: 3 May 2022
        • Received: 11 February 2021
        Published in taas Volume 17, Issue 3-4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed
      • Article Metrics

        • Downloads (Last 12 months)196
        • Downloads (Last 6 weeks)6

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!