Abstract
Many self-adaptive systems benefit from human involvement and oversight, where a human operator can provide expertise not available to the system and detect problems that the system is unaware of. One way of achieving this synergy is by placing the human operator on the loop—i.e., providing supervisory oversight and intervening in the case of questionable adaptation decisions. To make such interaction effective, an explanation can play an important role in allowing the human operator to understand why the system is making certain decisions and improve the level of knowledge that the operator has about the system. This, in turn, may improve the operator’s capability to intervene and, if necessary, override the decisions being made by the system. However, explanations may incur costs, in terms of delay in actions and the possibility that a human may make a bad judgment. Hence, it is not always obvious whether an explanation will improve overall utility and, if so, then what kind of explanation should be provided to the operator. In this work, we define a formal framework for reasoning about explanations of adaptive system behaviors and the conditions under which they are warranted. Specifically, we characterize explanations in terms of explanation content, effect, and cost. We then present a dynamic system adaptation approach that leverages a probabilistic reasoning technique to determine when an explanation should be used to improve overall system utility. We evaluate our explanation framework in the context of a realistic industrial control system with adaptive behaviors.
- [1] 2009. Software engineering for self-adaptive systems: A research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 1–26.Google Scholar
Digital Library
- [2] 2010. Software engineering for self-adaptive systems: A second research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 1–32.Google Scholar
- [3] . 2015. Task planning of cyber-human systems. In 13th International Conference on Software Engineering and Formal Methods. 293–309.Google Scholar
Cross Ref
- [4] . 2015. Reasoning about human participation in self-adaptive systems. In 10th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 146–156.Google Scholar
- [5] . 2017. Improving human-in-the-loop adaptive systems using brain-computer interaction. In 12th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 163–174.Google Scholar
- [6] . 2021. In-the-loop or on-the-loop? Interactional arrangements to support team coordination with a planning agent. Concurr. Computat.: Pract. Exper. 33, 8 (2021), e4082.Google Scholar
Cross Ref
- [7] Department of Homeland Security. 2022. ICS-CERT Advisories. Retrieved from https://ics-cert.us-cert.gov/advisories.Google Scholar
- [8] . 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI), Vol. 8. 8–13.Google Scholar
- [9] . 2011. Relationships between robot’s self-disclosures and human’s anxiety toward robots. In IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology. IEEE Computer Society, 66–69.Google Scholar
Digital Library
- [10] . Explanations for human-on-the-loop: A probabilistic model checking approach. In 15th International Symposium on Software Engineering for Adaptive and Self-managing Systems (SEAMS).Google Scholar
- [11] . 2018. Probabilistic Model Checking: Advances and Applications. Springer International Publishing, Cham, 73–121.Google Scholar
- [12] . 2022. Secure Water Treatment (SWaT). Retrieved from https://itrust.sutd.edu.sg/itrust-labs-home/itrust-labs_swat/.Google Scholar
- [13] . 2019. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell. 267 (2019), 1–38.Google Scholar
Cross Ref
- [14] . 1989. Explaining control strategies in problem solving. IEEE Expert 4, 1 (1989), 9–24.Google Scholar
Digital Library
- [15] . 1990. An architecture for rule based system explanation. In Fifth Conference on Artificial Intelligence for Space Applications, vol. 3073. NASA, 113.Google Scholar
- [16] . 1991. Generation and explanation: Building an explanation facility for the explainable expert systems framework. In Natural Language Generation in Artificial Intelligence and Computational Linguistics. Springer, 49–82.Google Scholar
- [17] . 2017. Improving robot controller transparency through autonomous policy explanation. In 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 303–312.Google Scholar
Digital Library
- [18] . 2017. Effects of agent transparency and communication framing on human-agent teaming. In IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3427–3432.Google Scholar
Digital Library
- [19] . 2018. Towards explainable NPCs: A relational exploration learning agent. In Workshops at the 32nd AAAI Conference on Artificial Intelligence.Google Scholar
- [20] . 2009. A methodology for developing self-explaining agents for virtual training. In International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems. Springer, 168–182.Google Scholar
- [21] . 2017. The role of emotion in self-explanations by cognitive agents. In 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 88–93.Google Scholar
Cross Ref
- [22] . 2013. Too much, too little, or just right? Ways explanations impact end users’ mental models. In IEEE Symposium on Visual Languages and Human Centric Computing. IEEE, 3–10.Google Scholar
- [23] 2018. Data Protection: A Practical Guide to UK and EU Law.Oxford University Press, Inc.Google Scholar
- [24] . 2018. Using perceptual and cognitive explanations for enhanced human-agent team performance. In 15th International Conference on Engineering Psychology and Cognitive Ergonomics. 204–214.Google Scholar
Digital Library
- [25] . 2018. Improving Transparency and Understandability of Multi-objective Probabilistic Planning. Thesis Proposal. School of Computer Science Institute for Software Research Software Engineering, Carnegie Mellon University.Google Scholar
- [26] . 2018. Understandable robotswhat, why, and how. Paladyn, J. Behav. Robot. 9, 1 (2018), 110–123.Google Scholar
Cross Ref
- [27] . 2009. Robots without faces: Non-verbal social human-robot Interaction. Doctoral dissertation, dissertation/Ph. D.’s thesis. University of South Florida.Google Scholar
- [28] . 2010. Do you get it? User-evaluated explainable BDI agents. In German Conference on Multiagent System Technologies. Springer, 28–39.Google Scholar
Cross Ref
- [29] . 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In 27th International Conference on Human Factors in Computing Systems. 2119–2128.Google Scholar
Digital Library
- [30] . 2017. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In 26th International Joint Conference on Artificial Intelligence. 156–163.Google Scholar
Cross Ref
- [31] . 2008. Policy explanation in factored Markov decision processes. In European Workshop on Probabilistic Graphical Models (PGM). 97–104.Google Scholar
- [32] . 2009. Minimal sufficient explanations for factored Markov decision processes. In 19th International Conference on Automated Planning and Scheduling.Google Scholar
Cross Ref
- [33] . 2018. Towards explainable multi-objective probabilistic planning. In 4th International Workshop on Software Engineering for Smart Cyber-physical Systems. 19–25.Google Scholar
- [34] . 2011. Attacks against process control systems: Risk assessment, detection, and response. In 6th ACM Symposium on Information, Computer and Communications Security. 355–366.Google Scholar
- [35] . 2018. Assessing the effectiveness of attack detection at a hackfest on industrial control systems. IEEE Trans. Sustain. Comput. 6, 2 (2018), 231–244.Google Scholar
- [36] . 2011. False data injection attacks against state estimation in electric power grids. ACM Trans. Inf. Syst. Secur. 14, 1 (2011), 1–33.Google Scholar
Digital Library
- [37] . 2014. A formal verification approach to revealing stealth attacks on networked control systems. In 3rd International Conference on High Confidence Networked Systems. 67–76.Google Scholar
Digital Library
- [38] . 2018. EPIC: An electric power testbed for research and training in cyber physical systems security. In Computer Security. Springer, 37–52.Google Scholar
- [39] . 2019. State of the art of cyber-physical systems security: An automatic control perspective. J. Syst. Softw. 149 (2019), 174–216.Google Scholar
Cross Ref
- [40] . 2017. Integrating six-step model with information flow diagrams for comprehensive analysis of cyber-physical system safety and security. In IEEE 18th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 41–48.Google Scholar
- [41] . 2019. Challenges in secure engineering of critical infrastructure systems. In 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). IEEE, 61–64.Google Scholar
Cross Ref
- [42] . 2018. An innovative soft computing system for smart energy grids cybersecurity. Adv. Build. Energ. Res. 12, 1 (2018), 3–24.Google Scholar
Cross Ref
- [43] . 2017. ARMET: Behavior-based secure and resilient industrial control systems. Proc. IEEE 106, 1 (2017), 129–143.Google Scholar
Cross Ref
- [44] . 2019. A dynamic security control architecture for industrial cyber-physical system. In IEEE International Conference on Industrial Internet (ICII). IEEE, 148–151.Google Scholar
Cross Ref
- [45] . 2019. Explainable software for cyber-physical systems (ES4CPS): Report from the GI Dagstuhl Seminar 19023, January 6-11 2019, Schloss Dagstuhl. arXiv preprint arXiv:1904.11851.Google Scholar
- [46] Tesla’s Trouble with Semi Trucks & Another Shakeup of the Autopilot Team – Is There a Connection? Retrieved 10 Oct., 2021 from https://cleantechnica.com/2019/05/21/teslas-trouble-with-trucks-and-another-shakeup-of-the-autopilot-team-is-there-a-connection/.Google Scholar
- [47] . 2021. Deep learning-based autonomous driving systems: A survey of attacks and defenses. IEEE Trans. Industr. Inf. 17, 12 (2021), 7897–7912.Google Scholar
Cross Ref
- [48] . 2011. Learning system abstractions for human operators. In International Workshop on Machine Learning Technologies in Software Engineering. 3–10.Google Scholar
- [49] . 1996. Oops, it didn’t arm.—A case study of two automation surprises. In 8th International Symposium on Aviation Psychology. 227–232.Google Scholar
- [50] . 2015. Proactive self-adaptation under uncertainty: A probabilistic model checking approach. In 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE. 1–12.Google Scholar
- [51] . 1995. Model checking of probabilistic and nondeterministic systems. In Foundations of Software Technology and Theoretical Computer Science, (Ed.). Springer Berlin.Google Scholar
Digital Library
- [52] . 2011. PRISM 4.0: Verification of probabilistic real-time systems. In 23rd International Conference on Computer Aided Verification. 585–591.Google Scholar
Digital Library
- [53] . 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.Google Scholar
Digital Library
- [54] . 2013. Model checking for probabilistic timed automata. Form. Meth. Syst. Des. 43, 2 (2013), 164–190.Google Scholar
Digital Library
- [55] . 2018. plot3D: Tools for Plotting 3-D and 2-D Data. Retrieved from https://cran.r-project.org/web/packages/plot3D/vignettes/plot3D.pdf.Google Scholar
- [56] . 2016. SWaT: A water treatment testbed for research and training on ICS security. In International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater). 31–36.Google Scholar
- [57] . 2016. Model-based security analysis of a water treatment system. In IEEE/ACM 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS). IEEE, 22–28.Google Scholar
- [58] . 2016. An investigation into the response of a water treatment system to cyber attacks. In IEEE 17th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 141–148.Google Scholar
- [59] . 2019. ICS-BlockOpS: Blockchain for operational data security in industrial control system. Pervas. Mob. Comput. 59 (2019), 101048.Google Scholar
Digital Library
- [60] . 2019. Learning-guided network fuzzing for testing cyber-physical system defences. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 962–973.Google Scholar
Digital Library
- [61] . 2020. Control behavior integrity for distributed cyber-physical systems. In ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 30–40.Google Scholar
Cross Ref
- [62] . 2020. Anomaly detection in industrial control systems using logical analysis of data. Comput. Secur. 96 (2020), 101935.Google Scholar
Cross Ref
- [63] . 2017. Integrating design and data centric approaches to generate invariants for distributed attack detection. In Workshop on Cyber-Physical Systems Security and PrivaCy. 131–136.Google Scholar
- [64] . 2017. A systematic literature review on methods that handle multiple quality attributes in architecture-based self-adaptive systems. Inf. Softw. Technol. 90 (2017), 1–26.Google Scholar
Cross Ref
Index Terms
Modeling and Analysis of Explanation for Secure Industrial Control Systems
Recommendations
Linking granger causality and the pearl causal model with settable systems
NIPSMINI'09: Proceedings of the 12th International Conference on Neural Information Processing Systems (NIPS)Mini-Symposium on Causality in Time SeriesThe causal notions embodied in the concept of Granger causality have been argued to belong to a different category than those of Judea Pearl's Causal Model, and so far their relation has remained obscure. Here, we demonstrate that these concepts are in ...
Complexity results for explanations in the structural-model approach
We analyze the computational complexity of Halpern and Pearl's (causal) explanations in the structural-model approach, which are based on their notions of weak and actual cause. In particular, we give a precise picture of the complexity of deciding ...






Comments