Abstract
Model-finders, such as SAT/SMT-solvers and Alloy, are used widely both directly and embedded in domain-specific tools. They support both conventional verification and, unlike other verification tools, property-free exploration. To do this effectively, they must produce output that helps users with these tasks. Unfortunately, the output of model-finders has seen relatively little rigorous human-factors study.
Conventionally, these tools tend to show one satisfying instance at a time. Drawing inspiration from the cognitive science literature, we investigate two aspects of model-finder output: how many instances to show at once, and whether all instances must actually satisfy the input constraints. Using both controlled studies and open-ended talk-alouds, we show that there is benefit to showing negative instances in certain settings; the impact of multiple instances is less clear. Our work is a first step in a theoretically grounded approach to understanding how users engage cognitively with model-finder output, and how those tools might better support users in doing so.
- Devdatta Akhawe, Adam Barth, Peifung E. Lam, John Mitchell, and Dawn Song. 2010. Towards a Formal Foundation of Web Security. In IEEE Computer Security Foundations Symposium. https://doi.org/10.1109/CSF.2010.27 Google Scholar
Digital Library
- Louis Alfieri, Timothy J. Nokes-Malach, and Christian D. Schunn. 2013. Learning Through Case Comparisons: A Meta-Analytic Review. Educational Psychologist, 48, 2 (2013), https://doi.org/10.1080/00461520.2013.775712 Google Scholar
Cross Ref
- R. K. Atkinson, S. J. Derry, A. Renkl, and D. Wortham. 2000. Learning from examples: instructional principles from the worked examples research. Review of Educational Research, 70, 2 (2000), https://doi.org/10.3102/00346543070002181 Google Scholar
Cross Ref
- John Backes, Sam Bayless, Byron Cook, Catherine Dodge, Andrew Gacek, Alan J. Hu, Temesghen Kahsai, Bill Kocik, Evgenii Kotelnikov, Jure Kukovec, Sean McLaughlin, Jason Reed, Neha Rungta, John Sizemore, Mark A. Stalzer, Preethi Srinivasan, Pavle Subotic, Carsten Varming, and Blake Whaley. 2019. Reachability Analysis for AWS-Based Networks. In Computer Aided Verification - 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part II, Isil Dillig and Serdar Tasiran (Eds.) (Lecture Notes in Computer Science, Vol. 11562). Springer, 231–241. https://doi.org/10.1007/978-3-030-25543-5_14 Google Scholar
Cross Ref
- John Backes, Pauline Bolignano, Byron Cook, Catherine Dodge, Andrew Gacek, Kasper Sœ Luckow, Neha Rungta, Oksana Tkachuk, and Carsten Varming. 2018. Semantic-based Automated Reasoning for AWS Access Policies using SMT. In Formal Methods in Computer-Aided Design. IEEE, 1–9. https://doi.org/10.23919/FMCAD.2018.8602994 Google Scholar
Cross Ref
- Clark Barrett, Christopher L. Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. 2011. CVC4. In International Conference on Computer Aided Verification. Springer Berlin Heidelberg, Berlin, Heidelberg. 171–177. isbn:978-3-642-22110-1 https://doi.org/10.1007/978-3-642-22110-1_14 Google Scholar
Cross Ref
- Irving Biederman and Margaret M. Shiffrar. 1987. Sexing Day-Old Chicks: A Case Study and Expert Systems Analysis of a Difficult Perceptual-Learning Task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 4 (1987), 640–645. https://doi.org/10.1037/0278-7393.13.4.640 Google Scholar
Cross Ref
- R. Brown and C. Hanlon. 1970. Derivational complexity and order of acquisition on child speech. In Cognition and the development of language, J. Hayes (Ed.). Wiley. isbn:978-0471364733Google Scholar
- François Chollet. 2019. On the Measure of Intelligence. arxiv:1911.01547.Google Scholar
- Robert Clarisó and Jordi Cabot. 2020. Diverse Scenario Exploration in Model Finders Using Graph Kernels and Clustering. In Rigorous State Based Methods. Springer International Publishing, Cham. 27–43. isbn:978-3-030-48077-6 https://doi.org/10.1007/978-3-030-48077-6_3 Google Scholar
Digital Library
- Edmund M. Clarke, E. Allen Emerson, and Joseph Sifakis. 2009. Model Checking: Algorithmic Verification and Debugging. Commun. ACM, 52, 11 (2009), Nov., 74–84. issn:0001-0782 https://doi.org/10.1145/1592761.1592781 Google Scholar
Digital Library
- Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 1 (1960), 37–46. https://doi.org/10.1177/001316446002000104 Google Scholar
Cross Ref
- Leda Cosmides and John Tooby. 1992. Cognitive Adaptions for Social Exchange. In The Adapted Mind: Evolutionary Psychology and the Generation of Culture, Leda Cosmides, John Tooby, and Jerome H. Barkow (Eds.). Oxford University Press. isbn:978-0195101072Google Scholar
- Alcino Cunha, Nuno Macedo, and Tiago Guimarães. 2014. Target Oriented Relational Model Finding. In International Conference on Fundamental Approaches to Software Engineering. 17–31. https://doi.org/10.1007/978-3-642-54804-8_2 Google Scholar
Digital Library
- Natasha Danas, Tim Nelson, Lane Harrison, Shriram Krishnamurthi, and Daniel J. Dougherty. 2017. User Studies of Principled Model Finder Output. In Software Engineering and Formal Methods. https://doi.org/10.1007/978-3-319-66197-1_11 Google Scholar
Cross Ref
- Leonardo de Moura and N. Bjørner. 2008. Z3: An efficient SMT solver. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems (Lecture Notes in Computer Science, Vol. 4963). Springer, 337. https://doi.org/10.1007/978-3-540-78800-3_24 Google Scholar
Cross Ref
- Charles W. Dunnett. 1955. A Multiple Comparison Procedure for Comparing Several Treatments with a Control. J. Amer. Statist. Assoc., 50, 272 (1955), 1096–1121. https://doi.org/10.1080/01621459.1955.10501294 arxiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1955.10501294. Google Scholar
Cross Ref
- Tristan Dyer and John Baugh. 2021. Sterling: A web-based visualizer for relational modeling languages. In Rigorous State Based Methods. ABZ 2021. Springer, Cham. 99–104. https://doi.org/10.1007/978-3-030-77543-8_7 Lecture Notes in Computer Science 12709 Google Scholar
Digital Library
- Tristan Dyer, Tim Nelson, Kathi Fisler, and Shriram Krishnamurthi. 2022. Applying Cognitive Principles to Model-Finding Output: The Positive Value of Negative Information (artifact). https://doi.org/10.5281/zenodo.6370152 Google Scholar
Digital Library
- Ari Fogel, Stanley Fung, Luis Pedrosa, Meg Walraed-Sullivan, Ramesh Govindan, Ratul Mahajan, and Todd Millstein. 2015. A General Approach to Network Configuration Analysis. In Networked Systems Design and Implementation. 469–483. isbn:978-1-931971-218 https://doi.org/10.5555/2789770.2789803Google Scholar
- Eleanor J. Gibson. 1969. Principles of Perceptual Learning and Development. Appleton-Century-Crofts. isbn:9780390361455Google Scholar
- James J. Gibson and Eleanor J. Gibson. 1955. Perceptual Learning: Differentiation or Enrichment? Psychological Review, 62, 1 (1955), 32–41. https://doi.org/10.1037/h0048826 Google Scholar
Cross Ref
- M. L. Gick and K. Paterson. 1992. Do contrasting examples facilitate schema acquisition and analogical transfer? Canadian Journal of Psychology, 46, 4 (1992), https://doi.org/10.1037/h0084333 Google Scholar
Cross Ref
- Rubi Hammer, Tomer Hertz, Shaul Hochstein, and Daphna Weinshall. 2009. Category learning from equivalence constraints. Cognitive Processing, 10, 3 (2009), 211–232. isbn:1612-4790 https://doi.org/10.1007/s10339-008-0243-x Google Scholar
Cross Ref
- William L. Hays. 1994. Statistics (5th ed.). Harcourt Brace College Publishers, Fort Worth. isbn:0030744679Google Scholar
- Daniel Jackson. 2012. Software Abstractions: Logic, Language, and Analysis (2 ed.). MIT Press. isbn:0262017156 https://doi.org/10.5555/2141100Google Scholar
- Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studies with Mechanical Turk. In SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA. 453–456. isbn:9781605580111 https://doi.org/10.1145/1357054.1357127 Google Scholar
Digital Library
- Chinmay Kulkarni, Steven P. Dow, and Scott R. Klemmer. 2014. Early and Repeated Exposure to Examples Improves Creative Work. Springer International Publishing, 49–62. isbn:978-3-319-01303-9 https://doi.org/10.1007/978-3-319-01303-9_4 Google Scholar
Cross Ref
- Angelika Kullberg, Ulla Runesson Kempe, and Ference Marton. 2017. What is made possible to learn when using the variation theory of learning in teaching mathematics? ZDM: The International Journal on Mathematics Education, 49, 3 (2017), https://doi.org/10.1007/s11858-017-0858-4 Google Scholar
Cross Ref
- Nuno Macedo, Alcino Cunha, and Tiago Guimarães. 2015. Exploring Scenario Exploration. In International Conference on Fundamental Approaches to Software Engineering. https://doi.org/10.1007/978-3-662-46675-9_20 Google Scholar
Cross Ref
- Ferney A. Maldonado-Lopez, Jaime Chavarriaga, and Yezid Donoso. 2014. Detecting Network Policy Conflicts Using Alloy. In Conference on Abstract State Machines, Alloy, B, and Z. https://doi.org/10.1007/978-3-662-43652-3_31 Google Scholar
Digital Library
- Shahar Maoz, Jan Oliver Ringert, and Bernhard Rumpe. 2011. CD2Alloy: Class Diagrams Analysis Using Alloy Revisited. In Model Driven Engineering Languages and Systems. https://doi.org/10.1007/978-3-642-24485-8_44 Google Scholar
Cross Ref
- Gary F. Marcus. 1993. Negative evidence in language acquisition. Cognition, 46 (1993), https://doi.org/10.1016/0010-0277(93)90022-N Google Scholar
Cross Ref
- Ference Marton. 2014. Necessary Conditions of Learning. Routledge. isbn:978-0415739146Google Scholar
- Ference Marton and Ming Fai Pang. 2013. Meanings are acquired from experiencing differences against a background of sameness, rather than from experiencing sameness against a background of difference: Putting a conjecture to the test by embedding it in a pedagogical tool. Frontline Learning Research, 1 (2013), https://doi.org/10.14786/flr.v1i1.16 Google Scholar
Cross Ref
- Vajih Montaghami and Derek Rayside. 2017. Bordeaux: A Tool for Thinking Outside the Box. In International Conference on Fundamental Approaches to Software Engineering. 22–39. https://doi.org/10.1007/978-3-662-54494-5_2 Google Scholar
Digital Library
- Timothy Nelson, Christopher Barratt, Daniel J. Dougherty, Kathi Fisler, and Shriram Krishnamurthi. 2010. The Margrave Tool for Firewall Analysis. In USENIX Large Installation System Administration Conference.Google Scholar
- Tim Nelson, Salman Saghafi, Daniel J. Dougherty, Kathi Fisler, and Shriram Krishnamurthi. 2013. Aluminum: Principled Scenario Exploration Through Minimality. In International Conference on Software Engineering. https://doi.org/10.1109/ICSE.2013.6606569 Google Scholar
Cross Ref
- Oded Padon, Kenneth L. McMillan, Aurojit Panda, Mooly Sagiv, and Sharon Shoham. 2016. Ivy: Safety Verification by Interactive Generalization. In Programming Language Design and Implementation (PLDI). Association for Computing Machinery, New York, NY, USA. 614–630. isbn:9781450342612 https://doi.org/10.1145/2908080.2908118 Google Scholar
Digital Library
- Sorawee Porncharoenwase, Tim Nelson, and Shriram Krishnamurthi. 2018. CompoSAT: Specification-Guided Coverage for Model Finding. In International Symposium on Formal Methods (FM). https://doi.org/10.1007/978-3-319-95582-7_34 Google Scholar
Cross Ref
- B. Rittle-Johnson and J. Star. 2007. Does comparing solution methods facilitate conceptual and procedural knowledge: An experimental study on learning to solve equations. Journal of Educational Psychology, 99 (2007), https://doi.org/10.1037/0022-0663.99.3.561 Google Scholar
Cross Ref
- B. Rittle-Johnson and J. R. Star. 2009. Compared with what? The effects of different comparisons on conceptual knowledge and procedural flexibility for equation solving. Journal of Educational Psychology, 101, 3 (2009), https://doi.org/10.1037/a0014224 Google Scholar
Cross Ref
- Santiago Perez De Rosso and Daniel Jackson. 2013. What’s Wrong with Git?: A Conceptual Design Analysis. In SPLASH Onward!. ACM, 37–52. https://doi.org/10.1145/2509578.2509584 Google Scholar
Digital Library
- Natali Ruchansky and Davide Proserpio. 2013. A (Not) NICE Way to Verify the OpenFlow Switch Specification: Formal Modelling of the OpenFlow Switch Using Alloy. ACM Computer Communication Review, 43, 4 (2013), Aug., 527–528. issn:0146-4833 https://doi.org/10.1145/2486001.2491711 Google Scholar
Digital Library
- Salman Saghafi, Natasha Danas, and Daniel J Dougherty. 2015. Exploring Theories with a Model-Finding Assistant. In International Conference on Automated Deduction. 434–449. https://doi.org/10.1007/978-3-319-21401-6_30 Google Scholar
Cross Ref
- Daniel L. Schwartz, Catherine C. Chase, Marily A. Oppezzo, and Doris B. Chin. 2011. Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103, 4 (2011), 759–775. https://doi.org/10.1037/a0025140 Google Scholar
Cross Ref
- Eric L. Seidel, Ranjit Jhala, and Westley Weimer. 2016. Dynamic Witnesses for Static Type Errors (or, Ill-Typed Programs Usually Go Wrong). In International Conference on Functional Programming (ICFP). Association for Computing Machinery, New York, NY, USA. 228–242. isbn:9781450342193 https://doi.org/10.1145/2951913.2951915 Google Scholar
Digital Library
- Abigail Siegel, Mia Santomauro, Tristan Dyer, Tim Nelson, and Shriram Krishnamurthi. 2021. Prototyping Formal Methods Tools: A Protocol Analysis Case Study. In Protocols, Strands, and Logic. 394–413. isbn:978-3-030-91631-2 https://doi.org/10.1007/978-3-030-91631-2_22 Google Scholar
Cross Ref
- Keith Stenning and Michiel van Lambalgen. 2008. Human Reasoning and Cognitive Science. MIT Press. isbn:9780262195836Google Scholar
- J. R. Stroop. 1935. Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18 (1935), 643–662. https://doi.org/10.1037/h0054651 Google Scholar
Cross Ref
- Allison Sullivan, Kaiyuan Wang, Razieh Nokhbeh Zaeem, and Sarfraz Khurshid. 2017. Automated Test Generation and Mutation Testing for Alloy. In Software Testing, Verification and Validation (ICST). https://doi.org/10.1109/ICST.2017.31 Google Scholar
Cross Ref
- Peter Cathcart Wason. 1966. Reasoning. In New Horizons in Psychology I, B. M. Foss (Ed.). Penguin. isbn:9780140207750Google Scholar
- Patrick H. Winston. 1970. Learning Structural Descriptions from Examples. Massachusetts Institute of Technology.Google Scholar
- Pamela Zave. 2012. Using Lightweight Modeling to Understand Chord. ACM Computer Communication Review, 42, 2 (2012), March, 49–57. https://doi.org/10.1145/2185376.2185383 Google Scholar
Digital Library
- Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. RAVEN: A Dataset for Relational and Analogical Visual REasoNing. In Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, 5317–5327. https://doi.org/10.1109/CVPR.2019.00546 Google Scholar
Cross Ref
Index Terms
Applying cognitive principles to model-finding output: the positive value of negative information
Recommendations
Beyond positive or negative
Sentiment analysis techniques are increasingly used to grasp reactions from social media users to unexpected and potentially stressful social events. This paper argues that, alongside assessments of the affective valence of social media content as ...
The False Dichotomy between Positive and Negative Affect in Game Play
CHI PLAY '15: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in PlayMost of the time games make us happy, but sometimes they are frustrating or make us feel sad. They allow us to experience pleasure, success and joy, but they can also yield feelings of frustration, failure, or sorrow from darker themes. In games, we can ...
Cognitive Computational Models of Emotions and Affective Behaviors
Emotions are one of the important subconscious mechanisms that influence human behaviors, attentions, and decision making. The emotion process helps to determine how humans perceive their internal status and needs in order to form consciousness of an ...






Comments