skip to main content

Applying cognitive principles to model-finding output: the positive value of negative information

Published:29 April 2022Publication History
Skip Abstract Section

Abstract

Model-finders, such as SAT/SMT-solvers and Alloy, are used widely both directly and embedded in domain-specific tools. They support both conventional verification and, unlike other verification tools, property-free exploration. To do this effectively, they must produce output that helps users with these tasks. Unfortunately, the output of model-finders has seen relatively little rigorous human-factors study.

Conventionally, these tools tend to show one satisfying instance at a time. Drawing inspiration from the cognitive science literature, we investigate two aspects of model-finder output: how many instances to show at once, and whether all instances must actually satisfy the input constraints. Using both controlled studies and open-ended talk-alouds, we show that there is benefit to showing negative instances in certain settings; the impact of multiple instances is less clear. Our work is a first step in a theoretically grounded approach to understanding how users engage cognitively with model-finder output, and how those tools might better support users in doing so.

References

  1. Devdatta Akhawe, Adam Barth, Peifung E. Lam, John Mitchell, and Dawn Song. 2010. Towards a Formal Foundation of Web Security. In IEEE Computer Security Foundations Symposium. https://doi.org/10.1109/CSF.2010.27 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Louis Alfieri, Timothy J. Nokes-Malach, and Christian D. Schunn. 2013. Learning Through Case Comparisons: A Meta-Analytic Review. Educational Psychologist, 48, 2 (2013), https://doi.org/10.1080/00461520.2013.775712 Google ScholarGoogle ScholarCross RefCross Ref
  3. R. K. Atkinson, S. J. Derry, A. Renkl, and D. Wortham. 2000. Learning from examples: instructional principles from the worked examples research. Review of Educational Research, 70, 2 (2000), https://doi.org/10.3102/00346543070002181 Google ScholarGoogle ScholarCross RefCross Ref
  4. John Backes, Sam Bayless, Byron Cook, Catherine Dodge, Andrew Gacek, Alan J. Hu, Temesghen Kahsai, Bill Kocik, Evgenii Kotelnikov, Jure Kukovec, Sean McLaughlin, Jason Reed, Neha Rungta, John Sizemore, Mark A. Stalzer, Preethi Srinivasan, Pavle Subotic, Carsten Varming, and Blake Whaley. 2019. Reachability Analysis for AWS-Based Networks. In Computer Aided Verification - 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part II, Isil Dillig and Serdar Tasiran (Eds.) (Lecture Notes in Computer Science, Vol. 11562). Springer, 231–241. https://doi.org/10.1007/978-3-030-25543-5_14 Google ScholarGoogle ScholarCross RefCross Ref
  5. John Backes, Pauline Bolignano, Byron Cook, Catherine Dodge, Andrew Gacek, Kasper Sœ Luckow, Neha Rungta, Oksana Tkachuk, and Carsten Varming. 2018. Semantic-based Automated Reasoning for AWS Access Policies using SMT. In Formal Methods in Computer-Aided Design. IEEE, 1–9. https://doi.org/10.23919/FMCAD.2018.8602994 Google ScholarGoogle ScholarCross RefCross Ref
  6. Clark Barrett, Christopher L. Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. 2011. CVC4. In International Conference on Computer Aided Verification. Springer Berlin Heidelberg, Berlin, Heidelberg. 171–177. isbn:978-3-642-22110-1 https://doi.org/10.1007/978-3-642-22110-1_14 Google ScholarGoogle ScholarCross RefCross Ref
  7. Irving Biederman and Margaret M. Shiffrar. 1987. Sexing Day-Old Chicks: A Case Study and Expert Systems Analysis of a Difficult Perceptual-Learning Task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 4 (1987), 640–645. https://doi.org/10.1037/0278-7393.13.4.640 Google ScholarGoogle ScholarCross RefCross Ref
  8. R. Brown and C. Hanlon. 1970. Derivational complexity and order of acquisition on child speech. In Cognition and the development of language, J. Hayes (Ed.). Wiley. isbn:978-0471364733Google ScholarGoogle Scholar
  9. François Chollet. 2019. On the Measure of Intelligence. arxiv:1911.01547.Google ScholarGoogle Scholar
  10. Robert Clarisó and Jordi Cabot. 2020. Diverse Scenario Exploration in Model Finders Using Graph Kernels and Clustering. In Rigorous State Based Methods. Springer International Publishing, Cham. 27–43. isbn:978-3-030-48077-6 https://doi.org/10.1007/978-3-030-48077-6_3 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Edmund M. Clarke, E. Allen Emerson, and Joseph Sifakis. 2009. Model Checking: Algorithmic Verification and Debugging. Commun. ACM, 52, 11 (2009), Nov., 74–84. issn:0001-0782 https://doi.org/10.1145/1592761.1592781 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 1 (1960), 37–46. https://doi.org/10.1177/001316446002000104 Google ScholarGoogle ScholarCross RefCross Ref
  13. Leda Cosmides and John Tooby. 1992. Cognitive Adaptions for Social Exchange. In The Adapted Mind: Evolutionary Psychology and the Generation of Culture, Leda Cosmides, John Tooby, and Jerome H. Barkow (Eds.). Oxford University Press. isbn:978-0195101072Google ScholarGoogle Scholar
  14. Alcino Cunha, Nuno Macedo, and Tiago Guimarães. 2014. Target Oriented Relational Model Finding. In International Conference on Fundamental Approaches to Software Engineering. 17–31. https://doi.org/10.1007/978-3-642-54804-8_2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Natasha Danas, Tim Nelson, Lane Harrison, Shriram Krishnamurthi, and Daniel J. Dougherty. 2017. User Studies of Principled Model Finder Output. In Software Engineering and Formal Methods. https://doi.org/10.1007/978-3-319-66197-1_11 Google ScholarGoogle ScholarCross RefCross Ref
  16. Leonardo de Moura and N. Bjørner. 2008. Z3: An efficient SMT solver. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems (Lecture Notes in Computer Science, Vol. 4963). Springer, 337. https://doi.org/10.1007/978-3-540-78800-3_24 Google ScholarGoogle ScholarCross RefCross Ref
  17. Charles W. Dunnett. 1955. A Multiple Comparison Procedure for Comparing Several Treatments with a Control. J. Amer. Statist. Assoc., 50, 272 (1955), 1096–1121. https://doi.org/10.1080/01621459.1955.10501294 arxiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1955.10501294. Google ScholarGoogle ScholarCross RefCross Ref
  18. Tristan Dyer and John Baugh. 2021. Sterling: A web-based visualizer for relational modeling languages. In Rigorous State Based Methods. ABZ 2021. Springer, Cham. 99–104. https://doi.org/10.1007/978-3-030-77543-8_7 Lecture Notes in Computer Science 12709 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Tristan Dyer, Tim Nelson, Kathi Fisler, and Shriram Krishnamurthi. 2022. Applying Cognitive Principles to Model-Finding Output: The Positive Value of Negative Information (artifact). https://doi.org/10.5281/zenodo.6370152 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Ari Fogel, Stanley Fung, Luis Pedrosa, Meg Walraed-Sullivan, Ramesh Govindan, Ratul Mahajan, and Todd Millstein. 2015. A General Approach to Network Configuration Analysis. In Networked Systems Design and Implementation. 469–483. isbn:978-1-931971-218 https://doi.org/10.5555/2789770.2789803Google ScholarGoogle Scholar
  21. Eleanor J. Gibson. 1969. Principles of Perceptual Learning and Development. Appleton-Century-Crofts. isbn:9780390361455Google ScholarGoogle Scholar
  22. James J. Gibson and Eleanor J. Gibson. 1955. Perceptual Learning: Differentiation or Enrichment? Psychological Review, 62, 1 (1955), 32–41. https://doi.org/10.1037/h0048826 Google ScholarGoogle ScholarCross RefCross Ref
  23. M. L. Gick and K. Paterson. 1992. Do contrasting examples facilitate schema acquisition and analogical transfer? Canadian Journal of Psychology, 46, 4 (1992), https://doi.org/10.1037/h0084333 Google ScholarGoogle ScholarCross RefCross Ref
  24. Rubi Hammer, Tomer Hertz, Shaul Hochstein, and Daphna Weinshall. 2009. Category learning from equivalence constraints. Cognitive Processing, 10, 3 (2009), 211–232. isbn:1612-4790 https://doi.org/10.1007/s10339-008-0243-x Google ScholarGoogle ScholarCross RefCross Ref
  25. William L. Hays. 1994. Statistics (5th ed.). Harcourt Brace College Publishers, Fort Worth. isbn:0030744679Google ScholarGoogle Scholar
  26. Daniel Jackson. 2012. Software Abstractions: Logic, Language, and Analysis (2 ed.). MIT Press. isbn:0262017156 https://doi.org/10.5555/2141100Google ScholarGoogle Scholar
  27. Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studies with Mechanical Turk. In SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA. 453–456. isbn:9781605580111 https://doi.org/10.1145/1357054.1357127 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Chinmay Kulkarni, Steven P. Dow, and Scott R. Klemmer. 2014. Early and Repeated Exposure to Examples Improves Creative Work. Springer International Publishing, 49–62. isbn:978-3-319-01303-9 https://doi.org/10.1007/978-3-319-01303-9_4 Google ScholarGoogle ScholarCross RefCross Ref
  29. Angelika Kullberg, Ulla Runesson Kempe, and Ference Marton. 2017. What is made possible to learn when using the variation theory of learning in teaching mathematics? ZDM: The International Journal on Mathematics Education, 49, 3 (2017), https://doi.org/10.1007/s11858-017-0858-4 Google ScholarGoogle ScholarCross RefCross Ref
  30. Nuno Macedo, Alcino Cunha, and Tiago Guimarães. 2015. Exploring Scenario Exploration. In International Conference on Fundamental Approaches to Software Engineering. https://doi.org/10.1007/978-3-662-46675-9_20 Google ScholarGoogle ScholarCross RefCross Ref
  31. Ferney A. Maldonado-Lopez, Jaime Chavarriaga, and Yezid Donoso. 2014. Detecting Network Policy Conflicts Using Alloy. In Conference on Abstract State Machines, Alloy, B, and Z. https://doi.org/10.1007/978-3-662-43652-3_31 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Shahar Maoz, Jan Oliver Ringert, and Bernhard Rumpe. 2011. CD2Alloy: Class Diagrams Analysis Using Alloy Revisited. In Model Driven Engineering Languages and Systems. https://doi.org/10.1007/978-3-642-24485-8_44 Google ScholarGoogle ScholarCross RefCross Ref
  33. Gary F. Marcus. 1993. Negative evidence in language acquisition. Cognition, 46 (1993), https://doi.org/10.1016/0010-0277(93)90022-N Google ScholarGoogle ScholarCross RefCross Ref
  34. Ference Marton. 2014. Necessary Conditions of Learning. Routledge. isbn:978-0415739146Google ScholarGoogle Scholar
  35. Ference Marton and Ming Fai Pang. 2013. Meanings are acquired from experiencing differences against a background of sameness, rather than from experiencing sameness against a background of difference: Putting a conjecture to the test by embedding it in a pedagogical tool. Frontline Learning Research, 1 (2013), https://doi.org/10.14786/flr.v1i1.16 Google ScholarGoogle ScholarCross RefCross Ref
  36. Vajih Montaghami and Derek Rayside. 2017. Bordeaux: A Tool for Thinking Outside the Box. In International Conference on Fundamental Approaches to Software Engineering. 22–39. https://doi.org/10.1007/978-3-662-54494-5_2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Timothy Nelson, Christopher Barratt, Daniel J. Dougherty, Kathi Fisler, and Shriram Krishnamurthi. 2010. The Margrave Tool for Firewall Analysis. In USENIX Large Installation System Administration Conference.Google ScholarGoogle Scholar
  38. Tim Nelson, Salman Saghafi, Daniel J. Dougherty, Kathi Fisler, and Shriram Krishnamurthi. 2013. Aluminum: Principled Scenario Exploration Through Minimality. In International Conference on Software Engineering. https://doi.org/10.1109/ICSE.2013.6606569 Google ScholarGoogle ScholarCross RefCross Ref
  39. Oded Padon, Kenneth L. McMillan, Aurojit Panda, Mooly Sagiv, and Sharon Shoham. 2016. Ivy: Safety Verification by Interactive Generalization. In Programming Language Design and Implementation (PLDI). Association for Computing Machinery, New York, NY, USA. 614–630. isbn:9781450342612 https://doi.org/10.1145/2908080.2908118 Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Sorawee Porncharoenwase, Tim Nelson, and Shriram Krishnamurthi. 2018. CompoSAT: Specification-Guided Coverage for Model Finding. In International Symposium on Formal Methods (FM). https://doi.org/10.1007/978-3-319-95582-7_34 Google ScholarGoogle ScholarCross RefCross Ref
  41. B. Rittle-Johnson and J. Star. 2007. Does comparing solution methods facilitate conceptual and procedural knowledge: An experimental study on learning to solve equations. Journal of Educational Psychology, 99 (2007), https://doi.org/10.1037/0022-0663.99.3.561 Google ScholarGoogle ScholarCross RefCross Ref
  42. B. Rittle-Johnson and J. R. Star. 2009. Compared with what? The effects of different comparisons on conceptual knowledge and procedural flexibility for equation solving. Journal of Educational Psychology, 101, 3 (2009), https://doi.org/10.1037/a0014224 Google ScholarGoogle ScholarCross RefCross Ref
  43. Santiago Perez De Rosso and Daniel Jackson. 2013. What’s Wrong with Git?: A Conceptual Design Analysis. In SPLASH Onward!. ACM, 37–52. https://doi.org/10.1145/2509578.2509584 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Natali Ruchansky and Davide Proserpio. 2013. A (Not) NICE Way to Verify the OpenFlow Switch Specification: Formal Modelling of the OpenFlow Switch Using Alloy. ACM Computer Communication Review, 43, 4 (2013), Aug., 527–528. issn:0146-4833 https://doi.org/10.1145/2486001.2491711 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Salman Saghafi, Natasha Danas, and Daniel J Dougherty. 2015. Exploring Theories with a Model-Finding Assistant. In International Conference on Automated Deduction. 434–449. https://doi.org/10.1007/978-3-319-21401-6_30 Google ScholarGoogle ScholarCross RefCross Ref
  46. Daniel L. Schwartz, Catherine C. Chase, Marily A. Oppezzo, and Doris B. Chin. 2011. Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103, 4 (2011), 759–775. https://doi.org/10.1037/a0025140 Google ScholarGoogle ScholarCross RefCross Ref
  47. Eric L. Seidel, Ranjit Jhala, and Westley Weimer. 2016. Dynamic Witnesses for Static Type Errors (or, Ill-Typed Programs Usually Go Wrong). In International Conference on Functional Programming (ICFP). Association for Computing Machinery, New York, NY, USA. 228–242. isbn:9781450342193 https://doi.org/10.1145/2951913.2951915 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Abigail Siegel, Mia Santomauro, Tristan Dyer, Tim Nelson, and Shriram Krishnamurthi. 2021. Prototyping Formal Methods Tools: A Protocol Analysis Case Study. In Protocols, Strands, and Logic. 394–413. isbn:978-3-030-91631-2 https://doi.org/10.1007/978-3-030-91631-2_22 Google ScholarGoogle ScholarCross RefCross Ref
  49. Keith Stenning and Michiel van Lambalgen. 2008. Human Reasoning and Cognitive Science. MIT Press. isbn:9780262195836Google ScholarGoogle Scholar
  50. J. R. Stroop. 1935. Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18 (1935), 643–662. https://doi.org/10.1037/h0054651 Google ScholarGoogle ScholarCross RefCross Ref
  51. Allison Sullivan, Kaiyuan Wang, Razieh Nokhbeh Zaeem, and Sarfraz Khurshid. 2017. Automated Test Generation and Mutation Testing for Alloy. In Software Testing, Verification and Validation (ICST). https://doi.org/10.1109/ICST.2017.31 Google ScholarGoogle ScholarCross RefCross Ref
  52. Peter Cathcart Wason. 1966. Reasoning. In New Horizons in Psychology I, B. M. Foss (Ed.). Penguin. isbn:9780140207750Google ScholarGoogle Scholar
  53. Patrick H. Winston. 1970. Learning Structural Descriptions from Examples. Massachusetts Institute of Technology.Google ScholarGoogle Scholar
  54. Pamela Zave. 2012. Using Lightweight Modeling to Understand Chord. ACM Computer Communication Review, 42, 2 (2012), March, 49–57. https://doi.org/10.1145/2185376.2185383 Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. RAVEN: A Dataset for Relational and Analogical Visual REasoNing. In Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, 5317–5327. https://doi.org/10.1109/CVPR.2019.00546 Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Applying cognitive principles to model-finding output: the positive value of negative information

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!