skip to main content
research-article
Open Access

FairSquare: probabilistic verification of program fairness

Published:12 October 2017Publication History
Skip Abstract Section

Abstract

With the range and sensitivity of algorithmic decisions expanding at a break-neck speed, it is imperative that we aggressively investigate fairness and bias in decision-making programs. First, we show that a number of recently proposed formal definitions of fairness can be encoded as probabilistic program properties. Second, with the goal of enabling rigorous reasoning about fairness, we design a novel technique for verifying probabilistic properties that admits a wide class of decision-making programs. Third, we present FairSquare, the first verification tool for automatically certifying that a program meets a given fairness property. We evaluate FairSquare on a range of decision-making programs. Our evaluation demonstrates FairSquare’s ability to verify fairness for a range of different programs, which we show are out-of-reach for state-of-the-art program analysis techniques.

References

  1. Ifeoma Ajunwa, Sorelle Friedler, Carlos E Scheidegger, and Suresh Venkatasubramanian. 2016. Hiring by algorithm: predicting and preventing disparate impact. Available at SSRN 2746078 (2016).Google ScholarGoogle ScholarCross RefCross Ref
  2. Aws Albarghouthi, Loris D’Antoni, and Samuel Drews. 2017. Repairing Decision-Making Programs Under Uncertainty. Springer, Cham, 181–200. Google ScholarGoogle ScholarCross RefCross Ref
  3. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks. https://www.propublica.org/article/ machine- bias- risk- assessments- in- criminal- sentencing . (May 2016). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  4. Eugene Asarin, Olivier Bournez, Thao Dang, and Oded Maler. 2000. Approximate reachability analysis of piecewise-linear dynamical systems. In International Workshop on Hybrid Systems: Computation and Control. Springer, 20–31. Google ScholarGoogle ScholarCross RefCross Ref
  5. Mike Barnett and K Rustan M Leino. 2005. Weakest-precondition of unstructured programs. In ACM SIGSOFT Software Engineering Notes, Vol. 31. ACM, 82–87.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Solon Barocas and Andrew D Selbst. 2014. Big data’s disparate impact. Available at SSRN 2477899 (2014).Google ScholarGoogle Scholar
  7. Gilles Barthe, Pedro R. D’Argenio, and Tamara Rezk. 2004. Secure Information Flow by Self-Composition. In CSFW. Google ScholarGoogle ScholarCross RefCross Ref
  8. Gilles Barthe, Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, César Kunz, and Pierre-Yves Strub. 2014. Proving Differential Privacy in Hoare Logic. In IEEE 27th Computer Security Foundations Symposium, CSF 2014, Vienna, Austria, 19-22 July, 2014. 411–424. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and Antonio Criminisi. 2016. Measuring Neural Net Robustness with Constraints. CoRR abs/1605.07262 (2016). http://arxiv.org/abs/1605.07262Google ScholarGoogle Scholar
  10. Saugata Basu, Richard Pollack, and Marie-Françoise Roy. 2006. Algorithms in Real Algebraic Geometry (Algorithms and Computation in Mathematics). Springer-Verlag New York, Inc., Secaucus, NJ, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Vaishak Belle, Guy Van den Broeck, and Andrea Passerini. 2016. Component Caching in Hybrid Domains with Piecewise Polynomial Densities. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. 3369–3375. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12542Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Vaishak Belle, Andrea Passerini, and Guy Van den Broeck. 2015a. Probabilistic Inference in Hybrid Domains by Weighted Model Integration. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015. 2770–2776. http://ijcai.org/Abstract/15/392Google ScholarGoogle Scholar
  13. Vaishak Belle, Guy Van den Broeck, and Andrea Passerini. 2015b. Hashing-based approximate probabilistic inference in hybrid domains. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (UAI).Google ScholarGoogle Scholar
  14. Nate Berg. 2014. Predicting crime, LAPD-style. https://www.theguardian.com/cities/2014/jun/25/ predicting- crime- lapd- los- angeles- police- data- analysis- algorithm- minority- report . (June 2014). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  15. Christopher M Bishop. 2006. Pattern recognition. Machine Learning 128 (2006).Google ScholarGoogle Scholar
  16. Olivier Bournez, Oded Maler, and Amir Pnueli. 1999. Orthogonal Polyhedra: Representation and Computation. Springer Berlin Heidelberg, Berlin, Heidelberg, 46–60. Google ScholarGoogle ScholarCross RefCross Ref
  17. Christopher W. Brown. 2003. QEPCAD B: A Program for Computing with Semi-algebraic Sets Using CADs. SIGSAM Bull. 37, 4 (Dec. 2003), 97–108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery 21, 2 (2010), 277–292. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Michael Carbin, Sasa Misailovic, and Martin C. Rinard. 2013. Verifying quantitative reliability for programs that execute on unreliable hardware. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013. 33–52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Swarat Chaudhuri, Martin Clochard, and Armando Solar-Lezama. 2014. Bridging boolean and quantitative synthesis using smoothed proof search. In POPL, Vol. 49. ACM, 207–220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Swarat Chaudhuri, Sumit Gulwani, Roberto Lublinerman, and Sara Navidpour. 2011. Proving Programs Robust. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, New York, NY, USA, 102–112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Siddhartha Chib and Edward Greenberg. 1995. Understanding the metropolis-hastings algorithm. The american statistician 49, 4 (1995), 327–335.Google ScholarGoogle Scholar
  23. Dmitry Chistikov, Rayna Dimitrova, and Rupak Majumdar. 2015. Approximate Counting in SMT and Value Estimation for Probabilistic Programs. In Tools and Algorithms for the Construction and Analysis of Systems - 21st International Conference, TACAS 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015. Proceedings. 320–334. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Guillaume Claret, Sriram K Rajamani, Aditya V Nori, Andrew D Gordon, and Johannes Borgström. 2013. Bayesian inference using data flow analysis. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 92–102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Edmund Clarke, Daniel Kroening, and Flavio Lerda. 2004. A tool for checking ANSI-C programs. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 168–176. Google ScholarGoogle ScholarCross RefCross Ref
  26. Ron Cytron, Jeanne Ferrante, Barry K Rosen, Mark N Wegman, and F Kenneth Zadeck. 1991. Efficiently computing static single assignment form and the control dependence graph. ACM Transactions on Programming Languages and Systems (TOPLAS) 13, 4 (1991), 451–490. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic Transparency via Quantitative Input Influence. In Proceedings of 37th IEEE Symposium on Security and Privacy.Google ScholarGoogle Scholar
  28. Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2015. Automated experiments on Ad privacy settings. Proceedings on Privacy Enhancing Technologies 2015, 1 (2015), 92–112. Google ScholarGoogle ScholarCross RefCross Ref
  29. JA De Loera, Brandon Dutra, Matthias Koeppe, Stanislav Moreinis, Gregory Pinto, and Jianqiu Wu. 2012. Software for exact integration of polynomials over polyhedra. ACM Communications in Computer Algebra 45, 3/4 (2012), 169–172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337–340. Google ScholarGoogle Scholar
  31. Cynthia Dwork. 2006. Differential privacy. In Automata, languages and programming. Springer, 1–12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2012. Fairness through awareness. In Innovations in Theoretical Computer Science 2012, Cambridge, MA, USA, January 8-10, 2012. 214–226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Martin Dyer, Alan Frieze, and Ravi Kannan. 1991. A random polynomial-time algorithm for approximating the volume of convex bodies. Journal of the ACM ( JACM) 38, 1 (1991), 1–17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Martin E. Dyer and Alan M. Frieze. 1988. On the complexity of computing the volume of a polyhedron. SIAM J. Comput. 17, 5 (1988), 967–974. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. EEOC. 2014. Code of Federal Regulations. https://www.gpo.gov/fdsys/pkg/CFR- 2014- title29- vol4/xml/ CFR- 2014- title29- vol4- part1607.xml . (July 2014). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  36. Virginia Eubanks. 2015. The dangers of letting algorithms enforce policy. http://www.slate.com/articles/technology/ future_tense/2015/04/the_dangers_of _letting_algorithms_enforce_policy.html . (April 2015). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  37. Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015. 259–268. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Antonio Filieri, Corina S Păsăreanu, and Willem Visser. 2013. Reliability analysis in symbolic pathfinder. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 622–631.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. CoRR abs/1609.07236 (2016). http://arxiv.org/abs/1609.07236Google ScholarGoogle Scholar
  40. Timon Gehr, Sasa Misailovic, and Martin Vechev. 2016. PSI: Exact Symbolic Inference for Probabilistic Programs. In Computer aided verification. Springer. Google ScholarGoogle ScholarCross RefCross Ref
  41. Jaco Geldenhuys, Matthew B Dwyer, and Willem Visser. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis. ACM, 166–176.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: a language for generative models. In UAI 2008, Proceedings of the 24th Conference in Uncertainty in Artificial Intelligence, Helsinki, Finland, July 9-12, 2008. 220–229.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. 2014. Probabilistic programming. In Proceedings of the on Future of Software Engineering. ACM, 167–181.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The WEKA data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10–18.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. CoRR abs/1610.02413 (2016). http://arxiv.org/abs/1610.02413Google ScholarGoogle Scholar
  46. Dejan Jovanović and Leonardo de Moura. 2013. Solving Non-linear Arithmetic. ACM Commun. Comput. Algebra 46, 3/4 (Jan. 2013), 104–105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Leonid Khachiyan. 1993. Complexity of polytope volume computation. Springer. Google ScholarGoogle ScholarCross RefCross Ref
  48. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent Trade-Offs in the Fair Determination of Risk Scores. In ITCS.Google ScholarGoogle Scholar
  49. Nicole Kobie. 2016. Who do you blame when an algorithm gets you fired? http://www.wired.co.uk/article/ make- algorithms- accountable . ( January 2016). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  50. Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: principles and techniques. MIT press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Dexter Kozen. 1981. Semantics of probabilistic programs. J. Comput. System Sci. 22, 3 (1981), 328–350. Google ScholarGoogle ScholarCross RefCross Ref
  52. Yi Li, Tian Huat Tan, and Marsha Chechik. 2014. Management of time requirements in component-based systems. In FM 2014: Formal Methods. Springer, 399–415. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Piotr Mardziel, Stephen Magill, Michael Hicks, and Mudhakar Srivatsa. 2011. Dynamic enforcement of knowledge-based security policies. In Computer Security Foundations Symposium (CSF), 2011 IEEE 24th. IEEE, 114–128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Claire Cain Miller. 2015. Can an Algorithm Hire Better Than a Human? http://www.nytimes.com/2015/06/26/upshot/ can- an- algorithm- hire- better- than- a- human.html . (June 2015). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  55. David Monniaux. 2000. Abstract interpretation of probabilistic semantics. In Static Analysis. Springer, 322–339. Google ScholarGoogle ScholarCross RefCross Ref
  56. David Monniaux. 2001a. An abstract Monte-Carlo method for the analysis of probabilistic programs. In ACM SIGPLAN Notices, Vol. 36. ACM, 93–101. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. David Monniaux. 2001b. Backwards abstract interpretation of probabilistic programs. In Programming Languages and Systems. Springer, 367–382. Google ScholarGoogle ScholarCross RefCross Ref
  58. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10). 807–814.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. 2008. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 560–568. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Walt L Perry. 2013. Predictive policing: The role of crime forecasting in law enforcement operations. Rand Corporation.Google ScholarGoogle Scholar
  61. John Rawls. 2009. A theory of justice. Harvard university press.Google ScholarGoogle Scholar
  62. Salvatore Ruggieri. 2014. Using t-closeness anonymity to control for non-discrimination. Transactions on Data Privacy 7, 2 (2014), 99–129.Google ScholarGoogle Scholar
  63. Adrian Sampson, Pavel Panchekha, Todd Mytkowicz, Kathryn S McKinley, Dan Grossman, and Luis Ceze. 2014. Expressing and verifying probabilistic assertions. In ACM SIGPLAN Notices, Vol. 49. ACM, 112–122.Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Sriram Sankaranarayanan, Aleksandar Chakarov, and Sumit Gulwani. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’13, Seattle, WA, USA, June 16-19, 2013. 447–458. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Latanya Sweeney. 2013. Discrimination in online ad delivery. Queue 11, 3 (2013), 10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Andrew Tutt. 2016. An FDA for Algorithms. Available at SSRN 2747994 (2016).Google ScholarGoogle Scholar
  67. Jennifer Valentino-Devries, Jeremy Singer-Vine, and Ashkan Soltani. 2012. Websites Vary Prices, Deals Based on Users’ Information. http://www.wsj.com/articles/SB10001424127887323777204578189391813881534 . (December 2012). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  68. Santosh Vempala. 2005. Geometric random walks: a survey. Combinatorial and computational geometry 52, 573-612 (2005), 2.Google ScholarGoogle Scholar
  69. WH. 2014. Big Data: Seizing Opportunities, Preserving Values. https://www.whitehouse.gov/sites/default/files/docs/ big_data_privacy_report_may_1_2014.pdf . (May 2014). (Accessed on 06/18/2016).Google ScholarGoogle Scholar
  70. WH. 2016. Preparing for the Future of Artificial Intelligence. https://www.whitehouse.gov/sites/default/files/whitehouse_ files/microsites/ostp/NSTC/preparing_for_the_future_of _ai.pdf . (Octorber 2016). (Accessed on 10/15/2016).Google ScholarGoogle Scholar
  71. Tal Zarsky. 2014. Understanding discrimination in the scored society. Washington Law Review 89, 4 (2014).Google ScholarGoogle Scholar
  72. Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. 2013. Learning Fair Representations. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013. 325–333. http://jmlr.org/proceedings/papers/v28/zemel13.htmlGoogle ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. FairSquare: probabilistic verification of program fairness

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!