Abstract
With the range and sensitivity of algorithmic decisions expanding at a break-neck speed, it is imperative that we aggressively investigate fairness and bias in decision-making programs. First, we show that a number of recently proposed formal definitions of fairness can be encoded as probabilistic program properties. Second, with the goal of enabling rigorous reasoning about fairness, we design a novel technique for verifying probabilistic properties that admits a wide class of decision-making programs. Third, we present FairSquare, the first verification tool for automatically certifying that a program meets a given fairness property. We evaluate FairSquare on a range of decision-making programs. Our evaluation demonstrates FairSquare’s ability to verify fairness for a range of different programs, which we show are out-of-reach for state-of-the-art program analysis techniques.
- Ifeoma Ajunwa, Sorelle Friedler, Carlos E Scheidegger, and Suresh Venkatasubramanian. 2016. Hiring by algorithm: predicting and preventing disparate impact. Available at SSRN 2746078 (2016).Google Scholar
Cross Ref
- Aws Albarghouthi, Loris D’Antoni, and Samuel Drews. 2017. Repairing Decision-Making Programs Under Uncertainty. Springer, Cham, 181–200. Google Scholar
Cross Ref
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks. https://www.propublica.org/article/ machine- bias- risk- assessments- in- criminal- sentencing . (May 2016). (Accessed on 06/18/2016).Google Scholar
- Eugene Asarin, Olivier Bournez, Thao Dang, and Oded Maler. 2000. Approximate reachability analysis of piecewise-linear dynamical systems. In International Workshop on Hybrid Systems: Computation and Control. Springer, 20–31. Google Scholar
Cross Ref
- Mike Barnett and K Rustan M Leino. 2005. Weakest-precondition of unstructured programs. In ACM SIGSOFT Software Engineering Notes, Vol. 31. ACM, 82–87.Google Scholar
Digital Library
- Solon Barocas and Andrew D Selbst. 2014. Big data’s disparate impact. Available at SSRN 2477899 (2014).Google Scholar
- Gilles Barthe, Pedro R. D’Argenio, and Tamara Rezk. 2004. Secure Information Flow by Self-Composition. In CSFW. Google Scholar
Cross Ref
- Gilles Barthe, Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, César Kunz, and Pierre-Yves Strub. 2014. Proving Differential Privacy in Hoare Logic. In IEEE 27th Computer Security Foundations Symposium, CSF 2014, Vienna, Austria, 19-22 July, 2014. 411–424. Google Scholar
Digital Library
- Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and Antonio Criminisi. 2016. Measuring Neural Net Robustness with Constraints. CoRR abs/1605.07262 (2016). http://arxiv.org/abs/1605.07262Google Scholar
- Saugata Basu, Richard Pollack, and Marie-Françoise Roy. 2006. Algorithms in Real Algebraic Geometry (Algorithms and Computation in Mathematics). Springer-Verlag New York, Inc., Secaucus, NJ, USA.Google Scholar
Digital Library
- Vaishak Belle, Guy Van den Broeck, and Andrea Passerini. 2016. Component Caching in Hybrid Domains with Piecewise Polynomial Densities. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. 3369–3375. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12542Google Scholar
Digital Library
- Vaishak Belle, Andrea Passerini, and Guy Van den Broeck. 2015a. Probabilistic Inference in Hybrid Domains by Weighted Model Integration. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015. 2770–2776. http://ijcai.org/Abstract/15/392Google Scholar
- Vaishak Belle, Guy Van den Broeck, and Andrea Passerini. 2015b. Hashing-based approximate probabilistic inference in hybrid domains. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (UAI).Google Scholar
- Nate Berg. 2014. Predicting crime, LAPD-style. https://www.theguardian.com/cities/2014/jun/25/ predicting- crime- lapd- los- angeles- police- data- analysis- algorithm- minority- report . (June 2014). (Accessed on 06/18/2016).Google Scholar
- Christopher M Bishop. 2006. Pattern recognition. Machine Learning 128 (2006).Google Scholar
- Olivier Bournez, Oded Maler, and Amir Pnueli. 1999. Orthogonal Polyhedra: Representation and Computation. Springer Berlin Heidelberg, Berlin, Heidelberg, 46–60. Google Scholar
Cross Ref
- Christopher W. Brown. 2003. QEPCAD B: A Program for Computing with Semi-algebraic Sets Using CADs. SIGSAM Bull. 37, 4 (Dec. 2003), 97–108. Google Scholar
Digital Library
- Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery 21, 2 (2010), 277–292. Google Scholar
Digital Library
- Michael Carbin, Sasa Misailovic, and Martin C. Rinard. 2013. Verifying quantitative reliability for programs that execute on unreliable hardware. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013. 33–52. Google Scholar
Digital Library
- Swarat Chaudhuri, Martin Clochard, and Armando Solar-Lezama. 2014. Bridging boolean and quantitative synthesis using smoothed proof search. In POPL, Vol. 49. ACM, 207–220. Google Scholar
Digital Library
- Swarat Chaudhuri, Sumit Gulwani, Roberto Lublinerman, and Sara Navidpour. 2011. Proving Programs Robust. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, New York, NY, USA, 102–112. Google Scholar
Digital Library
- Siddhartha Chib and Edward Greenberg. 1995. Understanding the metropolis-hastings algorithm. The american statistician 49, 4 (1995), 327–335.Google Scholar
- Dmitry Chistikov, Rayna Dimitrova, and Rupak Majumdar. 2015. Approximate Counting in SMT and Value Estimation for Probabilistic Programs. In Tools and Algorithms for the Construction and Analysis of Systems - 21st International Conference, TACAS 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015. Proceedings. 320–334. Google Scholar
Digital Library
- Guillaume Claret, Sriram K Rajamani, Aditya V Nori, Andrew D Gordon, and Johannes Borgström. 2013. Bayesian inference using data flow analysis. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 92–102. Google Scholar
Digital Library
- Edmund Clarke, Daniel Kroening, and Flavio Lerda. 2004. A tool for checking ANSI-C programs. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 168–176. Google Scholar
Cross Ref
- Ron Cytron, Jeanne Ferrante, Barry K Rosen, Mark N Wegman, and F Kenneth Zadeck. 1991. Efficiently computing static single assignment form and the control dependence graph. ACM Transactions on Programming Languages and Systems (TOPLAS) 13, 4 (1991), 451–490. Google Scholar
Digital Library
- Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic Transparency via Quantitative Input Influence. In Proceedings of 37th IEEE Symposium on Security and Privacy.Google Scholar
- Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2015. Automated experiments on Ad privacy settings. Proceedings on Privacy Enhancing Technologies 2015, 1 (2015), 92–112. Google Scholar
Cross Ref
- JA De Loera, Brandon Dutra, Matthias Koeppe, Stanislav Moreinis, Gregory Pinto, and Jianqiu Wu. 2012. Software for exact integration of polynomials over polyhedra. ACM Communications in Computer Algebra 45, 3/4 (2012), 169–172. Google Scholar
Digital Library
- Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337–340. Google Scholar
- Cynthia Dwork. 2006. Differential privacy. In Automata, languages and programming. Springer, 1–12. Google Scholar
Digital Library
- Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2012. Fairness through awareness. In Innovations in Theoretical Computer Science 2012, Cambridge, MA, USA, January 8-10, 2012. 214–226. Google Scholar
Digital Library
- Martin Dyer, Alan Frieze, and Ravi Kannan. 1991. A random polynomial-time algorithm for approximating the volume of convex bodies. Journal of the ACM ( JACM) 38, 1 (1991), 1–17. Google Scholar
Digital Library
- Martin E. Dyer and Alan M. Frieze. 1988. On the complexity of computing the volume of a polyhedron. SIAM J. Comput. 17, 5 (1988), 967–974. Google Scholar
Digital Library
- EEOC. 2014. Code of Federal Regulations. https://www.gpo.gov/fdsys/pkg/CFR- 2014- title29- vol4/xml/ CFR- 2014- title29- vol4- part1607.xml . (July 2014). (Accessed on 06/18/2016).Google Scholar
- Virginia Eubanks. 2015. The dangers of letting algorithms enforce policy. http://www.slate.com/articles/technology/ future_tense/2015/04/the_dangers_of _letting_algorithms_enforce_policy.html . (April 2015). (Accessed on 06/18/2016).Google Scholar
- Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015. 259–268. Google Scholar
Digital Library
- Antonio Filieri, Corina S Păsăreanu, and Willem Visser. 2013. Reliability analysis in symbolic pathfinder. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 622–631.Google Scholar
Digital Library
- Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. CoRR abs/1609.07236 (2016). http://arxiv.org/abs/1609.07236Google Scholar
- Timon Gehr, Sasa Misailovic, and Martin Vechev. 2016. PSI: Exact Symbolic Inference for Probabilistic Programs. In Computer aided verification. Springer. Google Scholar
Cross Ref
- Jaco Geldenhuys, Matthew B Dwyer, and Willem Visser. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis. ACM, 166–176.Google Scholar
Digital Library
- Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: a language for generative models. In UAI 2008, Proceedings of the 24th Conference in Uncertainty in Artificial Intelligence, Helsinki, Finland, July 9-12, 2008. 220–229.Google Scholar
Digital Library
- Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. 2014. Probabilistic programming. In Proceedings of the on Future of Software Engineering. ACM, 167–181.Google Scholar
Digital Library
- Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The WEKA data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10–18.Google Scholar
Digital Library
- Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. CoRR abs/1610.02413 (2016). http://arxiv.org/abs/1610.02413Google Scholar
- Dejan Jovanović and Leonardo de Moura. 2013. Solving Non-linear Arithmetic. ACM Commun. Comput. Algebra 46, 3/4 (Jan. 2013), 104–105. Google Scholar
Digital Library
- Leonid Khachiyan. 1993. Complexity of polytope volume computation. Springer. Google Scholar
Cross Ref
- Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent Trade-Offs in the Fair Determination of Risk Scores. In ITCS.Google Scholar
- Nicole Kobie. 2016. Who do you blame when an algorithm gets you fired? http://www.wired.co.uk/article/ make- algorithms- accountable . ( January 2016). (Accessed on 06/18/2016).Google Scholar
- Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: principles and techniques. MIT press.Google Scholar
Digital Library
- Dexter Kozen. 1981. Semantics of probabilistic programs. J. Comput. System Sci. 22, 3 (1981), 328–350. Google Scholar
Cross Ref
- Yi Li, Tian Huat Tan, and Marsha Chechik. 2014. Management of time requirements in component-based systems. In FM 2014: Formal Methods. Springer, 399–415. Google Scholar
Digital Library
- Piotr Mardziel, Stephen Magill, Michael Hicks, and Mudhakar Srivatsa. 2011. Dynamic enforcement of knowledge-based security policies. In Computer Security Foundations Symposium (CSF), 2011 IEEE 24th. IEEE, 114–128. Google Scholar
Digital Library
- Claire Cain Miller. 2015. Can an Algorithm Hire Better Than a Human? http://www.nytimes.com/2015/06/26/upshot/ can- an- algorithm- hire- better- than- a- human.html . (June 2015). (Accessed on 06/18/2016).Google Scholar
- David Monniaux. 2000. Abstract interpretation of probabilistic semantics. In Static Analysis. Springer, 322–339. Google Scholar
Cross Ref
- David Monniaux. 2001a. An abstract Monte-Carlo method for the analysis of probabilistic programs. In ACM SIGPLAN Notices, Vol. 36. ACM, 93–101. Google Scholar
Digital Library
- David Monniaux. 2001b. Backwards abstract interpretation of probabilistic programs. In Programming Languages and Systems. Springer, 367–382. Google Scholar
Cross Ref
- Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10). 807–814.Google Scholar
Digital Library
- Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. 2008. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 560–568. Google Scholar
Digital Library
- Walt L Perry. 2013. Predictive policing: The role of crime forecasting in law enforcement operations. Rand Corporation.Google Scholar
- John Rawls. 2009. A theory of justice. Harvard university press.Google Scholar
- Salvatore Ruggieri. 2014. Using t-closeness anonymity to control for non-discrimination. Transactions on Data Privacy 7, 2 (2014), 99–129.Google Scholar
- Adrian Sampson, Pavel Panchekha, Todd Mytkowicz, Kathryn S McKinley, Dan Grossman, and Luis Ceze. 2014. Expressing and verifying probabilistic assertions. In ACM SIGPLAN Notices, Vol. 49. ACM, 112–122.Google Scholar
Digital Library
- Sriram Sankaranarayanan, Aleksandar Chakarov, and Sumit Gulwani. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’13, Seattle, WA, USA, June 16-19, 2013. 447–458. Google Scholar
Digital Library
- Latanya Sweeney. 2013. Discrimination in online ad delivery. Queue 11, 3 (2013), 10.Google Scholar
Digital Library
- Andrew Tutt. 2016. An FDA for Algorithms. Available at SSRN 2747994 (2016).Google Scholar
- Jennifer Valentino-Devries, Jeremy Singer-Vine, and Ashkan Soltani. 2012. Websites Vary Prices, Deals Based on Users’ Information. http://www.wsj.com/articles/SB10001424127887323777204578189391813881534 . (December 2012). (Accessed on 06/18/2016).Google Scholar
- Santosh Vempala. 2005. Geometric random walks: a survey. Combinatorial and computational geometry 52, 573-612 (2005), 2.Google Scholar
- WH. 2014. Big Data: Seizing Opportunities, Preserving Values. https://www.whitehouse.gov/sites/default/files/docs/ big_data_privacy_report_may_1_2014.pdf . (May 2014). (Accessed on 06/18/2016).Google Scholar
- WH. 2016. Preparing for the Future of Artificial Intelligence. https://www.whitehouse.gov/sites/default/files/whitehouse_ files/microsites/ostp/NSTC/preparing_for_the_future_of _ai.pdf . (Octorber 2016). (Accessed on 10/15/2016).Google Scholar
- Tal Zarsky. 2014. Understanding discrimination in the scored society. Washington Law Review 89, 4 (2014).Google Scholar
- Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. 2013. Learning Fair Representations. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013. 325–333. http://jmlr.org/proceedings/papers/v28/zemel13.htmlGoogle Scholar
Digital Library
Index Terms
FairSquare: probabilistic verification of program fairness
Recommendations
Expressing and verifying probabilistic assertions
PLDI '14: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and ImplementationTraditional assertions express correctness properties that must hold on every program execution. However, many applications have probabilistic outcomes and consequently their correctness properties are also probabilistic (e.g., they identify faces in ...
Expressing and verifying probabilistic assertions
PLDI '14Traditional assertions express correctness properties that must hold on every program execution. However, many applications have probabilistic outcomes and consequently their correctness properties are also probabilistic (e.g., they identify faces in ...
Program verification as probabilistic inference
Proceedings of the 2007 POPL ConferenceIn this paper, we propose a new algorithm for proving the validity or invalidity of a pre/postcondition pair for a program. The algorithm is motivated by the success of the algorithms for probabilistic inference developed in the machine learning ...






Comments