Abstract
Differential privacy has emerged as a promising probabilistic formulation of privacy, generating intense interest within academia and industry. We present a push-button, automated technique for verifying ε-differential privacy of sophisticated randomized algorithms. We make several conceptual, algorithmic, and practical contributions: (i) Inspired by the recent advances on approximate couplings and randomness alignment, we present a new proof technique called coupling strategies, which casts differential privacy proofs as a winning strategy in a game where we have finite privacy resources to expend. (ii) To discover a winning strategy, we present a constraint-based formulation of the problem as a set of Horn modulo couplings (HMC) constraints, a novel combination of first-order Horn clauses and probabilistic constraints. (iii) We present a technique for solving HMC constraints by transforming probabilistic constraints into logical constraints with uninterpreted functions. (iv) Finally, we implement our technique in the FairSquare verifier and provide the first automated privacy proofs for a number of challenging algorithms from the differential privacy literature, including Report Noisy Max, the Exponential Mechanism, and the Sparse Vector Mechanism.
Supplemental Material
- Aws Albarghouthi, Loris D’Antoni, Samuel Drews, and Aditya V. Nori. 2017. FairSquare: Probabilistic Verification of Program Fairness. Proc. ACM Program. Lang. 1, OOPSLA, Article 80 (Oct. 2017), 30 pages. Google Scholar
Digital Library
- Arthur Azevedo de Amorim, Marco Gaboardi, Emilio Jesús Gallego Arias, and Justin Hsu. 2014. Really natural linear indexed type-checking. In Symposium on Implementation and Application of Functional Programming Languages (IFL), Boston, Massachusetts . ACM Press, 5:1–5:12. http://arxiv.org/abs/1503.04522Google Scholar
Digital Library
- Arthur Azevedo de Amorim, Marco Gaboardi, Justin Hsu, Shin-ya Katsumata, and Ikram Cherigui. 2017. A semantic account of metric preservation. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Paris, France . 545–556. http://arxiv.org/abs/1702.00374 Google Scholar
Digital Library
- Gilles Barthe, Thomas Espitau, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2017a. Proving uniformity and independence by self-composition and coupling. In International Conference on Logic for Programming, Artificial Intelligence and Reasoning (LPAR), Maun, Botswana (EPiC Series in Computing) , Vol. 46. 385–403. https://arxiv.org/abs/1701.06477Google Scholar
- Gilles Barthe, Thomas Espitau, Justin Hsu, Tetsuya Sato, and Pierre-Yves Strub. 2017b. ⋆-Liftings for differential privacy. In International Colloquium on Automata, Languages and Programming (ICALP), Warsaw, Poland . https://arxiv.org/abs/1705. 00133Google Scholar
- Gilles Barthe, Noémie Fong, Marco Gaboardi, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2016a. Advanced probabilistic couplings for differential privacy. In ACM SIGSAC Conference on Computer and Communications Security (CCS), Vienna, Austria . https://arxiv.org/abs/1606.07143 Google Scholar
Digital Library
- Gilles Barthe, Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, César Kunz, and Pierre-Yves Strub. 2014. Proving differential privacy in Hoare logic. In IEEE Computer Security Foundations Symposium (CSF), Vienna, Austria. 411–424. http://arxiv.org/abs/1407.2988 Google Scholar
Digital Library
- Gilles Barthe, Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, Aaron Roth, and Pierre-Yves Strub. 2015. Higher-order approximate relational refinement types for mechanism design and differential privacy. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Mumbai, India . 55–68. http://arxiv.org/abs/1407.6845 Google Scholar
Digital Library
- Gilles Barthe, Marco Gaboardi, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2016b. Proving differential privacy via probabilistic couplings. In IEEE Symposium on Logic in Computer Science (LICS), New York, New York. 749–758. http://arxiv.org/abs/1601.05047 Google Scholar
Digital Library
- Gilles Barthe, Marco Gaboardi, Justin Hsu, and Benjamin C. Pierce. 2016c. Programming language techniques for differential privacy. ACM SIGLOG News 3, 1 (Jan. 2016), 34–53. http://siglog.hosting.acm.org/wp-content/uploads/2016/01/siglog_ news_7.pdfGoogle Scholar
Digital Library
- Gilles Barthe, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2017c. Coupling proofs are probabilistic product programs. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Paris, France. 161–174. http://arxiv.org/abs/1607.03455 Google Scholar
Digital Library
- Gilles Barthe, Boris Köpf, Federico Olmedo, and Santiago Zanella-Béguelin. 2013. Probabilistic Relational Reasoning for Differential Privacy. ACM Transactions on Programming Languages and Systems 35, 3 (2013), 9. http://software.imdea. org/~bkoepf/papers/toplas13.pdfGoogle Scholar
Digital Library
- Gilles Barthe and Federico Olmedo. 2013. Beyond Differential Privacy: Composition Theorems and Relational Logic for f -divergences between Probabilistic Programs. In International Colloquium on Automata, Languages and Programming (ICALP), Riga, Latvia (Lecture Notes in Computer Science) , Vol. 7966. Springer-Verlag, 49–60. http://certicrypt.gforge.inria. fr/2013.ICALP.pdf Google Scholar
Digital Library
- Tewodros Beyene, Swarat Chaudhuri, Corneliu Popeea, and Andrey Rybalchenko. 2014. A Constraint-based Approach to Solving Games on Infinite Graphs. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), San Diego, California . New York, NY, USA, 221–233. Google Scholar
Digital Library
- Tewodros A Beyene, Corneliu Popeea, and Andrey Rybalchenko. 2013. Solving existentially quantified Horn clauses. In International Conference on Computer Aided Verification (CAV), Saint Petersburg, Russia . Springer-Verlag, 869–882. Google Scholar
Digital Library
- Dirk Beyer, Alessandro Cimatti, Alberto Griggio, M. Erkan Keremoglu, and Roberto Sebastiani. 2009. Software model checking via large-block encoding. In Formal Methods in Computer-Aided Design (FMCAD), Austin, Texas. IEEE, 25–32. Google Scholar
Cross Ref
- Nikolaj Bjørner, Arie Gurfinkel, Kenneth L. McMillan, and Andrey Rybalchenko. 2015. Horn clause solvers for program verification. In Fields of Logic and Computation II. Springer-Verlag, 24–51. Google Scholar
- James Bornholt, Emina Torlak, Dan Grossman, and Luis Ceze. 2016. Optimizing synthesis with metasketches. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Saint Petersburg, Florida . 775–788. Google Scholar
Digital Library
- T.-H. Hubert Chan, Elaine Shi, and Dawn Song. 2011. Private and continual release of statistics. ACM Transactions on Information and System Security 14, 3 (2011), 26. http://eprint.iacr.org/2010/076.pdfGoogle Scholar
- Alessandro Cimatti, Alberto Griggio, Bastiaan Joost Schaafsma, and Roberto Sebastiani. 2013. The MathSAT5 SMT Solver., Vol. 7795. Springer-Verlag, 93–107. Google Scholar
Digital Library
- Michael R. Clarkson and Fred B. Schneider. 2010. Hyperproperties. Journal of Computer Security 18, 6 (2010), 1157–1210. Google Scholar
Digital Library
- Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. 2006. Calibrating Noise to Sensitivity in Private Data Analysis. In IACR Theory of Cryptography Conference (TCC), New York, New York (Lecture Notes in Computer Science), Vol. 3876. Springer-Verlag, 265–284. Google Scholar
Digital Library
- Cynthia Dwork, Moni Naor, Toniann Pitassi, and Guy N. Rothblum. 2010. Differential privacy under continual observation. In ACM SIGACT Symposium on Theory of Computing (STOC), Cambridge, Massachusetts. 715–724. http://www.mit.edu/ ~rothblum/papers/continalobs.pdf Google Scholar
Digital Library
- Cynthia Dwork, Moni Naor, Omer Reingold, Guy N. Rothblum, and Salil Vadhan. 2009. On the complexity of differentially private data release: Efficient algorithms and hardness results. In ACM SIGACT Symposium on Theory of Computing (STOC), Bethesda, Maryland . 381–390. Google Scholar
Digital Library
- Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Differential Privacy. Vol. 9. Now Publishers, Inc. 211–407 pages.Google Scholar
- Marco Gaboardi, Andreas Haeberlen, Justin Hsu, Arjun Narayan, and Benjamin C. Pierce. 2013. Linear dependent types for differential privacy. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Rome, Italy. 357–370. http://dl.acm.org/citation.cfm?id=2429113 Google Scholar
Digital Library
- Susanne Graf and Hassen Saïdi. 1997. Construction of abstract state graphs with PVS. In International Conference on Computer Aided Verification (CAV), Haifa, Israel . Springer-Verlag, 72–83. Google Scholar
Cross Ref
- Sergey Grebenshchikov, Ashutosh Gupta, Nuno P. Lopes, Corneliu Popeea, and Andrey Rybalchenko. 2012a. HSF (C): A software verifier based on Horn clauses. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), Tallinn, Estonia . Springer-Verlag, 549–551. Google Scholar
Digital Library
- Sergey Grebenshchikov, Nuno P. Lopes, Corneliu Popeea, and Andrey Rybalchenko. 2012b. Synthesizing Software Verifiers from Proof Rules. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Beijing, China . 405–416. Google Scholar
Digital Library
- Sumit Gulwani, Susmit Jha, Ashish Tiwari, and Ramarathnam Venkatesan. 2011. Synthesis of loop-free programs. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), San Jose, California . Google Scholar
Digital Library
- Arie Gurfinkel, Temesghen Kahsai, and Jorge A. Navas. 2015. SeaHorn: A Framework for Verifying C Programs (Competition Contribution). In International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), London, England . 447–450. Google Scholar
Digital Library
- Hossein Hojjat, Filip Konečn`y, Florent Garnier, Radu Iosif, Viktor Kuncak, and Philipp Rümmer. 2012. A verification toolkit for numerical transition systems. In International Symposium on Formal Methods (FM), Paris, France. Springer-Verlag, 247–251. Google Scholar
Cross Ref
- Justin Hsu. 2017. Probabilistic Couplings for Probabilistic Reasoning. Ph.D. Dissertation. University of Pennsylvania. arXiv: cs.LO/1710.09951 https://arxiv.org/abs/1710.09951Google Scholar
- Boris Köpf and Andrey Rybalchenko. 2010. Approximation and randomization for quantitative information-flow analysis. In IEEE Computer Security Foundations Symposium (CSF), Edinburgh, Scotland. 3–14. Google Scholar
Digital Library
- Shuvendu K. Lahiri, Robert Nieuwenhuis, and Albert Oliveras. 2006. SMT techniques for fast predicate abstraction. In International Conference on Computer Aided Verification (CAV), Seattle, Washington . Springer-Verlag, 424–437. Google Scholar
Digital Library
- Torgny Lindvall. 2002. Lectures on the coupling method. Courier Corporation.Google Scholar
- Min Lyu, Dong Su, and Ninghui Li. 2017. Understanding the Sparse Vector Technique for Differential Privacy. In International Conference on Very Large Data Bases (VLDB), Munich, Germany , Vol. 10. 637–648. http://arxiv.org/abs/1603.01699Google Scholar
- Kenneth L. McMillan and Andrey Rybalchenko. 2013. Solving constrained Horn clauses using interpolation. Technical Report MSR-TR-2013-6. Microsoft Research.Google Scholar
- Frank McSherry and Kunal Talwar. 2007. Mechanism Design via Differential Privacy. In IEEE Symposium on Foundations of Computer Science (FOCS), Providence, Rhode Island . 94–103. Google Scholar
Digital Library
- George C. Necula. 2000. Translation validation for an optimizing compiler. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Vancouver, British Columbia , Vol. 35. 83–94. Google Scholar
Digital Library
- Federico Olmedo. 2014. Approximate relational reasoning for probabilistic programs. Ph.D. Dissertation. Universidad Politécnica de Madrid. http://oa.upm.es/23088/1/FEDERICO_OLMEDO.pdfGoogle Scholar
- Amir Pnueli, Michael Siegel, and Eli Singerman. 1998. Translation validation. International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), Lisbon, Portugal (1998), 151–166.Google Scholar
Digital Library
- Jason Reed and Benjamin C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Differential Privacy. In ACM SIGPLAN International Conference on Functional Programming (ICFP), Baltimore, Maryland. http://dl.acm.org/ citation.cfm?id=1863568Google Scholar
- Tetsuya Sato. 2016. Approximate Relational Hoare Logic for Continuous Random Samplings. In Conference on the Mathematical Foundations of Programming Semantics (MFPS), Pittsburgh, Pennsylvania (Electronic Notes in Theoretical Computer Science) , Vol. 325. Elsevier, 277–298. https://arxiv.org/abs/1603.01445Google Scholar
- Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. 2006. Combinatorial sketching for finite programs. In Asian Symposium on Programming Languages and Systems (APLAS), San Jose, California. 404–415. Google Scholar
Digital Library
- Marcelo Sousa and Isil Dillig. 2016. Cartesian Hoare Logic for Verifying k -safety Properties. ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Santa Barbara, California , 57–69. Google Scholar
Digital Library
- Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Foster. 2010. From Program Verification to Program Synthesis. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Madrid, Spain . New York, NY, USA, 313–326. Google Scholar
Digital Library
- Tachio Terauchi and Alexander Aiken. 2005. Secure information flow as a safety problem. In International Symposium on Static Analysis (SAS), London, England (Lecture Notes in Computer Science) , Vol. 3672. Springer-Verlag, 352–367. Google Scholar
Digital Library
- Daniel Winograd-Cort, Andreas Haeberlen, Aaron Roth, and Benjamin C. Pierce. 2017. A framework for adaptive differential privacy. In ACM SIGPLAN International Conference on Functional Programming (ICFP), Oxford, England. Google Scholar
Digital Library
- Danfeng Zhang and Daniel Kifer. 2017. LightDP: Towards Automating Differential Privacy Proofs. In ACM SIGPLAN–SIGACT Symposium on Principles of Programming Languages (POPL), Paris, France . 888–901. https://arxiv.org/abs/1607.08228Google Scholar
Index Terms
Synthesizing coupling proofs of differential privacy
Recommendations
LightDP: towards automating differential privacy proofs
POPL '17: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming LanguagesThe growing popularity and adoption of differential privacy in academic and industrial settings has resulted in the development of increasingly sophisticated algorithms for releasing information while preserving privacy. Accompanying this phenomenon is ...
Probabilistic Relational Reasoning for Differential Privacy
Differential privacy is a notion of confidentiality that allows useful computations on sensible data while protecting the privacy of individuals. Proving differential privacy is a difficult and error-prone task that calls for principled approaches and ...
Probabilistic relational reasoning for differential privacy
POPL '12: Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languagesDifferential privacy is a notion of confidentiality that protects the privacy of individuals while allowing useful computations on their private data. Deriving differential privacy guarantees for real programs is a difficult and error-prone task that ...






Comments