skip to main content
research-article
Free Access

Probabilistic Relational Reasoning for Differential Privacy

Published:01 November 2013Publication History
Skip Abstract Section

Abstract

Differential privacy is a notion of confidentiality that allows useful computations on sensible data while protecting the privacy of individuals. Proving differential privacy is a difficult and error-prone task that calls for principled approaches and tool support. Approaches based on linear types and static analysis have recently emerged; however, an increasing number of programs achieve privacy using techniques that fall out of their scope. Examples include programs that aim for weaker, approximate differential privacy guarantees and programs that achieve differential privacy without using any standard mechanisms. Providing support for reasoning about the privacy of such programs has been an open problem.

We report on CertiPriv, a machine-checked framework for reasoning about differential privacy built on top of the Coq proof assistant. The central component of CertiPriv is a quantitative extension of probabilistic relational Hoare logic that enables one to derive differential privacy guarantees for programs from first principles. We demonstrate the applicability of CertiPriv on a number of examples whose formal analysis is out of the reach of previous techniques. In particular, we provide the first machine-checked proofs of correctness of the Laplacian, Gaussian, and exponential mechanisms and of the privacy of randomized and streaming algorithms from the literature.

References

  1. Jose Bacelar Almeida, Manuel Barbosa, Endre Bangerter, Gilles Barthe, Stephan Krenn, and Santiago Zanella-Beguelin. 2012. Full proof cryptography: Verifiable compilation of efficient zero-knowledge protocols. In Proceedngs of the 19th ACM Conference on Computer and Communications Security (CCS’12). ACM Press, New York, 488--500. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Torben Amtoft and Anindya Banerjee. 2004. Information flow analysis in logical form. In Proceedings of the 11th International Symposium on Static Analysis (SAS’04). Lecture Notes in Computer Science, vol. 3148, Springer, 100--115.Google ScholarGoogle ScholarCross RefCross Ref
  3. Torben Amtoft, Sruthi Bandhakavi, and Anindya Banerjee. 2006. A logic for information flow in objectoriented programs. In Proceedings of the 33rd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’06). ACM Press, New York, 91--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Philippe Audebaud and Christine Paulin-Mohring. 2009. Proofs of randomized algorithms in coq. Sci. Comput. Program.74, 8, 568--589. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Michael Backes, Boris Kopf, and Andrey Rybalchenko. 2009. Automatic discovery and quantification of information leaks. In Proceedings of the 30th IEEE Symposium on Security and Privacy (S&P’’09). IEEE Computer Society, Los Alamitos, CA, 141--153. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. David Baelde, Pierre Courtieu, David Gross-Amblard, and Christine Paulin-Mohring. 2012. Towards provably robust watermarking. In Proceedings of the 3rd International Conference on Interactive Theorem Proving (ITP’12). Lecture Notes in Computer Science, vol. 7406, Springer, 201--216.Google ScholarGoogle ScholarCross RefCross Ref
  7. Gilles Barthe and Boris Kopf. 2011. Information-theoretic bounds for differentially private mechanisms. In Proceedings of the 24th IEEE Computer Security Foundations Symposium (CSF’11). IEEE Computer Society, Los Alamitos, CA, 191--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Gilles Barthe and Federico Olmedo. 2013. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In Proceedings of the 40th International Colloquium on Automata, Languages and Programming (ICALP’13). Lecture Notes in Computer Science, vol. 7966, Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Gilles Barthe, Pedro D’Argenio, and Tamara Rezk. 2004. Secure information flow by self-composition. In Proceedings of the 17th IEEE Workshop on Computer Security Foundations (CSFW’04). IEEE Computer Society, Los Alamitos, CA, 100--114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Gilles Barthe, Benjamin Gregoire, and Santiago Zanella-Beguelin. 2009. Formal certification of code-based cryptographic proofs. In Proceedings of the 36th ACMSIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’09). ACM Press, New York, 90--101. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Gilles Barthe, Benjamin Gregoire, Sylvain Heraud, and Santiago Zanella-Beguelin. 2011a. Computer-aided security proofs for the working cryptographer. In Advances in Cryptology -- CRYPTO 2011. Lecture Notes in Computer Science, vol. 6841, Springer, 71--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Gilles Barthe, Juan Manuel Crespo, and Cesar Kunz. 2011b. Relational verification using product progams. In Proceedings of the 17th International Symposium on Formal Methods (FM’11). Lecture Notes in Computer Science, vol. 6664, Springer, 200--214. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Gilles Barthe, Boris Kopf, Federico Olmedo, and Santiago Zanella-Beguelin. 2012. Probabilistic relational reasoning for differential privacy. In Proceedings of the 39th ACM SIGPLAN SIGACT Symposium on Principles of Programming Languages (POPL’12). ACM Press, New York, 97--110. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Gilles Barthe, George Danezis, Benjamin Gregoire, Cesar Kunz, and Santiago Zanella-Beguelin. 2013. Verified computational differential privacy with applications to smart metering. In Proceedings of the 26th IEEE Computer Security Foundations Symposium (CSF’13). IEEE Computer Society, Los Alamitos, CA. To appear. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Amos Beimel, Kobbi Nissim, and Eran Omri. 2008. Distributed private data analysis: Simultaneously solving how and what. In Advances in Cryptology -- CRYPTO 2008. Lecture Notes in Computer Science, vol. 5157, Springer, 451--468. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Nick Benton. 2004. Simple relational correctness proofs for static analyses and program transformations. In Proceedings of the 31st ACMSIGPLAN-SIGACT Symposium on Principles of Programming Languages, (POPL’04). ACM Press, New York, 14--25. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Jacob Burnim and Koushik Sen. 2009. Asserting and checking determinism for multithreaded programs. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/SIGSOFT FSE’09). ACM Press, New York, 3--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Rohit Chadha, Luis Cruz-Filipe, Paulo Mateus, and Amilcar Sernadas. 2007. Reasoning about probabilistic sequential programs. Theor. Comput. Sci. 379, 1--2, 142--165. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Terry-H. Hubert Chan, Elaine Shi, and Dawn Song. 2010. Private and continual release of statistics. In Proceedings of the 37th International Colloquium on Automata, Languages and Programming (ICALP’10). Lecture Notes in Computer Science, vol. 6199, Springer, 405--417. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Swarat Chaudhuri, Sumit Gulwani, Roberto Lublinerman, and Sara Navidpour. 2011. Proving programs robust. In Proceedings of the 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering and the 13th European Software Engineering Conference (ESEC/FSE’11). ACM Press, New York, 102--112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. David Clark, Sebastian Hunt, and Pasquale Malacaria. 2007. A static analysis for quantifying information flow in a simple imperative language. J. Comput. Secur. 15, 3, 321--371. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Michael R. Clarkson and Fred B. Schneider. 2010. Hyperproperties. J. Comput. Secur. 18, 6, 1157--1210. Google ScholarGoogle ScholarCross RefCross Ref
  23. Aaron R. Coble. 2008. Formalized information-theoretic proofs of privacy using the hol4 theorem-prover. In Proceedings of the 8th International Symposium on Privacy Enhancing Technologies (PETS’08). Lecture Notes in Computer Science, vol. 5134, Springer, 77--98. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Aaron R. Coble. 2010. Anonymity, information, and machine-assisted proof. Tech. rep. UCAMCL-TR-785. University of Cambridge, Computer Laboratory. http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-785.pdf.Google ScholarGoogle Scholar
  25. The Coq Development Team. 2010. The coq proof assistant reference manual version 8.3. http://coq.inria.fr.Google ScholarGoogle Scholar
  26. Jerry Den Hartog. 1999. Verifying probabilistic programs using a hoare like logic. In Advances in Computing Science -- ASIAN 1999. Lecture Notes in Computer Science, vol. 1742, Springer, 113--125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Josee Desharnais, Francois Laviolette, and Mathieu Tracol. 2008. Approximate analysis of probabilistic processes: Logic, simulation and games. In Proceedings of the 5th International Conference on Quantitative Evaluation of Systems (QEST’08). IEEE Computer Society, Los Alamitos, CA, 264--273. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Alessandra Di Pierro, Chris Hankin, and Herbert Wiklicky. 2004. Approximate non-interference. J. Comput. Secur. 12, 1, 37--82. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Cynthia Dwork. 2008. Differential privacy: A survey of results. In Theory and Applications of Models of Computation. Lecture Notes in Computer Science, vol. 4978, Springer, 1--19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Cynthia Dwork. 2011. A firm foundation for private data analysis. Comm. ACM 54, 1, 86--95. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Cynthia Dwork, Krishnaram Kenthapadi, Frank Mcsherry, Ilya Mironov, and Moni Naor. 2006a. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology -- EUROCRYPT 2006. Lecture Notes in Computer Science, vol. 4004, Springer, 486--503. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, and Adam Smith. 2006b. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Theory of Cryptography Conference (TCC’06). Lecture Notes in Computer Science, vol. 3876, Springer, 265--284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. 2010. Boosting and differential privacy. In Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS’10). IEEE Computer Society, Los Alamitos, CA, 51--60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Yishai A. Feldman and David Harel. 1984. A probabilistic dynamic logic. J. Comput. Syst. Sci 28, 2, 193--215.Google ScholarGoogle ScholarCross RefCross Ref
  35. Donald Goldfarb, Zhiying Jin, and James B. Orlin, J. B. 1997. Polynomial-time highest-gain augmenting path algorithms for the generalized circulation problem. Math. Oper. Res. 22, 4, 793--802. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Anupam Gupta, Katrina Ligett, Frank Mcsherry, Aaron Roth, and Kunal Talwar. 2010. Differentially private combinatorial optimization. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’10). SIAM, 1106--1125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Andreas Haeberlen, Benjamin C. Pierce, and Arjun Narayan. 2011. Differential privacy under fire. In Proceedings of the 20th USENIX Security Symposium. USENIX Association, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Joe Hurd. 2003. Formal verification of probabilistic algorithms. Tech. rep. UCAM-CL-TR-566, University of Cambridge, Computer Laboratory. http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-566.pdf.Google ScholarGoogle Scholar
  39. Joe Hurd, Annabelle Mciver, and Carroll Morgan. 2005. Probabilistic guarded commands mechanized in hol. Theor. Comput. Sci. 346, 1, 96--112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Claire Jones. 1993. Probabilistic non-determinism. Ph.D. dissertation, University of Edinburgh. http://www.lfcs.inf.ed.ac.uk/reports/90/ECS-LFCS-90-105/. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Bengt Jonsson, Wang Yi, and Kim G. Larsen. 2001. Probabilistic extensions of process algebras. In Handbook of Process Algebra. Elsevier, Amsterdam, 685--710.Google ScholarGoogle Scholar
  42. Shiva Prasad Kasiviswanathan and Adam Smith. 2008. A note on differential privacy: Defining resistance to arbitrary side information. Cryptology ePrint archive, report 2008/144.Google ScholarGoogle Scholar
  43. Daniel Kifer and Ashwin Machanavajjhala. 2011. No free lunch in data privacy. In Proceedings of the International Conference on Management of Data (SIGMOD’11). ACM Press, New York, 193--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Dexter Kozen. 1985. A probabilistic pdl. J. Comput. Syst. Sci. 30, 2,162--178.Google ScholarGoogle Scholar
  45. Eugene L. Lawler. 1976. Combinatorial Optimization: Networks and Matroids. Holt, Rinehart and Winston, New York.Google ScholarGoogle Scholar
  46. Frank Mcsherry. 2009. Privacy integrated queries: An extensible platform for privacy-preserving data analysis. In Proceedings of the 35th SIGMOD International Conference on Management of Data (SIGMOD’09). ACM Press, New York, 19--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Frank Mcsherry and Kunal Talwar. 2007. Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07). IEEE Computer Society, Los Alamitos, CA, 94--103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Tarek Mhamdi, Osman Hasan, and Sofiene Tahar. 2010. On the formalization of the lebesgue integration theory in hol. In Proceedings of the 1st International Conference on Interactive Theorem Proving (ITP’10). Lecture Notes in Computer Science, vol. 6172, Springer, 387--402. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Tarek Mhamdi, Osman Hasan, and Sofiene Tahar. 2011. Formalization of entropy measures in hol. In Proceedings of the 2nd International Conference on Interactive Theorem Proving (ITP’11). Lecture Notes in Computer Science, vol. 6898, Springer, 233--248. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Ilya Mironov, Omkant Pandey, Omer Reingold, and Salil Vadhan. 2009. Computational differential privacy. In Advances in Cryptology -- CRYPTO 2009. Lecture Notes in Computer Science, vol. 5677, Springer, 126--142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Carroll Morgan, Annabelle Mciver, and Karen Seidel. 1996. Probabilistic predicate transformers. ACM Trans. Program. Lang. Syst. 18, 3, 325--353. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Katta G. Murty. 1992. Network Programming. Prentice Hall, Englewood Cliffs, NJ. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Aleksandar Nikolov, Kunal Talwar, and Li Zhang. 2012. The geometry of differential privacy: The sparse and approximate cases. In Proceedings of the ACM Symposium on Theory of Computing. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Leonard Pitt. 1985. A simple probabilistic approximation algorithm for vertex cover. Tech. rep. TR-404, Yale University.Google ScholarGoogle Scholar
  55. Norman Ramsey and Avi Pfeffer. 2002. Stochastic lambda calculus and monads of probability distributions. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’02). ACM Press, New York, 154--165. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Jason Reed and Benjamin C. Pierce. 2010. Distance makes the types grow stronger: A calculus for differential privacy. In Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming (ICFP’10). ACM Press, New York, 157--168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. John H. Reif. 1980. Logics for probabilistic programming (extended abstract). In Proceedings of the 12th Annual ACM Symposium on Theory of Computing (STOC’80). ACM Press, New York, 8--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Indrajit Roy, Srinath T. V. Setty, Ann Kilzer, Vitaly Shmatikov, and Emmett Witchel. 2010. Airavat: Security and privacy for mapreduce. In Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation (NSDI’10). USENIX Association, Berkeley, CA, 297--312. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Andrei Sabelfeld and David Sands. 2000. Probabilistic noninterference for multi-threaded programs. In Proceedings of the 13th IEEE Workshop on Computer Security Foundations (CSFW’00). IEEE Computer Society, Los Alamitos, CA, 200--215. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Roberto Segala and Andrea Turrini. 2007. Approximated computationally bounded simulation relations for probabilistic automata. In Proceedings of the 20th IEEE Computer Security Foundations Symposium (CSF’07). IEEE Computer Society, Los Alamitos, 140--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Eva Tardos and Kevin Wayne. 1998. Simple generalized maximum flow algorithms. In Integer Programming and Combinatorial Optimization. Lecture Notes in Computer Science, vol. 1412, Springer, 310--324.Google ScholarGoogle Scholar
  62. Tachio Terauchi and Alex Aiken. 2005. Secure information flow as a safety problem. In Proceedings of the 12th International Symposium on Static Analysis (SAS’05). Lecture Notes in Computer Science, vol. 3672, Springer, 352--367. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Michael Carl Tschantz, Dilsun Kaynar, and Anupam Datta. 2011. Formal verification of differential privacy for interactive systems. Electron. Notes Theor. Comput. Sci. 276, 61--79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Anna Zaks and Amir Pnueli. 2008. CoVaC: Compiler validation by program analysis of the cross-product. In Proceedings of the 15th International Symposium on Formal Methods (FM’08). Lecture Notes in Computer Science, vol. 5014, Springer, 35--51. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Probabilistic Relational Reasoning for Differential Privacy

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM Transactions on Programming Languages and Systems
              ACM Transactions on Programming Languages and Systems  Volume 35, Issue 3
              November 2013
              156 pages
              ISSN:0164-0925
              EISSN:1558-4593
              DOI:10.1145/2542180
              Issue’s Table of Contents

              Copyright © 2013 ACM

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 1 November 2013
              • Accepted: 1 April 2013
              • Revised: 1 March 2013
              • Received: 1 September 2012
              Published in toplas Volume 35, Issue 3

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article
              • Research
              • Refereed

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!