Abstract
Differential privacy is a notion of confidentiality that allows useful computations on sensible data while protecting the privacy of individuals. Proving differential privacy is a difficult and error-prone task that calls for principled approaches and tool support. Approaches based on linear types and static analysis have recently emerged; however, an increasing number of programs achieve privacy using techniques that fall out of their scope. Examples include programs that aim for weaker, approximate differential privacy guarantees and programs that achieve differential privacy without using any standard mechanisms. Providing support for reasoning about the privacy of such programs has been an open problem.
We report on CertiPriv, a machine-checked framework for reasoning about differential privacy built on top of the Coq proof assistant. The central component of CertiPriv is a quantitative extension of probabilistic relational Hoare logic that enables one to derive differential privacy guarantees for programs from first principles. We demonstrate the applicability of CertiPriv on a number of examples whose formal analysis is out of the reach of previous techniques. In particular, we provide the first machine-checked proofs of correctness of the Laplacian, Gaussian, and exponential mechanisms and of the privacy of randomized and streaming algorithms from the literature.
- Jose Bacelar Almeida, Manuel Barbosa, Endre Bangerter, Gilles Barthe, Stephan Krenn, and Santiago Zanella-Beguelin. 2012. Full proof cryptography: Verifiable compilation of efficient zero-knowledge protocols. In Proceedngs of the 19th ACM Conference on Computer and Communications Security (CCS’12). ACM Press, New York, 488--500. Google Scholar
Digital Library
- Torben Amtoft and Anindya Banerjee. 2004. Information flow analysis in logical form. In Proceedings of the 11th International Symposium on Static Analysis (SAS’04). Lecture Notes in Computer Science, vol. 3148, Springer, 100--115.Google Scholar
Cross Ref
- Torben Amtoft, Sruthi Bandhakavi, and Anindya Banerjee. 2006. A logic for information flow in objectoriented programs. In Proceedings of the 33rd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’06). ACM Press, New York, 91--102. Google Scholar
Digital Library
- Philippe Audebaud and Christine Paulin-Mohring. 2009. Proofs of randomized algorithms in coq. Sci. Comput. Program.74, 8, 568--589. Google Scholar
Digital Library
- Michael Backes, Boris Kopf, and Andrey Rybalchenko. 2009. Automatic discovery and quantification of information leaks. In Proceedings of the 30th IEEE Symposium on Security and Privacy (S&P’’09). IEEE Computer Society, Los Alamitos, CA, 141--153. Google Scholar
Digital Library
- David Baelde, Pierre Courtieu, David Gross-Amblard, and Christine Paulin-Mohring. 2012. Towards provably robust watermarking. In Proceedings of the 3rd International Conference on Interactive Theorem Proving (ITP’12). Lecture Notes in Computer Science, vol. 7406, Springer, 201--216.Google Scholar
Cross Ref
- Gilles Barthe and Boris Kopf. 2011. Information-theoretic bounds for differentially private mechanisms. In Proceedings of the 24th IEEE Computer Security Foundations Symposium (CSF’11). IEEE Computer Society, Los Alamitos, CA, 191--204. Google Scholar
Digital Library
- Gilles Barthe and Federico Olmedo. 2013. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In Proceedings of the 40th International Colloquium on Automata, Languages and Programming (ICALP’13). Lecture Notes in Computer Science, vol. 7966, Springer. Google Scholar
Digital Library
- Gilles Barthe, Pedro D’Argenio, and Tamara Rezk. 2004. Secure information flow by self-composition. In Proceedings of the 17th IEEE Workshop on Computer Security Foundations (CSFW’04). IEEE Computer Society, Los Alamitos, CA, 100--114. Google Scholar
Digital Library
- Gilles Barthe, Benjamin Gregoire, and Santiago Zanella-Beguelin. 2009. Formal certification of code-based cryptographic proofs. In Proceedings of the 36th ACMSIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’09). ACM Press, New York, 90--101. Google Scholar
Digital Library
- Gilles Barthe, Benjamin Gregoire, Sylvain Heraud, and Santiago Zanella-Beguelin. 2011a. Computer-aided security proofs for the working cryptographer. In Advances in Cryptology -- CRYPTO 2011. Lecture Notes in Computer Science, vol. 6841, Springer, 71--90. Google Scholar
Digital Library
- Gilles Barthe, Juan Manuel Crespo, and Cesar Kunz. 2011b. Relational verification using product progams. In Proceedings of the 17th International Symposium on Formal Methods (FM’11). Lecture Notes in Computer Science, vol. 6664, Springer, 200--214. Google Scholar
Digital Library
- Gilles Barthe, Boris Kopf, Federico Olmedo, and Santiago Zanella-Beguelin. 2012. Probabilistic relational reasoning for differential privacy. In Proceedings of the 39th ACM SIGPLAN SIGACT Symposium on Principles of Programming Languages (POPL’12). ACM Press, New York, 97--110. Google Scholar
Digital Library
- Gilles Barthe, George Danezis, Benjamin Gregoire, Cesar Kunz, and Santiago Zanella-Beguelin. 2013. Verified computational differential privacy with applications to smart metering. In Proceedings of the 26th IEEE Computer Security Foundations Symposium (CSF’13). IEEE Computer Society, Los Alamitos, CA. To appear. Google Scholar
Digital Library
- Amos Beimel, Kobbi Nissim, and Eran Omri. 2008. Distributed private data analysis: Simultaneously solving how and what. In Advances in Cryptology -- CRYPTO 2008. Lecture Notes in Computer Science, vol. 5157, Springer, 451--468. Google Scholar
Digital Library
- Nick Benton. 2004. Simple relational correctness proofs for static analyses and program transformations. In Proceedings of the 31st ACMSIGPLAN-SIGACT Symposium on Principles of Programming Languages, (POPL’04). ACM Press, New York, 14--25. Google Scholar
Digital Library
- Jacob Burnim and Koushik Sen. 2009. Asserting and checking determinism for multithreaded programs. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/SIGSOFT FSE’09). ACM Press, New York, 3--12. Google Scholar
Digital Library
- Rohit Chadha, Luis Cruz-Filipe, Paulo Mateus, and Amilcar Sernadas. 2007. Reasoning about probabilistic sequential programs. Theor. Comput. Sci. 379, 1--2, 142--165. Google Scholar
Digital Library
- Terry-H. Hubert Chan, Elaine Shi, and Dawn Song. 2010. Private and continual release of statistics. In Proceedings of the 37th International Colloquium on Automata, Languages and Programming (ICALP’10). Lecture Notes in Computer Science, vol. 6199, Springer, 405--417. Google Scholar
Digital Library
- Swarat Chaudhuri, Sumit Gulwani, Roberto Lublinerman, and Sara Navidpour. 2011. Proving programs robust. In Proceedings of the 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering and the 13th European Software Engineering Conference (ESEC/FSE’11). ACM Press, New York, 102--112. Google Scholar
Digital Library
- David Clark, Sebastian Hunt, and Pasquale Malacaria. 2007. A static analysis for quantifying information flow in a simple imperative language. J. Comput. Secur. 15, 3, 321--371. Google Scholar
Digital Library
- Michael R. Clarkson and Fred B. Schneider. 2010. Hyperproperties. J. Comput. Secur. 18, 6, 1157--1210. Google Scholar
Cross Ref
- Aaron R. Coble. 2008. Formalized information-theoretic proofs of privacy using the hol4 theorem-prover. In Proceedings of the 8th International Symposium on Privacy Enhancing Technologies (PETS’08). Lecture Notes in Computer Science, vol. 5134, Springer, 77--98. Google Scholar
Digital Library
- Aaron R. Coble. 2010. Anonymity, information, and machine-assisted proof. Tech. rep. UCAMCL-TR-785. University of Cambridge, Computer Laboratory. http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-785.pdf.Google Scholar
- The Coq Development Team. 2010. The coq proof assistant reference manual version 8.3. http://coq.inria.fr.Google Scholar
- Jerry Den Hartog. 1999. Verifying probabilistic programs using a hoare like logic. In Advances in Computing Science -- ASIAN 1999. Lecture Notes in Computer Science, vol. 1742, Springer, 113--125. Google Scholar
Digital Library
- Josee Desharnais, Francois Laviolette, and Mathieu Tracol. 2008. Approximate analysis of probabilistic processes: Logic, simulation and games. In Proceedings of the 5th International Conference on Quantitative Evaluation of Systems (QEST’08). IEEE Computer Society, Los Alamitos, CA, 264--273. Google Scholar
Digital Library
- Alessandra Di Pierro, Chris Hankin, and Herbert Wiklicky. 2004. Approximate non-interference. J. Comput. Secur. 12, 1, 37--82. Google Scholar
Digital Library
- Cynthia Dwork. 2008. Differential privacy: A survey of results. In Theory and Applications of Models of Computation. Lecture Notes in Computer Science, vol. 4978, Springer, 1--19. Google Scholar
Digital Library
- Cynthia Dwork. 2011. A firm foundation for private data analysis. Comm. ACM 54, 1, 86--95. Google Scholar
Digital Library
- Cynthia Dwork, Krishnaram Kenthapadi, Frank Mcsherry, Ilya Mironov, and Moni Naor. 2006a. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology -- EUROCRYPT 2006. Lecture Notes in Computer Science, vol. 4004, Springer, 486--503. Google Scholar
Digital Library
- Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, and Adam Smith. 2006b. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Theory of Cryptography Conference (TCC’06). Lecture Notes in Computer Science, vol. 3876, Springer, 265--284. Google Scholar
Digital Library
- Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. 2010. Boosting and differential privacy. In Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS’10). IEEE Computer Society, Los Alamitos, CA, 51--60. Google Scholar
Digital Library
- Yishai A. Feldman and David Harel. 1984. A probabilistic dynamic logic. J. Comput. Syst. Sci 28, 2, 193--215.Google Scholar
Cross Ref
- Donald Goldfarb, Zhiying Jin, and James B. Orlin, J. B. 1997. Polynomial-time highest-gain augmenting path algorithms for the generalized circulation problem. Math. Oper. Res. 22, 4, 793--802. Google Scholar
Digital Library
- Anupam Gupta, Katrina Ligett, Frank Mcsherry, Aaron Roth, and Kunal Talwar. 2010. Differentially private combinatorial optimization. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’10). SIAM, 1106--1125. Google Scholar
Digital Library
- Andreas Haeberlen, Benjamin C. Pierce, and Arjun Narayan. 2011. Differential privacy under fire. In Proceedings of the 20th USENIX Security Symposium. USENIX Association, Berkeley, CA. Google Scholar
Digital Library
- Joe Hurd. 2003. Formal verification of probabilistic algorithms. Tech. rep. UCAM-CL-TR-566, University of Cambridge, Computer Laboratory. http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-566.pdf.Google Scholar
- Joe Hurd, Annabelle Mciver, and Carroll Morgan. 2005. Probabilistic guarded commands mechanized in hol. Theor. Comput. Sci. 346, 1, 96--112. Google Scholar
Digital Library
- Claire Jones. 1993. Probabilistic non-determinism. Ph.D. dissertation, University of Edinburgh. http://www.lfcs.inf.ed.ac.uk/reports/90/ECS-LFCS-90-105/. Google Scholar
Digital Library
- Bengt Jonsson, Wang Yi, and Kim G. Larsen. 2001. Probabilistic extensions of process algebras. In Handbook of Process Algebra. Elsevier, Amsterdam, 685--710.Google Scholar
- Shiva Prasad Kasiviswanathan and Adam Smith. 2008. A note on differential privacy: Defining resistance to arbitrary side information. Cryptology ePrint archive, report 2008/144.Google Scholar
- Daniel Kifer and Ashwin Machanavajjhala. 2011. No free lunch in data privacy. In Proceedings of the International Conference on Management of Data (SIGMOD’11). ACM Press, New York, 193--204. Google Scholar
Digital Library
- Dexter Kozen. 1985. A probabilistic pdl. J. Comput. Syst. Sci. 30, 2,162--178.Google Scholar
- Eugene L. Lawler. 1976. Combinatorial Optimization: Networks and Matroids. Holt, Rinehart and Winston, New York.Google Scholar
- Frank Mcsherry. 2009. Privacy integrated queries: An extensible platform for privacy-preserving data analysis. In Proceedings of the 35th SIGMOD International Conference on Management of Data (SIGMOD’09). ACM Press, New York, 19--30. Google Scholar
Digital Library
- Frank Mcsherry and Kunal Talwar. 2007. Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07). IEEE Computer Society, Los Alamitos, CA, 94--103. Google Scholar
Digital Library
- Tarek Mhamdi, Osman Hasan, and Sofiene Tahar. 2010. On the formalization of the lebesgue integration theory in hol. In Proceedings of the 1st International Conference on Interactive Theorem Proving (ITP’10). Lecture Notes in Computer Science, vol. 6172, Springer, 387--402. Google Scholar
Digital Library
- Tarek Mhamdi, Osman Hasan, and Sofiene Tahar. 2011. Formalization of entropy measures in hol. In Proceedings of the 2nd International Conference on Interactive Theorem Proving (ITP’11). Lecture Notes in Computer Science, vol. 6898, Springer, 233--248. Google Scholar
Digital Library
- Ilya Mironov, Omkant Pandey, Omer Reingold, and Salil Vadhan. 2009. Computational differential privacy. In Advances in Cryptology -- CRYPTO 2009. Lecture Notes in Computer Science, vol. 5677, Springer, 126--142. Google Scholar
Digital Library
- Carroll Morgan, Annabelle Mciver, and Karen Seidel. 1996. Probabilistic predicate transformers. ACM Trans. Program. Lang. Syst. 18, 3, 325--353. Google Scholar
Digital Library
- Katta G. Murty. 1992. Network Programming. Prentice Hall, Englewood Cliffs, NJ. Google Scholar
Digital Library
- Aleksandar Nikolov, Kunal Talwar, and Li Zhang. 2012. The geometry of differential privacy: The sparse and approximate cases. In Proceedings of the ACM Symposium on Theory of Computing. Google Scholar
Digital Library
- Leonard Pitt. 1985. A simple probabilistic approximation algorithm for vertex cover. Tech. rep. TR-404, Yale University.Google Scholar
- Norman Ramsey and Avi Pfeffer. 2002. Stochastic lambda calculus and monads of probability distributions. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’02). ACM Press, New York, 154--165. Google Scholar
Digital Library
- Jason Reed and Benjamin C. Pierce. 2010. Distance makes the types grow stronger: A calculus for differential privacy. In Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming (ICFP’10). ACM Press, New York, 157--168. Google Scholar
Digital Library
- John H. Reif. 1980. Logics for probabilistic programming (extended abstract). In Proceedings of the 12th Annual ACM Symposium on Theory of Computing (STOC’80). ACM Press, New York, 8--13. Google Scholar
Digital Library
- Indrajit Roy, Srinath T. V. Setty, Ann Kilzer, Vitaly Shmatikov, and Emmett Witchel. 2010. Airavat: Security and privacy for mapreduce. In Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation (NSDI’10). USENIX Association, Berkeley, CA, 297--312. Google Scholar
Digital Library
- Andrei Sabelfeld and David Sands. 2000. Probabilistic noninterference for multi-threaded programs. In Proceedings of the 13th IEEE Workshop on Computer Security Foundations (CSFW’00). IEEE Computer Society, Los Alamitos, CA, 200--215. Google Scholar
Digital Library
- Roberto Segala and Andrea Turrini. 2007. Approximated computationally bounded simulation relations for probabilistic automata. In Proceedings of the 20th IEEE Computer Security Foundations Symposium (CSF’07). IEEE Computer Society, Los Alamitos, 140--156. Google Scholar
Digital Library
- Eva Tardos and Kevin Wayne. 1998. Simple generalized maximum flow algorithms. In Integer Programming and Combinatorial Optimization. Lecture Notes in Computer Science, vol. 1412, Springer, 310--324.Google Scholar
- Tachio Terauchi and Alex Aiken. 2005. Secure information flow as a safety problem. In Proceedings of the 12th International Symposium on Static Analysis (SAS’05). Lecture Notes in Computer Science, vol. 3672, Springer, 352--367. Google Scholar
Digital Library
- Michael Carl Tschantz, Dilsun Kaynar, and Anupam Datta. 2011. Formal verification of differential privacy for interactive systems. Electron. Notes Theor. Comput. Sci. 276, 61--79. Google Scholar
Digital Library
- Anna Zaks and Amir Pnueli. 2008. CoVaC: Compiler validation by program analysis of the cross-product. In Proceedings of the 15th International Symposium on Formal Methods (FM’08). Lecture Notes in Computer Science, vol. 5014, Springer, 35--51. Google Scholar
Digital Library
Index Terms
Probabilistic Relational Reasoning for Differential Privacy
Recommendations
Probabilistic relational reasoning for differential privacy
POPL '12: Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languagesDifferential privacy is a notion of confidentiality that protects the privacy of individuals while allowing useful computations on their private data. Deriving differential privacy guarantees for real programs is a difficult and error-prone task that ...
Probabilistic relational reasoning for differential privacy
POPL '12Differential privacy is a notion of confidentiality that protects the privacy of individuals while allowing useful computations on their private data. Deriving differential privacy guarantees for real programs is a difficult and error-prone task that ...
Proving Differential Privacy in Hoare Logic
CSF '14: Proceedings of the 2014 IEEE 27th Computer Security Foundations SymposiumDifferential privacy is a rigorous, worst-case notion of privacy-preserving computation. Informally, a probabilistic program is differentially private if the participation of a single individual in the input database has a limited effect on the program'...






Comments