Abstract
Curators of sensitive datasets sometimes need to know whether queries against the data are differentially private. Two sorts of logics have been proposed for checking this property: (1) type systems and other static analyses, which fully automate straightforward reasoning with concepts like “program sensitivity” and “privacy loss,” and (2) full-blown program logics such as apRHL (an approximate, probabilistic, relational Hoare logic), which support more flexible reasoning about subtle privacy-preserving algorithmic techniques but offer only minimal automation.
We propose a three-level logic for differential privacy in an imperative setting and present a prototype implementation called Fuzzi. Fuzzi’s lowest level is a general-purpose logic; its middle level is apRHL; and its top level is a novel sensitivity logic adapted from the linear-logic-inspired type system of Fuzz, a differentially private functional language. The key novelty is a high degree of integration between the sensitivity logic and the two lower-level logics: the judgments and proofs of the sensitivity logic can be easily translated into apRHL; conversely, privacy properties of key algorithmic building blocks can be proved manually in apRHL and the base logic, then packaged up as typing rules that can be applied by a checker for the sensitivity logic to automatically construct privacy proofs for composite programs of arbitrary size.
We demonstrate Fuzzi’s utility by implementing four different private machine-learning algorithms and showing that Fuzzi’s checker is able to derive tight sensitivity bounds.
Supplemental Material
- Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16). ACM, New York, NY, USA, 308–318. Google Scholar
Digital Library
- Amal Ahmed. 2006. Step-Indexed Syntactic Logical Relations for Recursive and Quantified Types. In Programming Languages and Systems, Peter Sestoft (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 69–83. Google Scholar
Digital Library
- Aws Albarghouthi and Justin Hsu. 2017. Synthesizing Coupling Proofs of Differential Privacy. Proc. ACM Program. Lang. 2, POPL, Article 58 (Dec. 2017), 30 pages. Google Scholar
Digital Library
- Andrew W. Appel and David McAllester. 2001. An Indexed Model of Recursive Types for Foundational Proof-carrying Code. ACM Trans. Program. Lang. Syst. 23, 5 (Sept. 2001), 657–683. Google Scholar
Digital Library
- Andrew W. Appel, Paul-André Melliès, Christopher D. Richards, and Jérôme Vouillon. 2007. A Very Modal Model of a Modern, Major, General Type System. In Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’07). ACM, New York, NY, USA, 109–122. Google Scholar
Digital Library
- Apple. 2017. Apple Differential Privacy Whitepaper. https://images.apple.com/privacy/docs/Differential_Privacy_Overview. pdfGoogle Scholar
- Patrick Bahr and Tom Hvitved. 2011. Compositional Data Types. In Proceedings of the Seventh ACM SIGPLAN Workshop on Generic Programming (WGP ’11). ACM, New York, NY, USA, 83–94. Google Scholar
Digital Library
- Gilles Barthe, Marco Gaboardi, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2016. Proving Differential Privacy via Probabilistic Couplings. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS ’16). ACM, New York, NY, USA, 749–758. Google Scholar
Digital Library
- Nick Benton. 2004. Simple Relational Correctness Proofs for Static Analyses and Program Transformations. In Proceedings of the 31st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’04). ACM, New York, NY, USA, 14–25. Google Scholar
Digital Library
- T.-H. Hubert Chan, Elaine Shi, and Dawn Song. 2011. Private and Continual Release of Statistics. ACM Trans. Inf. Syst. Secur. 14, 3, Article 26 (Nov. 2011), 24 pages. Google Scholar
Digital Library
- Yan Chen and Ashwin Machanavajjhala. 2015. On the Privacy Properties of Variants on the Sparse Vector Technique. CoRR abs/1508.07306 (2015). arXiv: 1508.07306 http://arxiv.org/abs/1508.07306Google Scholar
- The Coq Development Team. 2018. The Coq Proof Assistant Reference Manual, version 8.8. http://coq.inria.frGoogle Scholar
- Dua Dheeru and Efi Karra Taniskidou. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/mlGoogle Scholar
- Zeyu Ding, Yuxin Wang, Guanhong Wang, Danfeng Zhang, and Daniel Kifer. 2018. Detecting Violations of Differential Privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS ’18). ACM, New York, NY, USA, 475–489. Google Scholar
Digital Library
- Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating Noise to Sensitivity in Private Data Analysis. In Proceedings of the Third Conference on Theory of Cryptography (TCC’06). Springer-Verlag, Berlin, Heidelberg, 265–284. Google Scholar
Digital Library
- Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Differential Privacy. Found. Trends Theor. Comput. Sci. 9, 3–4 (Aug. 2014), 211–407. Google Scholar
Digital Library
- Cynthia Dwork, Guy Rothblum, and Salil Vadhan. 2010. Boosting and Differential Privacy. In Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS ‘10). IEEE, IEEE, Las Vegas, NV, 51–60. Google Scholar
Digital Library
- The EasyCrypt Development Team. 2018. EasyCrypt Reference Manual, version 1.x. https://www.easycrypt.info/ documentation/refman.pdfGoogle Scholar
- R. A. Fisher. 1936. The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics 7, 7 (1936), 179–188.Google Scholar
Cross Ref
- Dan Frumin, Robbert Krebbers, and Lars Birkedal. 2018. ReLoC: A Mechanised Relational Logic for Fine-Grained Concurrency. In Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS ’18). ACM, New York, NY, USA, 442–451. Google Scholar
Digital Library
- Marco Gaboardi, Andreas Haeberlen, Justin Hsu, Arjun Narayan, and Benjamin C. Pierce. 2013. Linear Dependent Types for Differential Privacy. In Proceedings of the 40th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’13). ACM, New York, NY, USA, 357–370. Google Scholar
Digital Library
- Marco Gaboardi, James Honaker, Gary King, Kobbi Nissim, Jonathan Ullman, and Salil P. Vadhan. 2016. PSI ( Ψ): a Private data Sharing Interface. CoRR abs/1609.04340 (2016). arXiv: 1609.04340 http://arxiv.org/abs/1609.04340Google Scholar
- Andreas Haeberlen, Benjamin C. Pierce, and Arjun Narayan. 2011. Differential Privacy Under Fire. In Proceedings of the 20th USENIX Conference on Security (SEC’11). USENIX Association, Berkeley, CA, USA, 33–33. http://dl.acm.org/citation.cfm? id=2028067.2028100 Google Scholar
Digital Library
- C. A. R. Hoare. 1969. An Axiomatic Basis for Computer Programming. Commun. ACM 12, 10 (Oct. 1969), 576–580. Google Scholar
Digital Library
- Justin Hsu. 2018. Private Communication.Google Scholar
- Noah Johnson, Joseph P. Near, and Dawn Song. 2018. Towards Practical Differential Privacy for SQL Queries. Proc. VLDB Endow. 11, 5 (Jan. 2018), 526–539. Google Scholar
Digital Library
- Ralf Jung, Jacques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer. 2017. RustBelt: Securing the Foundations of the Rust Programming Language. Proc. ACM Program. Lang. 2, POPL, Article 66 (Dec. 2017), 34 pages. Google Scholar
Digital Library
- Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov 1998), 2278–2324.Google Scholar
Cross Ref
- Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/. (2010). http://yann.lecun.com/exdb/mnist/Google Scholar
- Min Lyu, Dong Su, and Ninghui Li. 2016. Understanding the Sparse Vector Technique for Differential Privacy. CoRR abs/1603.01699 (2016). arXiv: 1603.01699 http://arxiv.org/abs/1603.01699Google Scholar
- H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning Differentially Private Recurrent Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=BJ0hF1Z0bGoogle Scholar
- Frank McSherry and Ilya Mironov. 2009. Differentially Private Recommender Systems: Building Privacy into the Netflix Prize Contenders. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’09). ACM, New York, NY, USA, 627–636. Google Scholar
Digital Library
- Frank D. McSherry. 2009. Privacy Integrated Queries: An Extensible Platform for Privacy-preserving Data Analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data (SIGMOD ’09). ACM, New York, NY, USA, 19–30. Google Scholar
Digital Library
- Microsoft. 2017. Collecting telemetry data privately. https://www.microsoft.com/en-us/research/blog/collecting-telemetrydata-privately/Google Scholar
- Ilya Mironov. 2012. On Significance of the Least Significant Bits for Differential Privacy. In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS ’12). ACM, New York, NY, USA, 650–661. Google Scholar
Digital Library
- Aref N. Dajani, Amy D. Lauger, Phyllis E. Singer, Daniel Kifer, Jerome P. Reiter, Ashwin Machanavajjhala, Simson L. Garfinkel, Scot A. Dahl, Matthew Graham, Vishesh Karwa, Hang Kim, Philip Leclerc, Ian M. Schmutte, William N. Sexton, Lars Villhuber, and John M. Abowd. 2017. The modernization of statistical disclosure limitation at the U.S. Census Bureau. (September 2017). https://www2.census.gov/cac/sac/meetings/2017-09/statistical-disclosure-limitation.pdf {Online; posted September-2017}.Google Scholar
- Arjun Narayan and Andreas Haeberlen. 2012. DJoin: Differentially Private Join Queries over Distributed Databases. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI’12). USENIX Association, Berkeley, CA, USA, 149–162. http://dl.acm.org/citation.cfm?id=2387880.2387895 Google Scholar
Digital Library
- Joseph P. Near, David Darais, Tim Stevens, Paranav Gaddamadugu, Lun Wang, Neel Somani, Mu Zhang, Nikhil Sharma, Alex Shan, and Dawn Song. 2019. (2019). http://david.darais.com/assets/papers/duet/duet.pdfGoogle Scholar
- Travis E. Oliphant. 2015. Guide to NumPy (2nd ed.). CreateSpace Independent Publishing Platform, USA. Google Scholar
Digital Library
- Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. 2016. Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. arXiv:1610.05755 {cs, stat} (Oct. 2016). http://arxiv.org/abs/1610. 05755 arXiv: 1610.05755.Google Scholar
- Jason Reed and Benjamin C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Differential Privacy. SIGPLAN Not. 45, 9 (Sept. 2010), 157–168. Google Scholar
Digital Library
- John C. Reynolds. 2002. Separation Logic: A Logic for Shared Mutable Data Structures. In Proceedings of the 17th Annual IEEE Symposium on Logic in Computer Science (LICS ’02). IEEE Computer Society, Washington, DC, USA, 55–74. http: //dl.acm.org/citation.cfm?id=645683.664578 Google Scholar
Digital Library
- Ryan M Rogers, Aaron Roth, Jonathan Ullman, and Salil Vadhan. 2016. Privacy Odometers and Filters: Pay-as-you-Go Composition. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 1921–1929. http://papers.nips.cc/paper/6170-privacy-odometers-andfilters-pay-as-you-go-composition.pdf Google Scholar
Digital Library
- Tetsuya Sato. 2016. Approximate Relational Hoare Logic for Continuous Random Samplings. Electronic Notes in Theoretical Computer Science 325 (2016), 277 – 298.Google Scholar
Cross Ref
- Tetsuya Sato, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Shin-ya Katsumata. 2019. Approximate Span Liftings. CoRR abs/1710.09010 (2019). arXiv: 1710.09010 http://arxiv.org/abs/1710.09010Google Scholar
- Wouter Swierstra. 2008. Data Types à La Carte. J. Funct. Program. 18, 4 (July 2008), 423–436. Google Scholar
Digital Library
- Daniel Winograd-Cort, Andreas Haeberlen, Aaron Roth, and Benjamin C. Pierce. 2017. A Framework for Adaptive Differential Privacy. Proc. ACM Program. Lang. 1, ICFP, Article 10 (Aug. 2017), 29 pages. Google Scholar
Digital Library
- Danfeng Zhang and Daniel Kifer. 2017. LightDP: Towards Automating Differential Privacy Proofs. SIGPLAN Not. 52, 1 (Jan. 2017), 888–901.Google Scholar
- Dan Zhang, Ryan McKenna, Ios Kotsogiannis, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. 2018. Ektelo: A Framework for Defining Differentially-Private Computations. In SIGMOD Conference. Google Scholar
Digital Library
Index Terms
Fuzzi: a three-level logic for differential privacy
Recommendations
Duet: an expressive higher-order language and linear type system for statically enforcing differential privacy
During the past decade, differential privacy has become the gold standard for protecting the privacy of individuals. However, verifying that a particular program provides differential privacy often remains a manual task to be completed by an expert in ...
Proving Differential Privacy in Hoare Logic
CSF '14: Proceedings of the 2014 IEEE 27th Computer Security Foundations SymposiumDifferential privacy is a rigorous, worst-case notion of privacy-preserving computation. Informally, a probabilistic program is differentially private if the participation of a single individual in the input database has a limited effect on the program'...
Probabilistic relational reasoning for differential privacy
POPL '12: Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languagesDifferential privacy is a notion of confidentiality that protects the privacy of individuals while allowing useful computations on their private data. Deriving differential privacy guarantees for real programs is a difficult and error-prone task that ...






Comments