skip to main content
research-article
Open Access
Artifacts Available
Artifacts Evaluated & Functional

Fuzzi: a three-level logic for differential privacy

Published:26 July 2019Publication History
Skip Abstract Section

Abstract

Curators of sensitive datasets sometimes need to know whether queries against the data are differentially private. Two sorts of logics have been proposed for checking this property: (1) type systems and other static analyses, which fully automate straightforward reasoning with concepts like “program sensitivity” and “privacy loss,” and (2) full-blown program logics such as apRHL (an approximate, probabilistic, relational Hoare logic), which support more flexible reasoning about subtle privacy-preserving algorithmic techniques but offer only minimal automation.

We propose a three-level logic for differential privacy in an imperative setting and present a prototype implementation called Fuzzi. Fuzzi’s lowest level is a general-purpose logic; its middle level is apRHL; and its top level is a novel sensitivity logic adapted from the linear-logic-inspired type system of Fuzz, a differentially private functional language. The key novelty is a high degree of integration between the sensitivity logic and the two lower-level logics: the judgments and proofs of the sensitivity logic can be easily translated into apRHL; conversely, privacy properties of key algorithmic building blocks can be proved manually in apRHL and the base logic, then packaged up as typing rules that can be applied by a checker for the sensitivity logic to automatically construct privacy proofs for composite programs of arbitrary size.

We demonstrate Fuzzi’s utility by implementing four different private machine-learning algorithms and showing that Fuzzi’s checker is able to derive tight sensitivity bounds.

Skip Supplemental Material Section

Supplemental Material

a93-zhang.webm

References

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16). ACM, New York, NY, USA, 308–318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Amal Ahmed. 2006. Step-Indexed Syntactic Logical Relations for Recursive and Quantified Types. In Programming Languages and Systems, Peter Sestoft (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 69–83. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Aws Albarghouthi and Justin Hsu. 2017. Synthesizing Coupling Proofs of Differential Privacy. Proc. ACM Program. Lang. 2, POPL, Article 58 (Dec. 2017), 30 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Andrew W. Appel and David McAllester. 2001. An Indexed Model of Recursive Types for Foundational Proof-carrying Code. ACM Trans. Program. Lang. Syst. 23, 5 (Sept. 2001), 657–683. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Andrew W. Appel, Paul-André Melliès, Christopher D. Richards, and Jérôme Vouillon. 2007. A Very Modal Model of a Modern, Major, General Type System. In Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’07). ACM, New York, NY, USA, 109–122. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Apple. 2017. Apple Differential Privacy Whitepaper. https://images.apple.com/privacy/docs/Differential_Privacy_Overview. pdfGoogle ScholarGoogle Scholar
  7. Patrick Bahr and Tom Hvitved. 2011. Compositional Data Types. In Proceedings of the Seventh ACM SIGPLAN Workshop on Generic Programming (WGP ’11). ACM, New York, NY, USA, 83–94. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Gilles Barthe, Marco Gaboardi, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2016. Proving Differential Privacy via Probabilistic Couplings. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS ’16). ACM, New York, NY, USA, 749–758. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Nick Benton. 2004. Simple Relational Correctness Proofs for Static Analyses and Program Transformations. In Proceedings of the 31st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’04). ACM, New York, NY, USA, 14–25. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. T.-H. Hubert Chan, Elaine Shi, and Dawn Song. 2011. Private and Continual Release of Statistics. ACM Trans. Inf. Syst. Secur. 14, 3, Article 26 (Nov. 2011), 24 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Yan Chen and Ashwin Machanavajjhala. 2015. On the Privacy Properties of Variants on the Sparse Vector Technique. CoRR abs/1508.07306 (2015). arXiv: 1508.07306 http://arxiv.org/abs/1508.07306Google ScholarGoogle Scholar
  12. The Coq Development Team. 2018. The Coq Proof Assistant Reference Manual, version 8.8. http://coq.inria.frGoogle ScholarGoogle Scholar
  13. Dua Dheeru and Efi Karra Taniskidou. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/mlGoogle ScholarGoogle Scholar
  14. Zeyu Ding, Yuxin Wang, Guanhong Wang, Danfeng Zhang, and Daniel Kifer. 2018. Detecting Violations of Differential Privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS ’18). ACM, New York, NY, USA, 475–489. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating Noise to Sensitivity in Private Data Analysis. In Proceedings of the Third Conference on Theory of Cryptography (TCC’06). Springer-Verlag, Berlin, Heidelberg, 265–284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Differential Privacy. Found. Trends Theor. Comput. Sci. 9, 3–4 (Aug. 2014), 211–407. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Cynthia Dwork, Guy Rothblum, and Salil Vadhan. 2010. Boosting and Differential Privacy. In Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS ‘10). IEEE, IEEE, Las Vegas, NV, 51–60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. The EasyCrypt Development Team. 2018. EasyCrypt Reference Manual, version 1.x. https://www.easycrypt.info/ documentation/refman.pdfGoogle ScholarGoogle Scholar
  19. R. A. Fisher. 1936. The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics 7, 7 (1936), 179–188.Google ScholarGoogle ScholarCross RefCross Ref
  20. Dan Frumin, Robbert Krebbers, and Lars Birkedal. 2018. ReLoC: A Mechanised Relational Logic for Fine-Grained Concurrency. In Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS ’18). ACM, New York, NY, USA, 442–451. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Marco Gaboardi, Andreas Haeberlen, Justin Hsu, Arjun Narayan, and Benjamin C. Pierce. 2013. Linear Dependent Types for Differential Privacy. In Proceedings of the 40th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’13). ACM, New York, NY, USA, 357–370. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Marco Gaboardi, James Honaker, Gary King, Kobbi Nissim, Jonathan Ullman, and Salil P. Vadhan. 2016. PSI ( Ψ): a Private data Sharing Interface. CoRR abs/1609.04340 (2016). arXiv: 1609.04340 http://arxiv.org/abs/1609.04340Google ScholarGoogle Scholar
  23. Andreas Haeberlen, Benjamin C. Pierce, and Arjun Narayan. 2011. Differential Privacy Under Fire. In Proceedings of the 20th USENIX Conference on Security (SEC’11). USENIX Association, Berkeley, CA, USA, 33–33. http://dl.acm.org/citation.cfm? id=2028067.2028100 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. C. A. R. Hoare. 1969. An Axiomatic Basis for Computer Programming. Commun. ACM 12, 10 (Oct. 1969), 576–580. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Justin Hsu. 2018. Private Communication.Google ScholarGoogle Scholar
  26. Noah Johnson, Joseph P. Near, and Dawn Song. 2018. Towards Practical Differential Privacy for SQL Queries. Proc. VLDB Endow. 11, 5 (Jan. 2018), 526–539. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ralf Jung, Jacques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer. 2017. RustBelt: Securing the Foundations of the Rust Programming Language. Proc. ACM Program. Lang. 2, POPL, Article 66 (Dec. 2017), 34 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov 1998), 2278–2324.Google ScholarGoogle ScholarCross RefCross Ref
  29. Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/. (2010). http://yann.lecun.com/exdb/mnist/Google ScholarGoogle Scholar
  30. Min Lyu, Dong Su, and Ninghui Li. 2016. Understanding the Sparse Vector Technique for Differential Privacy. CoRR abs/1603.01699 (2016). arXiv: 1603.01699 http://arxiv.org/abs/1603.01699Google ScholarGoogle Scholar
  31. H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning Differentially Private Recurrent Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=BJ0hF1Z0bGoogle ScholarGoogle Scholar
  32. Frank McSherry and Ilya Mironov. 2009. Differentially Private Recommender Systems: Building Privacy into the Netflix Prize Contenders. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’09). ACM, New York, NY, USA, 627–636. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Frank D. McSherry. 2009. Privacy Integrated Queries: An Extensible Platform for Privacy-preserving Data Analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data (SIGMOD ’09). ACM, New York, NY, USA, 19–30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Microsoft. 2017. Collecting telemetry data privately. https://www.microsoft.com/en-us/research/blog/collecting-telemetrydata-privately/Google ScholarGoogle Scholar
  35. Ilya Mironov. 2012. On Significance of the Least Significant Bits for Differential Privacy. In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS ’12). ACM, New York, NY, USA, 650–661. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Aref N. Dajani, Amy D. Lauger, Phyllis E. Singer, Daniel Kifer, Jerome P. Reiter, Ashwin Machanavajjhala, Simson L. Garfinkel, Scot A. Dahl, Matthew Graham, Vishesh Karwa, Hang Kim, Philip Leclerc, Ian M. Schmutte, William N. Sexton, Lars Villhuber, and John M. Abowd. 2017. The modernization of statistical disclosure limitation at the U.S. Census Bureau. (September 2017). https://www2.census.gov/cac/sac/meetings/2017-09/statistical-disclosure-limitation.pdf {Online; posted September-2017}.Google ScholarGoogle Scholar
  37. Arjun Narayan and Andreas Haeberlen. 2012. DJoin: Differentially Private Join Queries over Distributed Databases. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI’12). USENIX Association, Berkeley, CA, USA, 149–162. http://dl.acm.org/citation.cfm?id=2387880.2387895 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Joseph P. Near, David Darais, Tim Stevens, Paranav Gaddamadugu, Lun Wang, Neel Somani, Mu Zhang, Nikhil Sharma, Alex Shan, and Dawn Song. 2019. (2019). http://david.darais.com/assets/papers/duet/duet.pdfGoogle ScholarGoogle Scholar
  39. Travis E. Oliphant. 2015. Guide to NumPy (2nd ed.). CreateSpace Independent Publishing Platform, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. 2016. Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. arXiv:1610.05755 {cs, stat} (Oct. 2016). http://arxiv.org/abs/1610. 05755 arXiv: 1610.05755.Google ScholarGoogle Scholar
  41. Jason Reed and Benjamin C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Differential Privacy. SIGPLAN Not. 45, 9 (Sept. 2010), 157–168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. John C. Reynolds. 2002. Separation Logic: A Logic for Shared Mutable Data Structures. In Proceedings of the 17th Annual IEEE Symposium on Logic in Computer Science (LICS ’02). IEEE Computer Society, Washington, DC, USA, 55–74. http: //dl.acm.org/citation.cfm?id=645683.664578 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Ryan M Rogers, Aaron Roth, Jonathan Ullman, and Salil Vadhan. 2016. Privacy Odometers and Filters: Pay-as-you-Go Composition. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 1921–1929. http://papers.nips.cc/paper/6170-privacy-odometers-andfilters-pay-as-you-go-composition.pdf Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Tetsuya Sato. 2016. Approximate Relational Hoare Logic for Continuous Random Samplings. Electronic Notes in Theoretical Computer Science 325 (2016), 277 – 298.Google ScholarGoogle ScholarCross RefCross Ref
  45. Tetsuya Sato, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Shin-ya Katsumata. 2019. Approximate Span Liftings. CoRR abs/1710.09010 (2019). arXiv: 1710.09010 http://arxiv.org/abs/1710.09010Google ScholarGoogle Scholar
  46. Wouter Swierstra. 2008. Data Types à La Carte. J. Funct. Program. 18, 4 (July 2008), 423–436. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Daniel Winograd-Cort, Andreas Haeberlen, Aaron Roth, and Benjamin C. Pierce. 2017. A Framework for Adaptive Differential Privacy. Proc. ACM Program. Lang. 1, ICFP, Article 10 (Aug. 2017), 29 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Danfeng Zhang and Daniel Kifer. 2017. LightDP: Towards Automating Differential Privacy Proofs. SIGPLAN Not. 52, 1 (Jan. 2017), 888–901.Google ScholarGoogle Scholar
  49. Dan Zhang, Ryan McKenna, Ios Kotsogiannis, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. 2018. Ektelo: A Framework for Defining Differentially-Private Computations. In SIGMOD Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Fuzzi: a three-level logic for differential privacy

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!