skip to main content

Deciding accuracy of differential privacy schemes

Published:04 January 2021Publication History
Skip Abstract Section

Abstract

Differential privacy is a mathematical framework for developing statistical computations with provable guarantees of privacy and accuracy. In contrast to the privacy component of differential privacy, which has a clear mathematical and intuitive meaning, the accuracy component of differential privacy does not have a generally accepted definition; accuracy claims of differential privacy algorithms vary from algorithm to algorithm and are not instantiations of a general definition. We identify program discontinuity as a common theme in existing ad hoc definitions and introduce an alternative notion of accuracy parametrized by, what we call, — the of an input x w.r.t.  a deterministic computation f and a distance d, is the minimal distance d(x,y) over all y such that f(y)≠ f(x). We show that our notion of accuracy subsumes the definition used in theoretical computer science, and captures known accuracy claims for differential privacy algorithms. In fact, our general notion of accuracy helps us prove better claims in some cases. Next, we study the decidability of accuracy. We first show that accuracy is in general undecidable. Then, we define a non-trivial class of probabilistic computations for which accuracy is decidable (unconditionally, or assuming Schanuel’s conjecture). We implement our decision procedure and experimentally evaluate the effectiveness of our approach for generating proofs or counterexamples of accuracy for common algorithms from the literature.

References

  1. Erika Ábrahám and Borzoo Bonakdarpour. 2018. HyperPCTL: A Temporal Logic for Probabilistic Hyperproperties. In Quantitative Evaluation of Systems-15th International Conference, QEST 2018, Beijing, China, September 4-7, 2018, Proceedings (Lecture Notes in Computer Science), Annabelle McIver and András Horváth (Eds.), Vol. 11024. Springer, 20-35. https://doi.org/10.1007/978-3-319-99154-2_2 Google ScholarGoogle ScholarCross RefCross Ref
  2. Aws Albarghouthi and Justin Hsu. 2018. Synthesizing coupling proofs of diferential privacy. PACMPL 2, POPL ( 2018 ), 58 : 1-58 : 30. https://doi.org/10.1145/3158146 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Gilles Barthe, Rohit Chadha, Vishal Jagannath, A. Prasad Sistla, and Mahesh Viswanathan. 2020a. Deciding Diefrential Privacy for Programs with Finite Inputs and Outputs. In LICS ' 20 : 35th Annual ACM /IEEE Symposium on Logic in Computer Science, Saarbrücken, Germany, July 8-11, 2020, Holger Hermanns, Lijun Zhang, Naoki Kobayashi, and Dale Miller (Eds.). ACM, 141-154. https://doi.org/10.1145/3373718.3394796 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Gilles Barthe, Rohit Chadha, Vishal Jagannath, A. Prasad Sistla, and Mahesh Viswanathan. 2020b. Deciding Diferential Privacy for Programs with Finite Inputs and Outputs. CORR abs/ 1910.04137 ( 2020 ). arXiv: 1910.04137 http://arxiv.org/ abs/ 1910.04137Google ScholarGoogle Scholar
  5. Gilles Barthe, Rohit Chadha, Paul Krogmeier, A. Prasad Sistla, and Mahesh Viswanathan. 2020c. Deciding Accuracy of Diferential Privacy Schemes. CoRR abs/ 2011.06404 ( 2020 ). arXiv: 2011.06404 https://arxiv.org/abs/ 2011.06404Google ScholarGoogle Scholar
  6. Gilles Barthe, Thomas Espitau, Luis María Ferrer Fioriti, and Justin Hsu. 2016a. Synthesizing Probabilistic Invariants via Doob's Decomposition. In Computer Aided Verification-28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part I (Lecture Notes in Computer Science), Swarat Chaudhuri and Azadeh Farzan (Eds.), Vol. 9779. Springer, 43-61. https://doi.org/10.1007/978-3-319-41528-4_3 Google ScholarGoogle ScholarCross RefCross Ref
  7. Gilles Barthe, Marco Gaboardi, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2016b. A program logic for union bounds. In International Colloquium on Automata, Languages and Programming (ICALP), Rome, Italy. arXiv:Yes http://arxiv.org/abs/1602.05681Google ScholarGoogle Scholar
  8. Gilles Barthe, Marco Gaboardi, Justin Hsu, and Benjamin C. Pierce. 2016c. Programming language techniques for diferential privacy. SIGLOG News 3, 1 ( 2016 ), 34-53. https://dl.acm.org/citation.cfm?id= 2893591Google ScholarGoogle Scholar
  9. Gilles Barthe, Boris Köpf, Federico Olmedo, and Santiago Zanella-Béguelin. 2013. Probabilistic Relational Reasoning for Diferential Privacy. ACM Transactions on Programming Languages and Systems 35, 3 ( 2013 ), 9. http://software.imdea. org/~bkoepf/papers/toplas13.pdfGoogle ScholarGoogle ScholarDigital LibraryDigital Library
  10. Raghav Bhaskar, Srivatsan Laxman, Adam D. Smith, and Abhradeep Thakurta. 2010. Discovering frequent patterns in sensitive data. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, July 25-28, 2010, Bharat Rao, Balaji Krishnapuram, Andrew Tomkins, and Qiang Yang (Eds.). ACM, 503-512. https://doi.org/10.1145/1835804.1835869 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Benjamin Bichsel, Timon Gehr, Dana Drachsler-Cohen, Petar Tsankov, and Martin T. Vechev. 2018. DP-Finder: Finding Diferential Privacy Violations by Sampling and Optimization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronto, ON, Canada, October 15-19, 2018, David Lie, Mohammad Mannan, Michael Backes, and XiaoFeng Wang (Eds.). ACM, 508-524. https://doi.org/10.1145/3243734.3243863 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Avrim Blum, Katrina Ligett, and Aaron Roth. 2013. A learning theory approach to noninteractive database privacy. J. ACM 60, 2 ( 2013 ), 12 : 1-12 : 25. https://doi.org/10.1145/2450142.2450148 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Aleksandar Chakarov and Sriram Sankaranarayanan. 2013. Probabilistic Program Analysis with Martingales. In Computer Aided Verification-25th International Conference, CAV 2013, Saint Petersburg, Russia, July 13-19, 2013. Proceedings (Lecture Notes in Computer Science), Natasha Sharygina and Helmut Veith (Eds.), Vol. 8044. Springer, 511-526. https: //doi.org/10.1007/978-3-642-39799-8_34 Google ScholarGoogle ScholarCross RefCross Ref
  14. T.-H. Hubert Chan, Elaine Shi, and Dawn Song. 2011. Private and continual release of statistics. ACM Transactions on Information and System Security 14, 3 ( 2011 ), 26. http://eprint.iacr.org/ 2010 /076.pdfGoogle ScholarGoogle Scholar
  15. Krishnendu Chatterjee, Hongfei Fu, Petr Novotný, and Rouzbeh Hasheminezhad. 2016. Algorithmic analysis of qualitative and quantitative termination problems for afine probabilistic programs. In Proceedings of the 43rd Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20-22, 2016, Rastislav Bodík and Rupak Majumdar (Eds.). ACM, 327-342. https://doi.org/10.1145/2837614.2837639 Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Michael R. Clarkson and Fred B. Schneider. 2010. Hyperproperties. J. Comput. Secur. 18, 6 ( 2010 ), 1157-1210. https: //doi.org/10.3233/JCS-2009-0393 Google ScholarGoogle ScholarCross RefCross Ref
  17. Patrick Cousot and Michael Monerau. 2012. Probabilistic Abstract Interpretation. In Programming Languages and Systems-21st European Symposium on Programming, ESOP 2012, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2012, Tallinn, Estonia, March 24-April 1, 2012. Proceedings (Lecture Notes in Computer Science), Helmut Seidl (Ed.), Vol. 7211. Springer, 169-193. https://doi.org/10.1007/978-3-642-28869-2_9 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Rayna Dimitrova, Bernd Finkbeiner, and Hazem Torfah. 2020. Probabilistic Hyperproperties of Markov Decision Processes. In Automated Technology for Verification and Analysis, Dang Van Hung and Oleg Sokolsky (Eds.). Springer International Publishing, Cham, 484-500.Google ScholarGoogle Scholar
  19. Zeyu Ding, Yuxin Wang, Guanhong Wang, Danfeng Zhang, and Daniel Kifer. 2018. Detecting Violations of Diferential Privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronto, ON, Canada, October 15-19, 2018, David Lie, Mohammad Mannan, Michael Backes, and XiaoFeng Wang (Eds.). ACM, 475-489. https://doi.org/10.1145/3243734.3243818 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In IACR Theory of Cryptography Conference (TCC), New York, New York. 265-284. http://dx.doi.org/10.1007/11681878_14 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Diferential Privacy. Foundations and Trends in Theoretical Computer Science 9, 3-4 ( 2014 ), 211-407. http://dx.doi.org/10.1561/0400000042 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Marco Gaboardi, Andreas Haeberlen, Justin Hsu, Arjun Narayan, and Benjamin C Pierce. 2013. Linear dependent types for diferential privacy. In ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), Rome, Italy. 357-370. http://dl.acm.org/citation.cfm?id= 2429113Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Anupam Gupta, Katrina Ligett, Frank McSherry, Aaron Roth, and Kunal Talwar. 2010. Diferentially private combinatorial optimization. In ACM-SIAM Symposium on Discrete Algorithms (SODA), Austin, Texas. 1106-1125. http://arxiv.org/pdf/ 0903.4510v2Google ScholarGoogle ScholarCross RefCross Ref
  24. Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. 2009. Refinement of Trace Abstraction. In Static Analysis, 16th International Symposium, SAS 2009, Los Angeles, CA, USA, August 9-11, 2009. Proceedings (Lecture Notes in Computer Science), Jens Palsberg and Zhendong Su (Eds.), Vol. 5673. Springer, 69-85. https://doi.org/10.1007/978-3-642-03237-0_7 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Benjamin Lucien Kaminski. 2019. Advanced weakest precondition calculi for probabilistic programs. Ph.D. Dissertation. RWTH Aachen University, Germany. http://publications.rwth-aachen.de/record/755408Google ScholarGoogle Scholar
  26. Joost-Pieter Katoen. 2016. The Probabilistic Model Checking Landscape. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science, LICS ' 16, New York, NY, USA, July 5-8, 2016, Martin Grohe, Eric Koskinen, and Natarajan Shankar (Eds.). ACM, 31-45. https://doi.org/10.1145/2933575.2934574 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Dexter Kozen. 1985. A Probabilistic PDL. J. Comput. System Sci. 30, 2 ( 1985 ), 162-178.Google ScholarGoogle ScholarCross RefCross Ref
  28. Satoshi Kura, Natsuki Urabe, and Ichiro Hasuo. 2019. Tail Probabilities for Randomized Program Runtimes via Martingales for Higher Moments. In Tools and Algorithms for the Construction and Analysis of Systems-25th International Conference, TACAS 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Prague, Czech Republic, April 6-11, 2019, Proceedings, Part II (Lecture Notes in Computer Science), Tomás Vojnar and Lijun Zhang (Eds.), Vol. 11428. Springer, 135-153. https://doi.org/10.1007/978-3-030-17465-1_8 Google ScholarGoogle ScholarCross RefCross Ref
  29. Marta Kwiatkowska, Gethin Norman, and David Parker. 2010. Advances and challenges of probabilistic model checking. In 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 1691-1698.Google ScholarGoogle ScholarCross RefCross Ref
  30. Serge Lang. 1966. Introduction to Transcendental Numbers. Addison-Wesley.Google ScholarGoogle Scholar
  31. Katrina Ligett, Seth Neel, Aaron Roth, Bo Waggoner, and Steven Z. Wu. 2017. Accuracy First: Selecting a Diferential Privacy Level for Accuracy Constrained ERM. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 2566-2576.Google ScholarGoogle Scholar
  32. Angus MacIntyre and Alex J. Wilkie. 1996. On the decidability of the real exponential field. In Kreiseliana. About and Around Georg Kreisel, Piergiorgio Odifreddi (Ed.). A.K. Peters, 441-467.Google ScholarGoogle Scholar
  33. Scott McCallum and Volker Weispfenning. 2012. Deciding polynomial-transcendental problems. Journal of Symbolic Computation 47, 1 ( 2012 ), 16-31.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Frank McSherry and Kunal Talwar. 2007. Mechanism Design via Diferential Privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2007 ), October 20-23, 2007, Providence, RI, USA, Proceedings. IEEE Computer Society, 94-103. https://doi.org/10.1109/FOCS. 2007.41 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. David Monniaux. 2000. Abstract Interpretation of Probabilistic Semantics. In Static Analysis, 7th International Symposium, SAS 2000, Santa Barbara, CA, USA, June 29-July 1, 2000, Proceedings (Lecture Notes in Computer Science), Jens Palsberg (Ed.), Vol. 1824. Springer, 322-339. https://doi.org/10.1007/978-3-540-45099-3_17 Google ScholarGoogle ScholarCross RefCross Ref
  36. Carroll Morgan, Annabelle McIver, and Karen Seidel. 1996. Probabilistic Predicate Transformers. ACM Transactions on Programming Languages and Systems 18, 3 ( 1996 ), 325-353.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Rajeev Motwani and Prabhakar Raghavan. 1995. Randomized Algorithms. Cambridge University Press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Jason Reed and Benjamin C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Diferential Privacy. In Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming (ICFP '10). Association for Computing Machinery, New York, NY, USA, 157-168. https://doi.org/10.1145/1863543.1863568 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Sriram Sankaranarayanan. 2020. Quantitative Analysis of Programs with Probabilities and Concentration of Measure Inequalities. In Foundations of Probabilistic Programming, Gilles Barthe, Joost-Pieter Katoen, and Alexandra Silva (Eds.). Cambridge University Press, TBA.Google ScholarGoogle Scholar
  40. Sriram Sankaranarayanan, Aleksandar Chakarov, and Sumit Gulwani. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '13, Seattle, WA, USA, June 16-19, 2013, Hans-Juergen Boehm and Cormac Flanagan (Eds.). ACM, 447-458. https://doi.org/10.1145/2491956.2462179 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Calvin Smith, Justin Hsu, and Aws Albarghouthi. 2019. Trace abstraction modulo probability. PACMPL 3, POPL ( 2019 ), 39 : 1-39 : 31. https://dl.acm.org/citation.cfm?id= 3290352Google ScholarGoogle Scholar
  42. A. Tarski. 1951. A decision method for Elementary Algebra and Geometry. University of California Press.Google ScholarGoogle Scholar
  43. Elisabet Lobo Vesga, Alejandro Russo, and Marco Gaboardi. 2019. A Programming Framework for Diferential Privacy with Accuracy Concentration Bounds. CoRR abs/ 1909.07918 ( 2019 ). arXiv: 1909.07918 http://arxiv.org/abs/ 1909.07918Google ScholarGoogle Scholar
  44. Di Wang, Jan Hofmann, and Thomas W. Reps. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018, Jefrey S. Foster and Dan Grossman (Eds.). ACM, 513-528. https: //doi.org/10.1145/3192366.3192408 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Di Wang, Jan Hofmann, and Thomas W. Reps. 2020. Tail Bound Analysis for Probabilistic Programs via Central Moments. CoRR abs/ 2001.10150 ( 2020 ). arXiv: 2001.10150 https://arxiv.org/abs/ 2001.10150Google ScholarGoogle Scholar
  46. Yu Wang, Siddhartha Nalluri, Borzoo Bonakdarpour, and Miroslav Pajic. 2019. Statistical Model Checking for Probabilistic Hyperproperties. CoRR abs/ 1902.04111 ( 2019 ). arXiv: 1902.04111 http://arxiv.org/abs/ 1902.04111Google ScholarGoogle Scholar
  47. Danfeng Zhang and Daniel Kifer. 2017. LightDP: towards automating diferential privacy proofs. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017, Giuseppe Castagna and Andrew D. Gordon (Eds.). ACM, 888-901. http://dl.acm.org/citation.cfm?id= 3009884Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Deciding accuracy of differential privacy schemes

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!