Designing Perceptual Puzzles by Differentiating Probabilistic Programs

ABSTRACT

We design new visual illusions by finding “adversarial examples” for principled models of human perception — specifically, for probabilistic models, which treat vision as Bayesian inference. To perform this search efficiently, we design a differentiable probabilistic programming language, whose API exposes MCMC inference as a first-class differentiable function. We demonstrate our method by automatically creating illusions for three features of human vision: color constancy, size constancy, and face perception.

References

1. Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. 2009. Action understanding as inverse planning. Cognition 113, 3 (2009), 329–349. https://dspace.mit.edu/handle/1721.1/60852
2. Sai Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein, Tzu-Mao Li, and Jonathan Ragan-Kelley. 2021. Systematically Differentiating Parametric Discontinuities. ACM Trans. Graph. 40, 107 (2021), 107:1–107:17. https://people.csail.mit.edu/sbangaru/projects/teg-2021/teg-2021.pdf
3. Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. 2013. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences 110, 45(2013), 18327–18332. https://www.pnas.org/content/pnas/110/45/18327.full.pdf
4. Adrien Bousseau, Emmanuelle Chapoulie, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Optimizing environment maps for material depiction. In Computer graphics forum, Vol. 30. 1171–1180. https://dl.acm.org/doi/abs/10.1111/j.1467-8659.2011.01975.x
5. David H Brainard and William T Freeman. 1997. Bayesian color constancy. JOSA A 14, 7 (1997), 1393–1411. http://people.csail.mit.edu/billf/publications/Bayesian_Color_Constancy.pdf
6. Andrew Campbell, Wenlong Chen, Vincent Stimper, Jose Miguel Hernandez-Lobato, and Yichuan Zhang. 2021. A Gradient Based Strategy for Hamiltonian Monte Carlo Hyperparameter Optimization. In International Conference on Machine Learning. PMLR, 1238–1248. http://proceedings.mlr.press/v139/campbell21a/campbell21a.pdf
7. Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. Journal of statistical software 76, 1 (2017), 1–32. https://www.jstatsoft.org/article/view/v076i01
8. Kartik Chandra, Chuma Kabaghe, and Gregory Valiant. 2021. Beyond Laurel/Yanny: An Autoencoder-Enabled Search for Polyperceivable Audio. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, Online, 593–598. https://doi.org/10.18653/v1/2021.acl-short.75
9. Ming-Te Chi, Tong-Yee Lee, Yingge Qu, and Tien-Tsin Wong. 2008. Self-Animating Images: Illusory Motion Using Repeated Asymmetric Patterns. ACM Trans. Graph. 27, 3 (aug 2008), 1–8. https://doi.org/10.1145/1360612.1360661
10. Hung-Kuo Chu, Wei-Hsin Hsu, Niloy J Mitra, Daniel Cohen-Or, Tien-Tsin Wong, and Tong-Yee Lee. 2010. Camouflage images.ACM Trans. Graph. 29, 4 (2010), 51–1. https://dl.acm.org/doi/abs/10.1145/1833349.1778788
11. Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. 2019. Exponential Family Estimation via Adversarial Dynamics Embedding. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/767d01b4bac1a1e8824c9b9f7cc79a04-Paper.pdf
12. Yilun Du, Shuang Li, B. Joshua Tenenbaum, and Igor Mordatch. 2021. Improved Contrastive Divergence Training of Energy Based Models. In Proceedings of the 38th International Conference on Machine Learning (ICML-21). https://arxiv.org/abs/2012.01316
13. Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. 1987. Hybrid Monte Carlo. Physics letters B 195, 2 (1987), 216–222. https://archive.org/download/wikipedia-scholarly-sources-corpus/10.1016%252F0361-9230%252887%252990129-8.zip/10.1016%252F0370-2693%252887%252991197-X.pdf
14. Frédo Durand, Maneesh Agrawala, Bruce Gooch, Victoria Interrante, Victor Ostromoukhov, and Denis Zorin. 2002. Perceptual and artistic principles for effective computer depiction. SIGGRAPH 2002 Course# 13 Notes(2002). http://people.csail.mit.edu/fredo/SIG02_ArtScience/DepictionNotes2.pdf
15. Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein. 2018. Adversarial examples that fool both computer vision and time-limited humans. Advances in neural information processing systems 31 (2018). https://proceedings.neurips.cc/paper/2018/file/8562ae5e286544710b2e7ebe9858833b-Paper.pdf
16. Wilson S Geisler and Daniel Kersten. 2002. Illusions, perception and Bayes. Nature neuroscience 5, 6 (2002), 508–510. https://www.cs.utexas.edu/~dana/NVGeisler2.pdf
17. Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter. 2018. Morphable face models-an open framework. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 75–82. https://ieeexplore.ieee.org/abstract/document/8373814
18. Noah D Goodman, Joshua B. Tenenbaum, and The ProbMods Contributors. 2016. Probabilistic Models of Cognition. http://probmods.org/v2. Accessed: 2021-10-15.
19. Nishad Gothoskar, Marco Cusumano-Towner, Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Garrett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 2021. 3DP3: 3D Scene Perception via Probabilistic Programming. Advances in Neural Information Processing Systems 34 (2021). https://proceedings.neurips.cc/paper/2021/file/4fc66104f8ada6257fa55f29a2a567c7-Paper.pdf
20. Aaron Hertzmann. 2020. Why do line drawings work? a realism hypothesis. Perception 49, 4 (2020), 439–451. https://arxiv.org/abs/2002.06260
21. Inbar Huberman and Raanan Fattal. 2015. Reducing Lateral Visual Biases in Displays. Computer Graphics Forum (CGF). https://www.cs.huji.ac.il/w~raananf/projects/lateral_biases/
22. Daniel Kersten, Pascal Mamassian, and Alan Yuille. 2004. Object perception as Bayesian inference. Annu. Rev. Psychol. 55(2004), 271–304. http://mamassian.free.fr/papers/KerstenMamassianYuille04.pdf
23. Erum Arif Khan, Erik Reinhard, Roland W. Fleming, and Heinrich H. Bülthoff. 2006. Image-Based Material Editing. ACM Trans. Graph. 25, 3 (jul 2006), 654–663. https://doi.org/10.1145/1141911.1141937
24. Diederik P Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. arxiv:1312.6114 [stat.ML] https://arxiv.org/pdf/1312.6114.pdf
25. David C Knill and Whitman Richards. 1996. Perception as Bayesian inference. Cambridge University Press.
26. Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. 2015. Picture: A probabilistic programming language for scene perception. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4390–4399. https://mrkulk.github.io/www_cvpr15/1999.pdf
27. Rosa Lafer-Sousa, Katherine L Hermann, and Bevil R Conway. 2015. Striking individual differences in color perception uncovered by ‘The Dress’ photograph. Current Biology 25, 13 (2015), R545–R546. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4921196/pdf/nihms686513.pdf
28. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft rasterizer: A differentiable renderer for image-based 3D reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7708–7717. https://arxiv.org/pdf/1904.01786.pdf
29. Margaret S Livingstone. 2000. Is it warm? Is it real? Or just low spatial frequency?Science 290, 5495 (2000), 1299–1299. https://livingstone.hms.harvard.edu/sites/livingstone.hms.harvard.edu/files/publications/2000_November17.%20Livingstone.%20Is%20It%20Warm_%20Is%20it%20Real_%20Or%20Just%20Low%20Spatial%20Frequency_.pdf
30. Li-Qian Ma, Kun Xu, Tien-Tsin Wong, Bi-Ye Jiang, and Shi-Min Hu. 2013. Change blindness images. IEEE transactions on visualization and computer graphics 19, 11(2013), 1808–1819. https://cg.cs.tsinghua.edu.cn/papers/TVCG-2013-changeblindness.pdf
31. Dougal Maclaurin, David Duvenaud, and Ryan Adams. 2015. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning. PMLR, 2113–2122. http://proceedings.mlr.press/v37/maclaurin15.pdf
32. Pascal Mamassian, Michael Landy, and Laurence T Maloney. 2002. Bayesian modelling of visual perception. Probabilistic models of the brain(2002), 13–36. http://mamassian.free.fr/papers/mamassian_mit02.pdf
33. Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. 2013. Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs. In Advances in Neural Information Processing Systems, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Vol. 26. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2013/file/fa14d4fe2f19414de3ebd9f63d5c0169-Paper.pdf
34. David Marr. 1982. Vision. W. H. Freeman and Company. https://mitpress.mit.edu/books/vision
35. Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. 2020. Monte Carlo Gradient Estimation in Machine Learning.J. Mach. Learn. Res. 21, 132 (2020), 1–62. https://arxiv.org/pdf/1906.10652.pdf
36. Hadi Mohasel Afshar and Justin Domke. 2015. Reflection, Refraction, and Hamiltonian Monte Carlo. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.). Vol. 28. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2015/file/8303a79b1e19a194f1875981be5bdb6f-Paper.pdf
37. Radford M Neal 2011. MCMC using Hamiltonian dynamics. Handbook of Markov chain Monte Carlo 2, 11 (2011), 2. https://arxiv.org/pdf/1206.1901.pdf
38. Yu Okano, Shogo Fukushima, Masahiro Furukawa, and Hiroyuki Kajimoto. 2010. Embedded Motion: Generating the Perception of Motion in Peripheral Vision. In ACM SIGGRAPH ASIA 2010 Posters (Seoul, Republic of Korea) (SA ’10). Association for Computing Machinery, New York, NY, USA, Article 41, 1 pages. https://doi.org/10.1145/1900354.1900400
39. Aude Oliva, Antonio Torralba, and Philippe G Schyns. 2006. Hybrid images. ACM Transactions on Graphics (TOG) 25, 3 (2006), 527–532. https://stanford.edu/class/ee367/reading/OlivaTorralb_Hybrid_Siggraph06.pdf
40. Tobias Ritschel, Kaleigh Smith, Matthias Ihrke, Thorsten Grosch, Karol Myszkowski, and Hans-Peter Seidel. 2008. 3D Unsharp Masking for Scene Coherent Enhancement. Vol. 27. Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/1360612.1360689
41. Tim Salimans, Diederik Kingma, and Max Welling. 2015. Markov chain Monte Carlo and variational inference: Bridging the gap. In International Conference on Machine Learning. PMLR, 1218–1226. http://proceedings.mlr.press/v37/salimans15.pdf
42. Roger N Shepard. 1994. Perceptual-cognitive universals as reflections of the world. Psychonomic Bulletin & Review 1, 1 (1994), 2–28. http://ruccs.rutgers.edu/images/personal-zenon-pylyshyn/docs/transfers/shepard_space_bbs2001.pdf
43. Fabian H. Sinz, Xaq Pitkow, Jacob Reimer, Matthias Bethge, and Andreas S. Tolias. 2019. Engineering a Less Artificial Intelligence. Neuron 103, 6 (2019), 967–979. https://doi.org/10.1016/j.neuron.2019.08.034
44. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations. http://arxiv.org/abs/1312.6199
45. Corey Toler-Franklin, Adam Finkelstein, and Szymon Rusinkiewicz. 2007. Illustration of complex real-world objects using images with normals. In Proceedings of the 5th international symposium on Non-photorealistic animation and rendering. 111–119. https://pixl.cs.princeton.edu/pubs/Toler-Franklin_2007_IOC/rgbn.pdf
46. Nikolaus F Troje and Ulrike Siebeck. 1998. Illumination-induced apparent shift in orientation of human heads. Perception 27, 6 (1998), 671–680. https://journals.sagepub.com/doi/pdf/10.1068/p270671#also-see-slack
47. Arash Vahdat, Evgeny Andriyash, and William Macready. 2020. Undirected graphical models as approximate posteriors. In International Conference on Machine Learning. PMLR, 9680–9689. http://proceedings.mlr.press/v119/vahdat20a/vahdat20a.pdf
48. Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. 2018. An introduction to probabilistic programming. arXiv preprint arXiv:1809.10756(2018). https://arxiv.org/pdf/1809.10756
49. Pascal Wallisch. 2017. Illumination assumptions account for individual differences in the perceptual interpretation of a profoundly ambiguous stimulus in the color domain:“The Dress”. Journal of Vision 17, 4 (2017), 5–5. https://jov.arvojournals.org/article.aspx?articleid=2617976&mbid=synd_yahoostyle
50. Pascal Wallisch and Michael Karlovich. 2019. Disagreeing about Crocs and socks: Creating profoundly ambiguous color displays. arXiv preprint arXiv:1908.05736(2019). https://arxiv.org/pdf/1908.05736.pdf
51. David R. Walton, Rafael Kuffner Dos Anjos, Sebastian Friston, David Swapp, Kaan Akşit, Anthony Steed, and Tobias Ritschel. 2021. Beyond Blur: Real-Time Ventral Metamers for Foveated Rendering. ACM Trans. Graph. 40, 4, Article 48 (jul 2021), 14 pages. https://doi.org/10.1145/3450626.3459943
52. Yair Weiss, Eero P Simoncelli, and Edward H Adelson. 2002. Motion illusions as optimal percepts. Nature neuroscience 5, 6 (2002), 598–604. https://www.nature.com/articles/nn858
53. Christoph Witzel and Matteo Toscani. 2020. How to make a #theDress. JOSA A 37, 4 (2020), A202–A211. https://www.osapublishing.org/josaa/abstract.cfm?uri=josaa-37-4-A202
54. Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. 2014. A New Approach to Probabilistic Programming Inference. In Proceedings of the 17th International conference on Artificial Intelligence and Statistics. 1024–1032. https://probprog.github.io/anglican/assets/pdf/wood-aistats-2014.pdf
55. Guodong Zhang, Kyle Hsu, Jianing Li, Chelsea Finn, and Roger B Grosse. 2021. Differentiable annealed importance sampling and the perils of gradient noise. Advances in Neural Information Processing Systems 34 (2021). https://proceedings.neurips.cc/paper/2021/file/a1a609f1ac109d0be28d8ae112db1bbb-Paper.pdf
56. Yuan Zhou, Bradley J. Gram-Hansen, Tobias Kohn, Tom Rainforth, Hongseok Yang, and Frank Wood. 2019. LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 148–157. https://proceedings.mlr.press/v89/zhou19b.html
57. Zhenglong Zhou and Chaz Firestone. 2019. Humans can decipher adversarial images. Nature communications 10, 1 (2019), 1–9. https://www.nature.com/articles/s41467-019-08931-6
58. David M Zoltowski, Diana Cai, and Ryan P Adams. 2020. Slice Sampling Reparameterization Gradients. In Third Symposium on Advances in Approximate Bayesian Inference. https://openreview.net/pdf?id=cT_RMSqVf4