User interfaces for high-dimensional design problems: from theories to implementations

ABSTRACT

We introduce techniques for effectively performing tasks encompassing manipulation or exploration in high-dimensional spaces by the user. Such tasks emerge from applications involving many parameters or high-dimensional latent variables, with examples ranging from image editing, material editing, and shape design, to sound generation, arising in both general design problems and those with generative models from machine learning. Mathematically, such a task can be formulated as an optimization problem, where the user wants to maximize his or her subjective goodness over candidates generated by a model that has way too many control parameters for the user to handle. The solution is to bypass direct manipulation in high-dimensional spaces by extracting much lower-dimensional meaningful subspaces, which in turn give rise to tractable user interfaces. We introduce two core techniques for extracting such subspaces: one based on Bayesian optimization and the other on differential subspace search. Bayesian optimization is useful when only point-sampling is possible for the relation between the goodness and the control parameters (thus, the user can treat the system as a black box), while differential subspace search is useful when differential information is further available for the given model. We introduce both theoretical and implementation aspects of these techniques, and show applications to image editing, material editing, shape design, and sound generation.

References

1. David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. 2019. Semantic Photo Manipulation with a Generative Image Prior. ACM Trans. Graph. 38, 4, Article 59 (July 2019), 11 pages.
2. Alain Berlinet and Christine Thomas. 2004. Reproducing kernel Hilbert spaces in Probability and Statistics. Kluwer Academic Publishers.
3. Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010a. A Bayesian Interactive Optimization Approach to Procedural Animation Design. In Proc. SCA '10. 103--112.
4. Eric Brochu, Vlad M Cora, and Nando De Freitas. 2010b. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010).
5. Eric Brochu, Nando de Freitas, and Abhijeet Ghosh. 2007. Active Preference Learning with Discrete Choice Data. In Proc. NIPS '07. 409--416. http://papers.nips.cc/paper/3219-active-preference-learning-with-discrete-choice-data
6. Adam D Bull. 2011. Convergence rates of efficient global optimization algorithms. The Journal of Machine Learning Research 12 (2011), 2879--2904.
7. Emmanuel J. Cand ès and Benjamin Recht. 2009. Exact Matrix Completion via Convex Optimization. Foundations of Computational Mathematics 9, 6 (2009), 717--772.
8. Siddhartha Chaudhuri, Evangelos Kalogerakis, Stephen Giguere, and Thomas Funkhouser. 2013. Attribit: Content Creation with Semantic Attributes. In Proc. UIST '13. 193--202.
9. Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In Proc. CVPR 2019. 5939--5948.
10. Chia-Hsing Chiu, Yuki Koyama, Yu-Chi Lai, Takeo Igarashi, and Yonghao Yue. 2020. Human-in-the-Loop Differential Subspace Search in High-Dimensional Latent Space. ACM Transactions on Graphics 39, 4 (July 2020), 85:1--85:15.
11. Toby Chong, I-Chao Shen, Issei Sato, and Takeo Igarashi. 2021. Interactive Optimization of Generative Image Modelling using Sequential Subspace Search and Content-based Guidance. Comput. Graph. Forum 40, 1 (2021), 279--292.
12. Wei Chu and Zoubin Ghahramani. 2005. Preference Learning with Gaussian Processes. In Proc. ICML '05. 137--144.
13. Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. 2019. GANSynth: Adversarial Neural Audio Synthesis. In Proc. ICLR 2019. https://openreview.net/forum?id=H1xQVn09FX
14. Elena Garces, Aseem Agarwala, Diego Gutierrez, and Aaron Hertzmann. 2014. A Similarity Measure for Illustration Style. ACM Trans. Graph. 33, 4 (July 2014), 93:1--93:9.
15. David Ginsbourger, Rodolphe Le Riche, and Laurent Carraro. 2010. Kriging is well-suited to parallelize optimization. In Computational Intelligence in Expensive Optimization Problems. Springer, 131--162.
16. Javier González, Zhenwen Dai, Andreas C. Damianou, and Neil D. Lawrence. 2017. Preferential Bayesian Optimization. In Proc. ICML '17. 1282--1291. http://proceedings.mlr.press/v70/gonzalez17a.html
17. David Haussler. 1999. Convolution Kernels on Discrete Structures. Technical Report UCS-CRL-99-10. Santa Cruz, CA, USA.
18. Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola. 2008. Kernel methods in machine learning. Annals of Statistics 36, 3 (2008), 1171--1220.
19. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal Unsupervised Image-to-image Translation. arXiv preprint arXiv:1804.04732 (2018).
20. D.R. Jones, M. Schonlau, and W.J. Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global optimization 13, 4 (1998), 455--492.
21. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018a. Progressive growing of GANs for improved quality, stability, and variation. In Proc. International Conference on Learning Representations (ICLR).
22. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018b. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In Proc. ICLR 2018. https://openreview.net/forum?id=Hk99zCeAb
23. Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction, Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes (Eds.). Oxford University Press, Chapter 6, 153--184.
24. Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-Powered Parameter Analysis for Visual Design Exploration. In Proc. UIST '14. 65--74.
25. Yuki Koyama, Issei Sato, and Masataka Goto. 2020. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. 39, 4 (July 2020), 88:1--88:12.
26. Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential Line Search for Efficient Visual Design Optimization by Crowds. ACM Transactions on Graphics 36, 4 (July 2017), 48:1--48:11.
27. Nils M. Kriege, Fredrik D. Johansson, and Christopher Morris. 2020. A survey on graph kernels. Applied Network Science 5, 6 (2020).
28. D. Lizotte. 2008. Practical Bayesian Optimization. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral Normalization for Generative Adversarial Networks. In Proc. ICLR 2018. https://openreview.net/forum?id=B1QRgziT-
29. J. Mockus, V. Tiesis, and A. Zilinskas. 1998. The application of Bayesian methods for seeking the extremum. Towards Global Optimization 2, 4 (1998), 117--129.
30. Yurii Nesterov. 2018. Lectures on Convex Optimization. Springer.
31. Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization (2nd ed.). Springer Science+Business Media.
32. Peter O'Donovan, Jundefinednis Lundefinedbeks, Aseem Agarwala, and Aaron Hertzmann. 2014. Exploratory Font Selection Using Crowd-sourced Attributes. ACM Trans. Graph. 33, 4, Article 92:1--92:9 (July 2014).
33. Tiziano Portenier, Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker. 2018. Faceshop: Deep Sketch-based Face Image Editing. ACM Trans. Graph. 37, 4, Article 99 (July 2018), 13 pages.
34. Carl Edward Rasmussen and Christopher K. I. Williams. 2005. Gaussian Processes for Machine Learning. The MIT Press. http://www.gaussianprocess.org/gpml/
35. Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proc. IEEE 104, 1 (January 2016), 148--175.
36. Ariel Shamir, Niloy J. Mitra, Nobuyuki Umetani, and Yuki Koyama. 2020. Intelligent Tools for Creative Graphics. In ACM SIGGRAPH 2020 Courses. Article 15:1--15:11.
37. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Proc. NIPS '12. 2951--2959. https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms
38. Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. [n.d.]. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning. 1015--1022.
39. Kristi Tsukida and Maya R. Gupta. 2011. How to Analyze Paired Comparison Data. Technical Report UWEETR-2011-0004. University of Washington, Department of Electrical Engineering. https://vannevar.ece.uw.edu/techsite/papers/refer/UWEETR-2011-0004.html
40. Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. (2018).
41. Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2020. Generative Melody Composition with Human-in-the-Loop Bayesian Optimization. In Proc. CSMC-MuMe '20. 21:1--21:10. https://boblsturm.github.io/aimusic2020/papers/CSMC__MuMe_2020_paper_21.pdf
42. Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2021. Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In Proc. IUI '21. 43--47.