Interactive images: cuboid proxies for smart image manipulation

Abstract

Images are static and lack important depth information about the underlying 3D scenes. We introduce interactive images in the context of man-made environments wherein objects are simple and regular, share various non-local relations (e.g., coplanarity, parallelism, etc.), and are often repeated. Our interactive framework creates partial scene reconstructions based on cuboid-proxies with minimal user interaction. It subsequently allows a range of intuitive image edits mimicking real-world behavior, which are otherwise difficult to achieve. Effectively, the user simply provides high-level semantic hints, while our system ensures plausible operations by conforming to the extracted non-local relations. We demonstrate our system on a range of real-world images and validate the plausibility of the results using a user study.

References

1. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM TOG (SIGGRAPH) 28, 3, 24:1--11.
2. Biederman, I., Mezzanotte, R., and Rabinowitz, J. 1982. Scene perception: Detecing and judging objects undergoing relational violations. Cognitive Psychology 14, 143--177.
3. Carroll, R., Agarwala, A., and Agrawala, M. 2010. Image warps for artistic perspective manipulation. ACM TOG (SIGGRAPH) 29, 4, 127:1--127:9.
4. Cheng, M.-M., Zhang, F.-L., Mitra, N. J., Huang, X., and Hu, S.-M. 2010. RepFinder: Finding approximately repeated scene elements for image editing. ACM TOG (SIGGRAPH) 29, 4, 83:1--83:8.
5. Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In IEEE ICCV, 1033--1038.
6. Fisher, M., Savva, M., and Hanrahan, P. 2011. Characterizing structural relationships in scenes using graph kernels. ACM TOG (SIGGRAPH) 30, 34:1--34:12.
7. Gal, R., Sorkine, O., Mitra, N. J., and Cohen-Or, D. 2009. iWIRES: an analyze-and-edit approach to shape manipulation. ACM TOG (SIGGRAPH) 28, 33:1--33:10.
8. Gibson, J. J. 1979. The Ecological Approach to Visual Perception. MIT Press.
9. Guo, R., Dai, Q., and Hoiem, D. 2011. Single-image shadow detection and removal using paired regions. In IEEE CVPR, 2033--2040.
10. Gupta, A., Efros, A. A., and Hebert, M. 2010. Blocks world revisited: Image understanding using qualitative geometry and mechanics. In ECCV, 119--153.
11. Gupta, A., Satkin, S., Efros, A. A., and Hebert, M. 2011. From 3d scene geometry to human workspace. In IEEE CVPR, 1961--1968.
12. Hartley, A., and Zisserman, A. 2006. Multiple view geometry in computer vision (2. Ed.). Cambridge University Press.
13. Hays, J., and Efros, A. 2007. Scene completion using millions of photographs. ACM TOG (SIGGRAPH) 26, 3, 87--94.
14. Hedau, V., Hoiem, D., and Forsyth, D. 2010. Thinking inside the box: Using appearance models and context based on room geometry. In ECCV, 224--237.
15. Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM TOG (SIGGRAPH) 24, 3, 577--584.
16. Jain, A., Thormählen, T., Seidel, H.-P., and Theobalt, C. 2010. Moviereshape: Tracking and reshaping of humans in videos. ACM TOG (SIGGRAPH Asia) 29, 5, 148:1--148:9.
17. Jiang, N., Tan, P., and Cheong, L.-F. 2009. Symmetric architecture modeling with a single image. ACM TOG (SIGGRAPH Asia) 28, 5, 113:1--113:8.
18. Karsch, K., Hedau, V., Forsyth, D., and Hoiem, D. 2011. Rendering synthetic objects into legacy photographs. ACM TOG (SIGGRAPH Asia) 30, 6, 157:1--157:12.
19. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM TOG (SIGGRAPH) 26, 3 (August), 3.
20. Lourakis, M., 2004. levmar: Levenberg-marquardt non-linear least squares algorithms in C/C++. {web page} http://www.ics.forth.gr/~lourakis/levmar/.
21. Mitra, N. J., Yang, Y.-L., Yan, D.-M., Li, W., and Agrawala, M. 2010. Illustrating how mechanical assemblies work. ACM TOG (SIGGRAPH) 29, 4, 58:1--58:12.
22. Norman, D. 1990. Design of Everyday Things. MIT Press.
23. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In ACM SIGGRAPH, 433--442.
24. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM TOG (SIGGRAPH) 22, 3, 313--318.
25. Rother, C., Kolmogorov, V., and Blake, A. 2004. "grab-cut": interactive foreground extraction using iterated graph cuts. ACM TOG (SIGGRAPH) 23, 3, 309--314.
26. Rubinstein, M., Shamir, A., and Avidan, S. 2009. Multi-operator media retargeting. ACM TOG (SIGGRAPH) 28, 3, 23:1--23:11.
27. Saxena, A., Sun, M., and Ng, A. 2009. Make3D: Learning 3D scene structure from a single still image. IEEE PAMI 31, 5, 824--840.
28. Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In IEEE CVPR, 519--528.
29. Shapira, L., Shamir, A., and Cohen-Or, D. 2009. Image appearance exploration by model-based navigation. CGF 28, 2, 629--638.
30. Sinha, S. N., Steedly, D., Szeliski, R., Agrawala, M., and Pollefeys, M. 2008. Interactive 3D architectural modeling from unordered photo collections. ACM TOG (SIGGRAPH Asia) 27, 5, 159:1--159:10.
31. Sun, J., Yuan, L., Jia, J., and Shum, H. 2005. Image completion with structure propagation. ACM TOG (SIGGRAPH) 24, 3, 861--868.
32. Wang, Y.-S., Tai, C.-L., Sorkine, O., and Lee, T.-Y. 2008. Optimized scale-and-stretch for image resizing. ACM TOG (SIGGRAPH Asia) 27, 5, 118:1--118:8.
33. Wei, L.-Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the art in example-based texture synthesis. In EG-STAR, 93--117.
34. Wilczkowiak, M., Sturm, P. F., and Boyer, E. 2005. Using geometric constraints through parallelepipeds for calibration and 3D modeling. IEEE PAMI 27, 2, 194--207.
35. Wu, H., Wang, Y.-S., Feng, K.-C., Wong, T.-T., Lee, T.-Y., and Heng, P.-A. 2010. Resizing by symmetry-summarization. ACM TOG (SIGGRAPH Asia) 29, 6, 159:1--159:9.
36. Xue, T., Liu, J., and Tang, X. 2010. Object cut: Complex 3d object reconstruction through line drawing separation. In IEEE CVPR, 1149--1156.
37. Yang, Y.-L., Yang, Y.-J., Pottmann, H., and Mitra, N. J. 2011. Shape space exploration of constrained meshes. ACM TOG (SIGGRAPH Asia) 30, 6.
38. Zheng, Y., Fu, H., Cohen-Or, D., Au, O. K.-C., and Tai, C.-L. 2011. Component-wise controllers for structure-preserving shape manipulation. CGF 30, 2, 563--572.
39. Zhou, S., Fu, H., Liu, L., Cohen-Or, D., and Han, X. 2010. Parametric reshaping of human bodies in images. ACM TOG (SIGGRAPH) 29, 4, 126:1--126:10.
40. Zisserman, A., Reid, I. D., and Criminisi, A. 1999. Single view metrology. In IEEE ICCV, 434--441.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

  • Computer graphics

    • Image manipulation

    • Shape modeling