Image-based material editing

Abstract

Photo editing software allows digital images to be blurred, warped or re-colored at the touch of a button. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. Here we present a method for automatically replacing one material with another, completely different material, starting with only a single high dynamic range image as input. Our approach exploits the fact that human vision is surprisingly tolerant of certain (sometimes enormous) physical inaccuracies, while being sensitive to others. By adjusting our simulations to be careful about those aspects to which the human visual system is sensitive, we are for the first time able to demonstrate significant material changes on the basis of a single photograph as input.

References

1. Adelson, E. 2001. On seeing stuff: The perception of materials by humans. In Proc. of the SPIE, Human Vision and Electronic Imaging VI, vol. 4299, 1--12.
2. Belhumeur, P., Kriegman, D., and Yuille, A. 1999. The bas-relief ambiguity. International Journal of Computer Vision 1, 33--44.
3. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. In SIGGRAPH '00: Proceedings of the 27th annual conference on computer graphics and interactive techniques, 417--424.
4. Blake, A., and Bülthoff, H. H. 1990. Does the brain know the physics of specular reflection? Nature 343, 6254, 165--168.
5. Blake, A., and Bülthoff, H. H. 1991. Shape from specularities: Computation and psychophysics. Philosophical Transactions of the Royal Society (London) Series B 331, 237--252.
6. Chuang, Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D., and Szeliski, R. 2000. Environment matting extensions: Towards higher accuracy and real-time capture. In SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 121--130.
7. Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 Conference Proceedings, 369--378.
8. Debevec, P. E. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with illumination and high dynamic range photography. In SIGGRAPH 98 Conference Proceedings, 45--50.
9. Debevec, P. 2002. A tutorial on image-based lighting. IEEE Computer Graphics and Applications 22, 2, 26--34.
10. Drew, M. S. 1994. Robust specularity detection from a single multi-illuminant color image. Computer Vision, Graphics, and Image Processing: Image Understanding 59, 3, 320--327.
11. Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Fragment-based image completion. ACM Transactions on Graphics 22, 3, 303--312.
12. Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Transactions on Graphics 21, 3, 257--266.
13. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3, 673--678.
14. Fang, H., and Hart, J. C. 2004. Textureshop: Texture synthesis as a photographic editing tool. ACM Transactions on Graphics 23, 3, 354--358.
15. Fleming, R. W., and Bülthoff, H. H. 2005. Low-level image cues in the perception of translucent materials. ACM Trans. on Applied Perception 2, 3, 346--382.
16. Freeman, W. T. 1994. The generic viewpoint assumption in a framework for visual perception. Nature 368, 542--545.
17. Freeman, W. T. 1996. Exploiting the generic viewpoint assumption. International Journal of Computer Vision 20, 3, 243--261.
18. Hart, J. C., Francis, G. K., and Kauffman, L. H. 1994. Visualizing quaternion rotation. ACM Transactions on Graphics 13, 3, 256--276.
19. Igarashi, T., Matsuoka, S., and Tanaka, H. 1999. Teddy: a sketching interface for 3d freeform design. In SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 409--416.
20. Itu, 1990. International Telecommunication Union ITU-R Recommendation BT.709, Basic Parameter Values for the HDTV Standard for the Studio and for International Programme Exchange. Geneva. Formerly CCIR Rec. 709.
21. Kajiya, J. T. 1986. The rendering equation. In SIGGRAPH '86: Proc. of the 13th annual conference on Computer graphics and interactive techniques, 143--150.
22. Kang, S. 1998. Depth-painting for image based rendering applications. Tech. rep., Microsoft.
23. Klinker, G. J., Shafer, S. A., and Kanade, T. 1988. The measurement of highlights in color images. International Journal of Computer Vision 2, 1, 7--32.
24. Koenderink, J. J., and van Doorn, A. J. 1979. The internal representation of solid shape with respect to vision. Biological Cybernetics 32, 211--216.
25. Koenderink, J. J., van Doorn, A. J., Kappers, A., and Todd, J. 2001. Ambiguity and the 'mental eye' in pictorial relief. Perception 30, 4, 431--448.
26. Langer, M. S., and Bülthoff, H. H. 2000. Depth discrimination from shading under diffuse lighting. Perception 29, 6, 649--660.
27. Langer, M. S., and Zucker, S. W. 1994. Shape-from-shading on a cloudy day. Journal of the Optical Society of America A 11, 467--478.
28. Lawson, R., Bülthoff, H. H., and Dumbell, S. 2003. Interactions between view changes and shape changes in picture - picture matching. Perception 32, 1465--1498.
29. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Transactions on Graphics 23, 3, 303--308.
30. Matusik, W., Pfister, H., Brand, M., and McMillan, L. 2003. A data-driven reflectance model. ACM Transactions on Graphics 22, 3, 759--769.
31. McGuire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. ACM Transactions on Graphics 24, 3, 567--576.
32. Metzger, W. 1975. Gesetze des Sehens (Laws of Seeing). Waldemar Kramer, Frankfurt.
33. Motoyoshi, I., Nishida, S., and Adelson, E. H. 2005. Luminance re-mapping for the control of apparent material. In Proceedings of the 2nd ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization, 165.
34. Naka, K. I., and Rushton, W. A. H. 1966. S-potentials from luminosity units in the retina of fish (cyprinidae). Journal of Physiology 185, 587--599.
35. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 433--442.
36. Oliveira, M. M. 2002. Image-based modeling and rendering techniques: A survey. RITA - Revista de Informática Teórica e Aplicada IX (October), 37--66.
37. Ostromoukhov, V., Donohue, C., and Jodoin, P. 2004. Fast hierarchical importance sampling with blue noise properties. ACM Transactions on Graphics 23, 3, 488--495.
38. Ostrovsky, Y., Cavanagh, P., and Sinha, P. 2005. Perceiving illumination inconsistencies. Perception 34, 1301--1314.
39. te Pas, S. F., and Pont, S. C. 2005. A comparison of material and illumination discrimination performance for real rough, real smooth and computer generated smooth spheres. In APGV '05: Proceedings of the 2nd ACM Symposium on Appied Perception in Graphics and Visualization, 75--81.
40. te Pas, S. F., and Pont, S. C. 2005. Estimations of light source direction depend critically on material brdfs. Perception 34, 212. Supplement ECVP05, A Coruña.
41. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for reflection. ACM Transactions on Graphics 23, 4, 1004--1042.
42. Reinhard, E., and Khan, E. A. 2005. Depth-of-field-based alpha-matte-extraction. In APGV '05: Proceedings of the 2nd ACM Symposium on Appied Perception in Graphics and Visualization, 95--102.
43. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21 (September/October), 34--41.
44. Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. on Graphics 21, 3, 267--276.
45. Reinhard, E., Shirley, P., Ashikhmin, M., and Troscianko, T. 2004. Second order image statistics for computer graphics. In APGV '04: Proc. of the 1st ACM Symposium on Applied Perception in Graphics and Visualization (APGV), 99--106.
46. Reinhard, E., Khan, E. A., Akyüz, A. O., Fleming, R. W., and Bülthoff, H. H. 2005. Image-based material editing. ACM SIGGRAPH Computer Animation Festival/Video Review, 152.
47. Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers, San Francisco.
48. Rother, C., Kolmogorov, V., and Blake, A. 2004. "grabcut" --- interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 3, 309--314.
49. Ruderman, D. L. 1997. The statistics of natural images. Network: Computation in Neural Systems 5, 4, 517--548.
50. van der Schaaf, A. 1998. Natural image statistics and visual processing. PhD thesis, Rijksuniversiteit Groningen, The Netherlands.
51. Simons, D. J., and Levin, D. T. 1997. Change blindness. Trends in Cognitive Science 1, 261--267.
52. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In SIGGRAPH '96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 259--268.
53. Sun, J., Yuan, L., Jia, J., and Shum, H.-Y. 2005. Image completion with structure propagation. ACM Transactions on Graphics 24, 3, 861--868.
54. Tarr, M. J., Kersten, D., and Bülthoff, H. H. 1999. Why the visual recognition system might encode the effects of illumination. Vision Research 39, 2259--2275.
55. Todd, J. T., and Mingolla, E. 1983. Perception of surface curvature and direction of illuminant from patterns of shading. Journal of Experimental Psychology: Human Perception and Performance 9, 583--595.
56. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. IEEE International Conference on Computer Vision, 836--846.
57. Torralba, A., and Oliva, A. 2003. Statistics of natural image categories. Network: Computation in Neural Systems 14, 391--412.
58. Zelinka, S., Fang, H., Garland, M., and Hart, J. C. 2005. Interactive material replacement in photographs. In GI '05: Proceedings of the 2005 conference on Graphics interface, 227--232.
59. Zhang, R., Tsai, P., Cryer, J., and Shah, M. 1999. Shape from shading: a survey. IEEE Trans. on Pattern Analysis and Machine Intelligence 21, 8, 690--706.
60. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In SIGGRAPH '99: Proc. of the 26th annual conference on Computer graphics and interactive techniques, 205--214.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Image representations