Perception of perspective distortions in image-based rendering

Abstract

Image-based rendering (IBR) creates realistic images by enriching simple geometries with photographs, e.g., mapping the photograph of a building façade onto a plane. However, as soon as the viewer moves away from the correct viewpoint, the image in the retina becomes distorted, sometimes leading to gross misperceptions of the original geometry. Two hypotheses from vision science state how viewers perceive such image distortions, one claiming that they can compensate for them (and therefore perceive scene geometry reasonably correctly), and one claiming that they cannot compensate (and therefore can perceive rather significant distortions). We modified the latter hypothesis so that it extends to street-level IBR. We then conducted a rigorous experiment that measured the magnitude of perceptual distortions that occur with IBR for façade viewing. We also conducted a rating experiment that assessed the acceptability of the distortions. The results of the two experiments were consistent with one another. They showed that viewers' percepts are indeed distorted, but not as severely as predicted by the modified vision science hypothesis. From our experimental results, we develop a predictive model of distortion for street-level IBR, which we use to provide guidelines for acceptability of virtual views and for capture camera density. We perform a confirmatory study to validate our predictions, and illustrate their use with an application that guides users in IBR navigation to stay in regions where virtual views yield acceptable perceptual distortions.

References

1. Adams, K. R. 1972. Perspective and the viewpoint. Leonardo 5, 3, 209--217.
2. Bakeman, R. 2005. Recommended effect size statistics for repeated measures designs. Behavior Research Methods 37, 3, 379--384.
3. Banks, M. S., Held, R. T., and Girshick, A. R. 2009. Perception of 3-D layout in stereo displays. Information Display 25, 1, 12--16.
4. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 2001, 425--432.
5. Cooper, E. A., Piazza, E. A., and Banks, M. S. 2012. The perceptual basis of common photographic practice. Journal of Vision 12, 5, 8:1--14.
6. Debevec, P., Yu, Y., and Borshukov, G. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Proceedings of EGWR '98, 105--116.
7. Ernst, M. O., and Banks, M. S. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870, 429--433.
8. Goldstein, E. B. 1987. Spatial layout, orientation relative to the observer, and perceived projection in pictures viewed at an angle. Journal of Experimental Psychology: Human Perception and Performance 13, 2, 256.
9. Kopf, J., Chen, B., Szeliski, R., and Cohen, M. 2010. Street slide: browsing street level imagery. ACM Transactions on Graphics 29, 3, 96:1--8.
10. Lumsden, E. A. 1983. Perception of radial distance as a function of magnification and truncation of depicted spatial layout. Attention, Perception, & Psychophysics 33, 2, 177--182.
11. Morvan, Y., and O'Sullivan, C. 2009. Handling occluders in transitions from panoramic images: A perceptual study. ACM Transactions on Applied Perception 6, 4, 25:1--15.
12. Perkins, D. N. 1972. Visual discrimination between rectangular and nonrectangular parallelopipeds. Attention, Perception, & Psychophysics 12, 5, 396--400.
13. Pharr, M., and Humphreys, G. 2010. Physically Based Rendering: From Theory to Implementation, 2nd ed. Morgan Kaufmann.
14. Pirenne, M. H. 1970. Optics, Painting and Photography. Cambridge University Press.
15. Revelle, W. 2008. psych: Procedures for Psychological, Psychometric, and Personality Research. R package version 1.0-42+.
16. Rosinski, R. R., Mulholland, T., Degelman, D., and Farber, J. 1980. Picture perception: An analysis of visual compensation. Attention, Perception, & Psychophysics 28, 6, 521--526.
17. Sedgwick, H. A. 1991. The effects of viewpoint on the virtual space of pictures. In Pictorial Communication in Virtual and Real Environments, S. R. Ellis, Ed. Taylor & Francis, 460--479.
18. Shum, H. Y., Chan, S. C., and Kang, S. B. 2006. Image-based rendering, vol. 2. Springer.
19. Smith, P. C., and Smith, O. W. 1961. Ball throwing responses to photographically portrayed targets. Journal of Experimental Psychology 62, 3, 223.
20. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3D. ACM Transactions on Graphics 25, 3, 835--846.
21. Steinicke, F., Bruder, G., and Kuhl, S. 2011. Realistic perspective projections for virtual objects and environments. ACM Transactions on Graphics 30, 5, 112:1--10.
22. Stich, T., Linz, C., Wallraven, C., Cunningham, D., and Magnor, M. 2011. Perception-motivated interpolation of image sequences. ACM Transactions on Applied Perception 8, 2, 11:1--25.
23. Todorović, D. 2008. Is pictorial perception robust? the effect of the observer vantage point on the perceived depth structure of linear-perspective images. Perception 37, 1, 106.
24. Todorović, D. 2009. The effect of the observer vantage point on perceived distortions in linear perspective images. Attention, Perception, & Psychophysics 71, 1, 183--193.
25. Vangorp, P., Chaurasia, G., Laffont, P.-Y., Fleming, R. W., and Drettakis, G. 2011. Perception of visual artifacts in image-based rendering of façades. Computer Graphics Forum 30, 4 (Proceedings of EGSR 2011), 1241--1250.
26. Vincent, L. 2007. Taking online maps down to street level. Computer 40, 118--120.
27. Vishwanath, D., Girshick, A. R., and Banks, M. S. 2005. Why pictures look right when viewed from the wrong place. Nature Neuroscience 8, 10, 1401--1410.
28. Wallach, H., and Marshall, F. 1986. Shape constancy in pictorial representation. Attention, Perception, & Psychophysics 39, 233--235.
29. Watt, S. J., Akeley, K., Ernst, M. O., and Banks, M. S. 2005. Focus cues affect perceived depth. Journal of Vision 5, 10, 7:834--862.
30. Yang, T., and Kubovy, M. 1999. Weakening the robustness of perspective: Evidence for a modified theory of compensation in picture perception. Attention, Perception, & Psychophysics 61, 3, 456--467.
31. Yu, J., McMillan, L., and Sturm, P. 2010. Multiperspective modeling, rendering and imaging. Computer Graphics Forum 29, 1, 227--246.
32. Zar, J. H. 2010. Biostatistical Analysis, 5th ed. Prentice Hall.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation