Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment

Abstract

This paper presents a complete system for constructing panoramic image mosaics from sequences of images. Our mosaic representation associates a transformation matrix with each input image, rather than explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to construct a full view panorama, we introduce a rotational mosaic representation that associates a rotation matrix (and optionally a focal length) with each input image. A patch-based alignment algorithm is developed to quickly align two images given motion models. Techniques for estimating and refining camera focal lengths are also presented.

In order to reduce accumulated registration errors, we apply global alignment (block adjustment) to the whole sequence of images, which results in an optimally registered image mosaic. To compensate for small amounts of motion parallax introduced by translations of the camera and other unmodeled distortions, we use a local alignment (deghosting) technique which warps each image based on the results of pairwise local image registrations. By combining both global and local alignment, we significantly improve the quality of our image mosaics, thereby enabling the creation of full view panoramic mosaics with hand-held cameras.

We also present an inverse texture mapping algorithm for efficiently extracting environment maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture-mapped polyhedron surrounding the origin, we can explore the virtual environment using standard 3D graphics viewers and hardware without requiring special-purpose players.

References

1. Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3):283-310.
2. Anandan, P., Sawhrey, H., Shashua, A., and Simoncelli, E. (Eds.) 1995. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, IEEE Computer Society Press.
3. Ayache, N. 1989. Vision Stéréoscopique et Perception Multisensorielle . InterEditions: Paris.
4. Bergen, J.R., Anandan, P., Hanna, K.J., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Second European Conference on Computer Vision (ECCV'92), Santa Margherita Liguere, Italy, Springer-Verlag, pp. 237-252.
5. Black, M.J. and Rangarajan, A. 1996. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. International Journal of Computer Vision, 19(1):57- 91.
6. Burt, P.J. and Adelson, E.H. 1983. A multiresolution spline with applications to image mosaics. ACM Transactions on Graphics, 2(4):217-236.
7. Chen, S. and Williams, L. 1993. View interpolation for image synthesis. In Computer Graphics (SIGGRAPH'93), pp. 279-288.
8. Chen, S.E. 1995. QuickTime VR--an image-based approach to virtual environment navigation. In Computer Graphics (SIGGRAPH'95) , pp. 29-38.
9. Chiang, M.-C. and Boult, T.E. 1996. Efficient image warping and super-resolution. In IEEE Workshop on Applications of Computer Vision (WACV'96), Sarasota, Florida, IEEE Computer Society, pp. 56-61.
10. Danielsson, P.E. 1980. Euclidean distance mapping. Computer Graphics and Image Processing, 14:227-248.
11. Davis, J. 1998. Mosaics of scenes with moving objects. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'98), Santa Barbara, pp. 354-360.
12. Fua, P. and Leclerc, Y.G. 1994. Using 3-dimensional meshes to combine image-based and geometry-based constraints. In Third European Conference on Computer Vision (ECCV'94), Stockholm, Sweden, Springer-Verlag, pp. 281-291.
13. Golub, G. and Van Loan, C.F. 1996. Matrix Computation, 3rd edn. The John Hopkins University Press: Baltimore.
14. Gortler, S.J., Grzeszczuk, R., Szeliski, R., and Cohen, M.F. 1996. The lumigraph. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'96, New Orleans, ACM SIGGRAPH, pp. 43-54.
15. Greene, N. 1986. Environment mapping and other applications of world projections. IEEE Computer Graphics and Applications, 6(11):21-29.
16. Hansen, M., Anandan, P., Dana, K., van der Wal, G., and Burt, P. 1994. Real-time scene stabilization and mosaic construction. In IEEE Workshop on Applications of Computer Vision (WACV'94), Sarasota, Florida, pp. 54-62.
17. Hartley, R.I. 1994. Self-calibration from multiple views of a rotating camera. In Third European Conference on Computer Vision (ECCV'94), Stockholm, Sweden, Springer-Verlag, pp. 471-478.
18. Heckbert, P. 1989. Fundamentals of Texture Mapping and Image Warping. Master's Thesis, The University of California at Berkeley.
19. Horn, B.K.P. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A, 4(4):629-642.
20. Horn, B.K.P., Hilden, H.M., and Negahdaripour, S. 1988. Closed-form solution of absolute orientation using orthonormal matrices. Journal of the Optical Society of America A, 5(7):1127-1135.
21. Irani, M. and Peleg, S. 1991. Improving resolution by image registration. Graphical Models and Image Processing, 53(3):231-239.
22. Irani, M., Anandan, P., and Hsu, S. 1995a. Mosaic based representations of video sequences and their applications. In Fifth International Conference on Computer Vision (ICCV'95), Cambridge, Massachusetts, pp. 605-611.
23. Irani, M., Hsu, S., and Anandan, P. 1995b. Video compression using mosaic representations. Signal Processing: Image Communication , 7:529-552.
24. Kanatani, K. 1994. Analyis of 3-D rotation fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):543-549.
25. Kang, S.B. 1997. A survey of image-based rendering techniques. Technical Report 97/4, Digital Equipment Corporation, Cambridge Research Lab.
26. Kang, S.B. and Weiss, R. 1997. Characterization of errors in compositing panoramic images. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 103-109.
27. Kumar, R., Anandan, P., and Hanna, K. 1994. Shape recovery from multiple views: A parallax based approach. In Image Understanding Workshop, Morgan Kaufmann Publishers: Monterey, CA, pp. 947-955.
28. Kumar, R., Anandan, P., Irani, M., Bergen, J., and Hanna, K. 1995. Representation of scenes from collections of images. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, pp. 10-17.
29. Le Gall, D. 1991. MPEG: A video compression standard for multimedia applications. Communications of the ACM, 34(4):44-58.
30. Lee, M.-C. et al. 1997. A layered video object coding system using sprite and affine motion model. IEEE Transactions on Circuits and Systems for Video Technology, 7(1):130-145.
31. Levoy, M. and Hanrahan, P. 1996. Light field rendering. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'96, New Orleans, ACM SIGGRAPH, pp. 31-42.
32. Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application in stereo vision. In Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), Vancouver, pp. 674-679.
33. Malde, H.E. 1983. Panoramic photographs. American Scientist, 71(2):132-140.
34. Mann, S. and Picard, R.W. 1994. Virtual bellows: Constructing high-quality images from video. In First IEEE International Conference on Image Processing (ICIP-94), Austin, Texas, pp. 363-367.
35. McMillan, L. and Bishop, G. 1995. Plenoptic modeling: An image-based rendering system. In Computer Graphics (SIGGRAPH'95), pp. 39-46.
36. Meehan, J. 1990. Panoramic Photography. Watson-Guptill.
37. Milgram, D.L. 1977. Adaptive techniques for photomosaicking. IEEE Transactions on Computers, C-26(11):1175-1180.
38. Moffitt, F.H. and Mikhail, E.M. 1980. Photogrammetry, 3rd edn. Harper & Row: New York.
39. Murray, R.M., Li, Z.X., and Sastry, S.S. 1994. A Mathematical Introduction to Robotic Manipulation. CRC Press.
40. Nayar, S. 1997. Catadioptric omnidirectional camera. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 482-488.
41. Nielson, G.M. 1993. Scattered data modeling. IEEE Computer Graphics and Applications, 13(1):60-70.
42. Peleg, S. and Herman, J. 1997. Panoramic mosaics by manifold projection. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 338-343.
43. Press, W.H., Flannery, B.P., Teukolsky, S.A., and Vetterling, W.T. 1992. Numerical Recipes in C: The Art of Scientific Computing, 2nd edn. Cambridge University Press: Cambridge, England.
44. Quam, L.H. 1984. Hierarchical warp stereo. In Image Understanding Workshop, Science Applications International Corporation, New Orleans, Louisiana, pp. 149-155.
45. Rosenfeld, A. and Kak, A.C. 1976. Digital Picture Processing. Academic Press: New York.
46. Sato, Y., Wheeler, M., and Ikeuchi, K. 1997. Object shape and reflectance modeling from observation. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'97, Los Angeles, ACM SIGGRAPH, pp. 379-387.
47. Sawhney, H.S. 1994. Simplifying motion and structure analysis using planar parallax and image warping. In Twelfth International Conference on Pattern Recognition (ICPR'94), Jerusalem, Israel, IEEE Computer Society Press, pp. 403-408.
48. Sawhney, H.S. and Ayer, S. 1996. Compact representation of videos through dominant multiple motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):814-830.
49. Sawhney, H.S., Hsu, S., and Kumar, R. 1998. Robust video mosaicing through topology inference and local to global alignment. In Fifth European Conference on Computer Vision (ECCV'98), Freiburg, Germany, Springer-Verlag, pp. 103-119.
50. Shi, J. and Tomasi, C. 1994. Good features to track. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'94), Seattle, Washington, IEEE Computer Society, pp. 593-600.
51. Shum, H.-Y. and Szeliski, R. 1997. Panoramic image mosaicing. Technical Report MSR-TR-97-23, Microsoft Research.
52. Shum, H.-Y. and Szeliski, R. 1998. Construction and refinement of panoramic mosaics with global and local alignment. In Sixth International Conference on Computer Vision (ICCV'98), Bombay, pp. 953-958.
53. Stein, G. 1995. Accurate internal camera calibration using rotation, with analysis of sources of error. In Fifth International Conference on Computer Vision (ICCV'95), Cambridge, Massachusetts, pp. 230-236.
54. Szeliski, R. 1994. Image mosaicing for tele-reality applications. In IEEE Workshop on Applications of Computer Vision (WACV'94), Sarasota, Florida, IEEE Computer Society, pp. 44-53.
55. Szeliski, R. and Kang, S.B. 1995. Direct methods for visual scene reconstruction. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, pp. 26-33.
56. Szeliski, R. 1996. Video mosaics for virtual environments. IEEE Computer Graphics and Applications, 16(2):22-30.
57. Szeliski, R. and Shum, H.-Y. 1997. Creating full view panoramic image mosaics and texture-mapped models. In Computer Graphics (SIGGRAPH'97), pp. 251-258.
58. Tian, Q. and Huhns, M.N. 1986. Algorithms for subpixel registration. Computer Vision, Graphics, and Image Processing, 35:220- 233.
59. Tsai, R.Y. 1987. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation, RA-3(4):323-344.
60. Weghorst, H., Hooper, G., and Greenberg, D.P. 1984. Improved computational methods for ray tracing. ACM Transactions on Graphics, 3(1):52069.
61. Westover, L. 1990. Footprint evaluation for volume rendering. In Computer Graphics (SIGGRAPH '90 Proceedings), Baskett, F. (Eds.), pp. 367-376.
62. Williams, L. 1983. Pyramidal parametrics. Computer Graphics, 17(3):1-11.
63. Wolberg, G. 1990. Digital Image Warping. IEEE Computer Society Press: Los Alamitos, California.
64. Wolf, P.R. 1974. Elements of Photogrammetry. McGraw-Hill: New York.
65. Wood, D.N., Finkelstein, A., Hughes, J.F., Thayer, C.E., and Salesin, D.H. 1997. Multiperspective panoramas for cel animation. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'97, Los Angeles, ACM SIGGRAPH, pp. 243- 250.
66. Xiong, Y. and Turkowski, K. 1997. Creating image-based VR using a self-calibrating fisheye lens. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 237-243.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

    • Image manipulation

      • Image processing

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Mixed / augmented reality

      • Virtual reality