Panoramic Video from Unstructured Camera Arrays

Abstract

We describe an algorithm for generating panoramic video from unstructured camera arrays. Artifact-free panorama stitching is impeded by parallax between input views. Common strategies such as multi-level blending or minimum energy seams produce seamless results on quasi-static input. However, on video input these approaches introduce noticeable visual artifacts due to lack of global temporal and spatial coherence. In this paper we extend the basic concept of local warping for parallax removal. Firstly, we introduce an error measure with increased sensitivity to stitching artifacts in regions with pronounced structure. Using this measure, our method efficiently finds an optimal ordering of pair-wise warps for robust stitching with minimal parallax artifacts. Weighted extrapolation of warps in non-overlap regions ensures temporal stability, while at the same time avoiding visual discontinuities around transitions between views. Remaining global deformation introduced by the warps is spread over the entire panorama domain using constrained relaxation, while staying as close as possible to the original input views. In combination, these contributions form the first system for spatiotemporally stable panoramic video stitching from unstructured camera array input.

References

1. <label>{AAC*06}¿¿</label> Agarwala A., Agrawala M., Cohen M.F., Salesin D., Szeliski R.: Photographing long scenes with multi-viewpoint panoramas. ACM Trans. Graph. Volume 25, Issue 3 2006, pp.853-861. 3
2. <label>{AZP*05}¿¿</label> Agarwala A., Zheng K.C., Pal C., Agrawala M., Cohen M.F., Curless B., Salesin D., Szeliski R.: Panoramic video textures. ACM Trans. Graph. Volume 24, Issue 3 2005, pp.821-827. 3
3. <label>{BA83}¿¿</label> Burt P.J., Adelson E.H.: A multiresolution spline with application to image mosaics. ACM Trans. Graph. Volume 2, Issue 4 1983, pp.217-236. 2, 7
4. <label>{BBPW04}¿¿</label> Brox T., Bruhn A., Papenberg N., Weickert J.: High accuracy optical flow estimation based on a theory for warping</otherTitle>. In <otherTitle>ECCV 2004. 4, 10, 11
5. <label>{d'A07}¿¿</label> d'Angelo P.: Radiometric alignment and vignetting calibration</otherTitle>. <otherTitle>Camera Calibration Methods for Computer Vision Systems 2007. 8
6. <label>{EdDM*08}¿¿</label> Eisemann M., de Decker B., Magnor M.A., Bekaert P., de Aguiar E., Ahmed N., Theobalt C., Sellent A.: Floating textures. Comput. Graph. Forum Volume 27, Issue 2 2008, pp.409-418. 3
7. <label>{EF01}¿¿</label> Efros A.A., Freeman W.T.: Image quilting for texture synthesis and transfer</otherTitle>. In <otherTitle>SIGGRAPH 2001, pp. pp.341-346. 2
8. <label>{FH12}¿¿</label> Felzenszwalb P.F., Huttenlocher D.P.: Distance transforms of sampled functions. Theory of Computing Volume 8, Issue 1 2012, pp.415-428. 7
9. <label>{Gal83}¿¿</label> Galil Z.: Efficient algorithms for finding maximal matching in graphs</otherTitle>. In <otherTitle>CAAP 1983, pp. pp.90-113. 6
10. <label>{HCS13}¿¿</label> He K., Chang H., Sun J.: Rectangling panoramic images via warping. ACM Trans. Graph. Volume 32, Issue 4 2013, pp.79. 2
11. <label>{HZ06}¿¿</label> Hartley A., Zisserman A.: Multiple view geometry in computer vision 2. ed.. Cambridge University Press, 2006. 4, 8
12. <label>{IAB*96}¿¿</label> Irani M., Anandan P., Bergen J., Kumar R., Hsu S.C.: Efficient representations of video sequences and their applications. Sig. Proc.: Image Comm. Volume 8, Issue 4 1996, pp.327-351. 3
13. <label>{JT08}¿¿</label> Jia J., Tang C.-K.: Image stitching using structure deformation. IEEE Trans. Pattern Anal. Mach. Intell. Volume 30, Issue 4 2008, pp.617-631. 3, 6
14. <label>{KCSC10}¿¿</label> Kopf J., Chen B., Szeliski R., Cohen M.F.: Street slide: browsing street level imagery. ACM Trans. Graph. Volume 29, Issue 4 2010. 3
15. <label>{KKDK12}¿¿</label> Kopf J., Kienzle W., Drucker S.M., Kang S.B.: Quality prediction for image completion. ACM Trans. Graph. Volume 31, Issue 6 2012, pp.131:1-131:8. 3, 4
16. <label>{KLD*09}¿¿</label> Kopf J., Lischinski D., Deussen O., Cohen-Or D., Cohen M.F.: Locally adapted projections to reduce panorama distortions. Comput. Graph. Forum Volume 28, Issue 4 2009, pp.1083-1089. 2
17. <label>{KSU04}¿¿</label> Kang S.B., Szeliski R., Uyttendaele M.: Seamless Stitching using Multi-Perspective Plane Sweep. Tech. Rep. MSR-TR-2004-48, Microsoft Research, 2004. 3
18. <label>{KUDC07}¿¿</label> Kopf J., Uyttendaele M., Deussen O., Cohen M.F.: Capturing and viewing gigapixel images. ACM Trans. Graph. Volume 26, Issue 3 2007. 2
19. <label>{LLM*11}¿¿</label> Lin W.-Y., Liu S., Matsushita Y., Ng T.-T., Cheong L.F.: Smoothly varying affine stitching</otherTitle>. In <otherTitle>CVPR 2011. 3
20. <label>{LYTS13}¿¿</label> Liu S., Yuan L., Tan P., Sun J.: Bundled camera paths for video stabilization. ACM Trans. Graph. Volume 32, Issue 4 2013, pp.78. 3
21. <label>{LZPW04}¿¿</label> Levin A., Zomet A., Peleg S., Weiss Y.: Seamless image stitching in the gradient domain</otherTitle>. In <otherTitle>ECCV 2004. 2
22. <label>{Mic14}¿¿</label> Microsoft: Mcrosoft image composite editor, 2014. 9
23. <label>{Nay97}¿¿</label> Nayar S.K.: Catadioptric omnidirectional camera</otherTitle>. <otherTitle>CVPR 1997. 3
24. <label>{NZN07}¿¿</label> Nomura Y., Zhang L., Nayar S.K.: Scene collages and flexible camera arrays</otherTitle>. In <otherTitle>Rendering Techniques 2007, pp. pp.127-138. 3
25. <label>{PCD* 12}¿¿</label> Pirk S., Cohen M.F., Deussen O., Uyttendaele M., Kopf J.: Video enhanced gigapixel panoramas</otherTitle>. <otherTitle>SIGGRAPH Asia Technical Briefs 2012, pp.7:1-7:4. 3
26. <label>{PRRAZ00}¿¿</label> Peleg S., Rousso B., Rav-Acha A., Zomet A.: Mosaicing on adaptive manifolds. IEEE Trans. Pattern Anal. Mach. Intell. Volume 22, Issue 10 2000, pp.1144-1154. 3
27. <label>{RPLP05}¿¿</label> Rav-Acha A., Pritch Y., Lischinski D., Peleg S.: Dynamosaics: Video mosaics with non-chronological time</otherTitle>. In <otherTitle>CVPR 2005. 3
28. <label>{RWSG13}¿¿</label> Rüegg J., Wang O., Smolic A., Gross M.H.: Ducttake: Spatiotemporal video compositing. Comput. Graph. Forum Volume 32, Issue 2 2013, pp.51-61. 8, 9
29. <label>{SCSI08}¿¿</label> Simakov D., Caspi Y., Shechtman E., Irani M.: Summarizing visual data using bidirectional similarity</otherTitle>. In <otherTitle>CVPR 2008. 3
30. <label>{SFW*13}¿¿</label> Schreer O., Feldmann I., Weissig C., Kauff P., Schäfer R.: Ultrahigh-resolution panoramic imaging for format-agnostic video production. Proceedings of the IEEE Volume 101, Issue 1 2013, pp.99-114. 3
31. <label>{SS00}¿¿</label> Shum H., Szeliski R.: Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. IJCV Volume 36, Issue 2 2000, pp.101-130. 2, 3, 4, 5, 11
32. <label>{STP12}¿¿</label> Summa B., Tierny J., Pascucci V.: Panorama weaving: fast and flexible seam processing. ACM Trans. Graph. Volume 31, Issue 4 2012, pp.83. 2
33. <label>{Sze06}¿¿</label> Szeliski R.: Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics and Vision Volume 2, Issue 1 2006. 2
34. <label>{WSI07}¿¿</label> Wexler Y., Shechtman E., Irani M.: Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. Volume 29, Issue 3 2007, pp.463-476. 3, 4
35. <label>{WSZ*14}¿¿</label> Wang O., Schroers C., Zimmer H., Gross M., Sorkine-Hornung A.: Videosnapping: Interactive synchronization for multiple videos. ACM Trans. Graph. Volume 33, Issue 4 2014. 8
36. <label>{ZC12}¿¿</label> Zhi Q., Cooperstock J.R.: Toward dynamic image mosaic generation with robustness to parallax. IEEE Transactions on Image Processing Volume 21, Issue 1 2012, pp.366-378. 3
37. <label>{ZCT*14}¿¿</label> Zaragoza J., Chin T., Tran Q., Brown M.S., Suter D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. Volume 36, Issue 7 2014, pp.1285-1298. 3
38. <label>{ZL14}¿¿</label> Zhang F., Liu F.: Parallax-tolerant image stitching</otherTitle>. In <otherTitle>CVPR 2014. 3, 9, 11
39. <label>{ZMP07}¿¿</label> Zelnik-Manor L., Perona P.: Automating joiners</otherTitle>. In <otherTitle>NPAR 2007, pp. pp.121-131. 3