skip to main content
research-article

Visual Comfort for Stereoscopic 3D by Using Motion Sensors on 3D Mobile Devices

Published:21 October 2015Publication History
Skip Abstract Section

Abstract

Advanced 3D mobile devices attract a lot of attentions for 3D visualization nowadays. Stereoscopic images and video taken from the 3D mobile devices are uncomfortable for 3D viewing experiences due to the limited hardware for stereoscopic 3D stabilization. The existing stereoscopic 3D stabilization methods are computationally inefficient for the 3D mobile devices. In this article, we point out that this critical issue deteriorates the 3D viewing experiences on the 3D mobile devices. To improve visual comfort, we propose an efficient and effective algorithm to stabilize the stereoscopic images and video for the 3D mobile devices. To rectify the video jitter, we use the gyroscope and accelerometer embedded on the mobile devices to obtain the geometry information of the cameras. Using a different method than video-content-based motion estimation, our algorithm based on the gyroscope and acceleration data can achieve higher accuracy to effectively stabilize the video. Therefore, our approach is robust in video stabilization even under poor lighting and substantial foreground motion. Our algorithm outperforms previous approaches in not only smaller running time but also the better comfort of the stereoscopic 3D visualization for the 3D mobile devices.

References

  1. M. Abramowitz and I. A. Stegun. 1972. Handbook of Mathematical Functions. Dover Publications, New York, 72--89.Google ScholarGoogle Scholar
  2. R. S. Allison. 2007. Analysis of the influence of vertical disparities arising in toed-in stereoscopic cameras. J. Imag. Sci. Technol. 51, 4, 317--327.Google ScholarGoogle ScholarCross RefCross Ref
  3. Pravin Bhat, C. Lawrence Zitnick, Noah Snavely, Aseem Agrawala, Michael Cohen, Brian Curless, and Sing Bing Kang. 2007. Using photographs to enhance videos of a static scene. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR'07). Eurographics Association, Aire-la-Ville, Switzerland, 327--338. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2011. A perceptual model for disparity. ACM Trans. Graphics 30, 4 (2011), Article 96. DOI:10.1145/2010324.1964991 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. L. Falkenhagen. 1994. Depth estimation from stereoscopic image pairs assuming piecewise continuous surfaces. In Image Processing for Broadcast and Video Production, Springer, 115--127.Google ScholarGoogle Scholar
  6. M. A. Fischler, and R. C. Bolles. 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM 24, 6, 381--395. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Simon Heinzle, Pierre Greisen, David Gallup, Christine Chen, Daniel Saner, Aljoscha Smolic, Andreas Burg, Wojciech Matusik, and Markus Gross. 2011. Computational stereo camera system with programmable control loop. ACM Trans. Graphics 30, 4 (2011), Article 94. DOI:10.1145/2010324.1964989 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. T. S. Huang and A. N. Netravali. 1994. Motion and structure from feature correspondences: A review. Proc. IEEE 82, 2 (1994), 252--268.Google ScholarGoogle ScholarCross RefCross Ref
  9. S. Jain and U. Neumann. 2006. Real-time camera pose and focal length estimation. In Proceedings of the IEEE International Conference on Pattern Recognition. IEEE, 551--555. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. S. B. Kang. 1999. A survey of image-based rendering techniques. Ph.D dissertation, University of North Carolina at Chapel Hill. In VideoMetrics, SPIE, 2--16.Google ScholarGoogle Scholar
  11. Bahadir Karasulu and Sendar Korukoglu. 2013. Performance Evaluation Software: Moving Object Detection and Tracking in Videos. SpringerBriefs in Computer Science, Springer, 63--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Frank L. Kooi and Alexander Toet. 2004. Visual comfort of binocular and 3D displays. Displays 25, 2--3 (2004), 99--108.Google ScholarGoogle ScholarCross RefCross Ref
  13. Marc Lambooij, Wijnand IJsselsteijn, Marten Fortuin, and Ingrid Heynderickx. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. J. Imag. Sci. Technol. 53, 3 (2009), 030201-1--030201-14.Google ScholarGoogle ScholarCross RefCross Ref
  14. Manuel Lang, Alexander, Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, and Markus Gross. 2010. Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graphics 29, 4 (2010), Article 75. DOI:10.1145/1778765.1778812 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Jeehong Lee, Kyu-yeol Chae, and S. Ji. 2012. The 3D video processing method in the stereoscopic camera for mobile devices. In Proceedings of the IEEE International Conference on Emerging Signal Processing Applications (ESPA). IEEE, 139--142.Google ScholarGoogle Scholar
  16. Ken-Yi Lee, Yung-Yu Chuang, Bing-Yu Chen, and Ming Ouhyoung. 2009. Video stabilization using robust feature trajectories. In Proceedings of the IEEE 12th International Conference on Computer Vision. IEEE, 1397--1404.Google ScholarGoogle Scholar
  17. K. Levenberg. 1944. A method for the solution of certain non-linear problems in least squares. Quart. Appl. Math. 2. 164--168.Google ScholarGoogle Scholar
  18. Chun-Wei Liu, Tz-Huan Huang, Ming-Hsu Chang, Ken-Yi Lee, Chia-Kai Liang, and Yung-Yu Chuang. 2011. 3D cinematography principles and their applications to stereoscopic media processing. In Proceeings of the 19th ACM International Conference on Multimedia (MM'11). ACM, New York, 253--262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graphics 28, 3 (2009), Article 44. DOI:10.1145/1531326.1531350 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Wan-Yen Lo., Jeroen van Baar, Claude Knaus, Matthias Zwicker, and Markus Gross. 2010. Stereoscopic 3D copy & paste. ACM Trans. Graphics 29, 6 (2010), Article 147. DOI:10.1145/1882261.1866173 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. Mangiat and J. Gibson. 2012. Disparity remapping for handheld 3D video communications. In Proceedings of the 2012 IEEE International Conference on Emerging Signal Processing Applications (ESPA). IEEE, 147--150.Google ScholarGoogle Scholar
  22. Wojciech Matusik and Hanspeter Pfister. 2004. 3D TV: A scalable for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graphics 23, 3 (August 2004), 814--824. DOI:10.1145/1015706.1015805 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. L. McMillan. 1997. An image-based approach on three-dimensional computer graphics. Ph.D dissertation. University of North Carolina at Chapel Hill, Chapel Hill, N.C. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. P. Mendapara. A. Baradarani, and Q. M. J. Wu. 2010. An efficient depth map estimation technique using complex wavelets. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1409--1414.Google ScholarGoogle ScholarCross RefCross Ref
  25. C. Morimoto and R. Chellapa. 1998. Evaluation of image stabilization algorithms. In Proceedings of the IEEE Internationsl Conference on Acoustics, Speech and Signal Processing. Vol. 5, IEEE, 2789--2792.Google ScholarGoogle Scholar
  26. Nguyen Ho Quoc Phuong, Hee-Jun Kang, Young-Soo Suh, and Young-Sik Ro. 2009. A DCM based orientation estimation algorithm with an inertial measurement unit and a magnetic compass. J. Univ. Comput. Sci. 15, 4, 859--876.Google ScholarGoogle Scholar
  27. V. Rabaud and S. Belongie. 2006. Counting crowded moving objects. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Rahul Raguram, Jan-Michael Frahm and Marc Pollefeys. 2008. A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus. In Proceedings of the 10th European Conference on Computer Vision, Part II (Computer Vision -- ECCV 2008). Lecture Notes in Computer Science, vol. 5303, Springer, Berlin, Heidelberg, 500--513. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. N. Ritter, R. Owens, J. Cooper, R. H. Eikelboom, and P. P. Van Saarloos. 1999. Registration of stereo and temporal images of the retina. IEEE Trans. Med. Imag. 18, 5, 404--418.Google ScholarGoogle ScholarCross RefCross Ref
  30. A. Sabatini. 2006. Quaternion-based extended Kalman filter for determining orientation by inertial and magnetic sensing. IEEE Trans. Biomed. Engin. 53, 7.Google ScholarGoogle ScholarCross RefCross Ref
  31. D. Scharstein and R. Szeliski. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 1, 7--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered depth images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'98). ACM, New York, 231--242. DOI:10.1145/280814.280882 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. J. Shi and C. Tomasi. 1994. Good features to track. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle Scholar
  34. Takashi Shibata, Joohwan Kim, David M. Hoffman, and Martin S. Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. J. Vis. 11, 8--11.Google ScholarGoogle ScholarCross RefCross Ref
  35. B. M. Smith, L. Zhang, H. Jin, and A. Agarwala. 2009. Light field video stabilization. In Proceedings of the IEEE 12th International Conference on Computer Vision. IEEE, 341--348.Google ScholarGoogle Scholar
  36. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. In Proceedings of ACM SIGGRAPH 2006 (SIGGRAPH'06). ACM, New York, 835--846. DOI:10.1145/1179352.1141964 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Filippo Speranza, Wa J. Tam, Ron Renaud, and Namho Hur. 2006. Effect of disparity and motion on visual comfort of stereoscopic images. In Proceedings of the SPIE Stereoscopic Displays and Virtual Reality Systems XIII, Vol. 6055, 94--103.Google ScholarGoogle ScholarCross RefCross Ref
  38. Y. S. Suh. 2010. Orientation estimation using a quarternion-based Kalman filter with adative estimation acceleration. IEEE Trans. Instrum. Measure. 59, 12, 3296--3305.Google ScholarGoogle ScholarCross RefCross Ref
  39. Geng Sun and Nick Holliman. 2009. Evaluating methods for controlling depth perception in stereoscopic cinematography. Proc. SPIE, vol. 7237, Stereoscopic Displays and Applications XX, 72370I (2009). DOI:10.1117/12.807136Google ScholarGoogle ScholarCross RefCross Ref
  40. Wa James Tam, F. Speranza, S. Yano, K. Shimono, and H. Ono. 2011. Stereoscopic 3D-TV: Visual comfort. IEEE Trans. Broadcast. 57, 2, 335--346.Google ScholarGoogle ScholarCross RefCross Ref
  41. Wa James Tam and L. Zhang. 2006. 3D-TV content generation: 2D-to-3D conversion. In Proceedings of the IEEE International Conference on Multimedia and Expo. IEEE, 1869--1872.Google ScholarGoogle Scholar
  42. C. Tomasi, and R. Manduchi. 1998. Blateral filtering for gray and color images. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 839--846. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. N. Uchida, T. Shibahara, T. Aoki, H. Nakajima, and K. Kobayashi. 2005. 3D face recognition using passive stereo vision. In Proceedings of the IEEE International Conference on Image Processing (ICIP'05). IEEE, 950--953.Google ScholarGoogle Scholar
  44. Chiao Wang and Alexander A. Sawchuk. 2008. Disparity manipulation for stereo images and video. Proc. SPIE, vol. 6803, Stereoscopic Displays and Applications XIX, 68031E (February 29, 2008). DOI:10.1117/12.767702Google ScholarGoogle Scholar
  45. J. M. Wang, H. P. Chou, S. W. Chen, and C. S. Fuh. 2009. Video stabilization for a hand-held camera based on 3D motion model. In Proceedings of the 16th IEEE International Conference on Image Processing (ICIP). IEEE, 3477--3480. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. O. Wang, M. Lang, M. Frei, A. Hornung, A. Smolic, and M. Gross. 2011. Stereobrush: Interactive 2D to 3D conversion using discontinuous warps. In Proceedings of the EUROGRAPHICS Symposium on Sketch-Based Interfaces and Modeling. 47--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. L. Zhang and W. J. Tam. 2005. Stereoscopic image generation based on depth images for 3D TV. IEEE Trans. Broadcast. 51, 2, 191--199.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Visual Comfort for Stereoscopic 3D by Using Motion Sensors on 3D Mobile Devices

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 12, Issue 1s
        Special Issue on Smartphone-Based Interactive Technologies, Systems, and Applications and Special Issue on Extended Best Papers from ACM Multimedia 2014
        October 2015
        317 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/2837676
        Issue’s Table of Contents

        Copyright © 2015 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 21 October 2015
        • Accepted: 1 June 2015
        • Revised: 1 April 2015
        • Received: 1 January 2015
        Published in tomm Volume 12, Issue 1s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!