skip to main content
research-article

A real-time remote rendering system for interactive mobile graphics

Authors Info & Claims
Published:16 October 2012Publication History
Skip Abstract Section

Abstract

Mobile devices are gradually changing people's computing behaviors. However, due to the limitations of physical size and power consumption, they are not capable of delivering a 3D graphics rendering experience comparable to desktops. Many applications with intensive graphics rendering workloads are unable to run on mobile platforms directly. This issue can be addressed with the idea of remote rendering: the heavy 3D graphics rendering computation runs on a powerful server and the rendering results are transmitted to the mobile client for display. However, the simple remote rendering solution inevitably suffers from the large interaction latency caused by wireless networks, and is not acceptable for many applications that have very strict latency requirements.

In this article, we present an advanced low-latency remote rendering system that assists mobile devices to render interactive 3D graphics in real-time. Our design takes advantage of an image based rendering technique: 3D image warping, to synthesize the mobile display from the depth images generated on the server. The research indicates that the system can successfully reduce the interaction latency while maintaining the high rendering quality by generating multiple depth images at the carefully selected viewpoints. We study the problem of viewpoint selection, propose a real-time reference viewpoint prediction algorithm, and evaluate the algorithm performance with real-device experiments.

References

  1. Azuma, R. T. 1997. A survey of augmented reality. Presence: Teleoper. Virtual Environ. 6, 4, 355--385.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Bao, P. and Gourlay, D. 2004. Remote walkthrough over mobile networks using 3-D image warping and streaming. Vis. Image Signal Processing, IEE Proceedings 151, 4, 329--336.Google ScholarGoogle ScholarCross RefCross Ref
  3. Baratto, R. A., Kim, L. N., and Nieh, J. 2005. Thinc: A virtual display architecture for thin-client computing. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP'05). ACM, New York, 277--290. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Beigbeder, T., Coughlan, R., Lusher, C., Plunkett, J., Agu, E., and Claypool, M. 2004. The effects of loss and latency on user performance in unreal tournament 2003. In Proceedings of NetGames'04. 144--151. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Boukerche, A. and Pazzi, R. W. N. 2006. Remote rendering and streaming of progressive panoramas for mobile devices. In Proceedings of the 14th Annual ACM International Conference on Multimedia (Multimedia'06). ACM, New York, 691--694. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Chang, C.-F. and Ger, S.-H. 2002. Enhancing 3D graphics on mobile devices by image-based rendering. In Proceedings of PCM'02. 1105--1111. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Chen, S. E. 1995. Quicktime Vr: An image-based approach to virtual environment navigation. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'95). ACM, New York, 29--38. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Duguet, F. and Drettakis, G. 2004. Flexible point-based rendering on mobile devices. IEEE Trans. Comput. Graph. Appl. 24, 4, 57--63. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Eisert, P. and Fechteler, P. 2008. Low delay streaming of computer graphics. In Proceedings of the 15th IEEE International Conference on Image Processing, (ICIP 2008). 2704--2707.Google ScholarGoogle Scholar
  10. Engel, K., Westermann, R., and Ertl, T. 1999. Isosurface extraction techniques for web-based volume visualization. IEEE Visualiz. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Epic Games. 2007. Unreal tournament 3. www.unrealtournament.com/.Google ScholarGoogle Scholar
  12. Karlton, P. Womack, P., and Leech, J. 2005. Opengl graphics with the x window system (version 1.4). http://www.opengl.org/documentation/specs/.Google ScholarGoogle Scholar
  13. Kum, S.-U. and Mayer-Patel, K. 2005. Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl. 1, 128--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Kurillo, G., Vasudevan, R., Lobaton, E., and Bajcsy, R. 2008. A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In Proceedings of the 10th IEEE International Symposium on Multimedia (ISM'08). 111--118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Lamberti, F. and Sanna, A. 2007. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Trans. Vis. Comput. Graph. 13, 2, 247--260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Levoy, M. 1995. Polygon-assisted jpeg and mpeg compression of synthetic images. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, 21--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Lien, J.-M., Kurillo, G., and Bajcsy, R. 2007. Seleton-based data compression for multi-camera tele-immersion system. In Proceedings of the 3rd International Conference on Advances in Visual Computing (ISVC'07). Vol. 1, Springer-Verlag, Berlin, Heidelberg, 714--723. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Mark, W. R. 1999. Post-rendering 3D image warping: Visibility, reconstruction, and performance for depth-image warping. Ph.D. dissertation. Department of Computer Science. University of North Carolina at Chapel Hill. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Marquez, J., Domenech, J., Gil, J., and Pont, A. 2008. Exploring the benefits of caching and prefetching in the mobile web. In Proceedings of WCITD'08.Google ScholarGoogle Scholar
  20. McMillan, L. 1997. An image-based approach to three-dimensional computer graphics. Ph.D. Dissertation. Department of Computer Science. University of North Carolina at Chapel Hill. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. McMillan, L. and Bishop, G. 1995. Plenoptic modeling: an image-based rendering system. In Proceedings of the 22nd Annual Conference on Computer Graphics and Internetive Techniques (SIGGRAPH'95). 39--46. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Noimark, Y. and Cohen-Or, D. 2003. Streaming scenes to mpeg-4 video-enabled devices. IEEE Comput. Graph. Appl. 23, 58--64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Ohazama, C. 1999. Opengl vizserver. White paper, Silicon Graphics, Inc.Google ScholarGoogle Scholar
  24. Penta, S. K. and Narayanan, P. 2005. Compression of multiple depth maps for ibr. Vis. Comput. 21, 611--618. 10.1007/s00371-005-0337-8.Google ScholarGoogle ScholarCross RefCross Ref
  25. Redert, A., de Beeck, M. O., Fehn, C., IJsselsteijn, W., Pollefeys, M., Gool, L. J. V., Ofek, E., Sexton, I., and Surman, P. 2002. Attest: Advanced three-dimensional television system technologies. In Proceedings of 1st International Symposium on 3D Data Processing Visualization Transmission (3DPVT). IEEE Computer Society, 313--319.Google ScholarGoogle Scholar
  26. Riva, O. and Kangasharju, J. 2008. Challenges and lessons in developing middleware on smart phones. IEEE Computer 41, 10, 77--85. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. ScanView. 2003. A system for remote visualization of scanned 3D models. http://graphics.stanford.edu/software/scanview/.Google ScholarGoogle Scholar
  28. Schmalstieg, D. 1997. The remote rendering pipeline - managing geometry and bandwidth in distributed virtual environments. Ph.D. dissertation, Institute of Computer Graphics and Algorithms, Vienna University of Technology, Favoritenstrasse 9-11/186, A-1040 Vienna, Austria.Google ScholarGoogle Scholar
  29. Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. In Proceedings of the 28th Annual Conference on Computer Graphics and Internetive Techniques (SIGGRAPH'98). 231--242. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Shi, S., Hsu, C.-H., Nahrstedt, K., and Campbell, R. 2011a. Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, NY, USA, 103--112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Shi, S., Nahrstedt, K., and Campbell, R. H. 2011b. Distortion over latency: Novel metric for measuring interactive performance in remote rendering systems. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'11). Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Singhal, S. and Zyda, M. 1999. Networked Virtual Environments: Design and Implementation. ACM Press/Addison-Wesley Publishing Co., New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Smit, F. A., van Liere, R., Beck, S., and Fröhlich, B. 2009. An image-warping architecture for Vr: Low latency versus image quality. In Proceedings of the IEEE Symposium on Virtual Reality (VR). IEEE, 27--34.Google ScholarGoogle Scholar
  34. Zhu, M., Mondet, S., Morin, G., Ooi, W. T., and Cheng, W. 2011. Towards peer-assisted rendering in networked virtual environments. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, 183--192. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A real-time remote rendering system for interactive mobile graphics

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 8, Issue 3s
          Special section of best papers of ACM multimedia 2011, and special section on 3D mobile multimedia
          September 2012
          173 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/2348816
          Issue’s Table of Contents

          Copyright © 2012 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 16 October 2012
          • Accepted: 1 June 2012
          • Revised: 1 May 2012
          • Received: 1 January 2012
          Published in tomm Volume 8, Issue 3s

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!