Abstract
Mobile devices are gradually changing people's computing behaviors. However, due to the limitations of physical size and power consumption, they are not capable of delivering a 3D graphics rendering experience comparable to desktops. Many applications with intensive graphics rendering workloads are unable to run on mobile platforms directly. This issue can be addressed with the idea of remote rendering: the heavy 3D graphics rendering computation runs on a powerful server and the rendering results are transmitted to the mobile client for display. However, the simple remote rendering solution inevitably suffers from the large interaction latency caused by wireless networks, and is not acceptable for many applications that have very strict latency requirements.
In this article, we present an advanced low-latency remote rendering system that assists mobile devices to render interactive 3D graphics in real-time. Our design takes advantage of an image based rendering technique: 3D image warping, to synthesize the mobile display from the depth images generated on the server. The research indicates that the system can successfully reduce the interaction latency while maintaining the high rendering quality by generating multiple depth images at the carefully selected viewpoints. We study the problem of viewpoint selection, propose a real-time reference viewpoint prediction algorithm, and evaluate the algorithm performance with real-device experiments.
- Azuma, R. T. 1997. A survey of augmented reality. Presence: Teleoper. Virtual Environ. 6, 4, 355--385.Google Scholar
Digital Library
- Bao, P. and Gourlay, D. 2004. Remote walkthrough over mobile networks using 3-D image warping and streaming. Vis. Image Signal Processing, IEE Proceedings 151, 4, 329--336.Google Scholar
Cross Ref
- Baratto, R. A., Kim, L. N., and Nieh, J. 2005. Thinc: A virtual display architecture for thin-client computing. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP'05). ACM, New York, 277--290. Google Scholar
Digital Library
- Beigbeder, T., Coughlan, R., Lusher, C., Plunkett, J., Agu, E., and Claypool, M. 2004. The effects of loss and latency on user performance in unreal tournament 2003. In Proceedings of NetGames'04. 144--151. Google Scholar
Digital Library
- Boukerche, A. and Pazzi, R. W. N. 2006. Remote rendering and streaming of progressive panoramas for mobile devices. In Proceedings of the 14th Annual ACM International Conference on Multimedia (Multimedia'06). ACM, New York, 691--694. Google Scholar
Digital Library
- Chang, C.-F. and Ger, S.-H. 2002. Enhancing 3D graphics on mobile devices by image-based rendering. In Proceedings of PCM'02. 1105--1111. Google Scholar
Digital Library
- Chen, S. E. 1995. Quicktime Vr: An image-based approach to virtual environment navigation. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'95). ACM, New York, 29--38. Google Scholar
Digital Library
- Duguet, F. and Drettakis, G. 2004. Flexible point-based rendering on mobile devices. IEEE Trans. Comput. Graph. Appl. 24, 4, 57--63. Google Scholar
Digital Library
- Eisert, P. and Fechteler, P. 2008. Low delay streaming of computer graphics. In Proceedings of the 15th IEEE International Conference on Image Processing, (ICIP 2008). 2704--2707.Google Scholar
- Engel, K., Westermann, R., and Ertl, T. 1999. Isosurface extraction techniques for web-based volume visualization. IEEE Visualiz. Google Scholar
Digital Library
- Epic Games. 2007. Unreal tournament 3. www.unrealtournament.com/.Google Scholar
- Karlton, P. Womack, P., and Leech, J. 2005. Opengl graphics with the x window system (version 1.4). http://www.opengl.org/documentation/specs/.Google Scholar
- Kum, S.-U. and Mayer-Patel, K. 2005. Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl. 1, 128--150. Google Scholar
Digital Library
- Kurillo, G., Vasudevan, R., Lobaton, E., and Bajcsy, R. 2008. A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In Proceedings of the 10th IEEE International Symposium on Multimedia (ISM'08). 111--118. Google Scholar
Digital Library
- Lamberti, F. and Sanna, A. 2007. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Trans. Vis. Comput. Graph. 13, 2, 247--260. Google Scholar
Digital Library
- Levoy, M. 1995. Polygon-assisted jpeg and mpeg compression of synthetic images. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, 21--28. Google Scholar
Digital Library
- Lien, J.-M., Kurillo, G., and Bajcsy, R. 2007. Seleton-based data compression for multi-camera tele-immersion system. In Proceedings of the 3rd International Conference on Advances in Visual Computing (ISVC'07). Vol. 1, Springer-Verlag, Berlin, Heidelberg, 714--723. Google Scholar
Digital Library
- Mark, W. R. 1999. Post-rendering 3D image warping: Visibility, reconstruction, and performance for depth-image warping. Ph.D. dissertation. Department of Computer Science. University of North Carolina at Chapel Hill. Google Scholar
Digital Library
- Marquez, J., Domenech, J., Gil, J., and Pont, A. 2008. Exploring the benefits of caching and prefetching in the mobile web. In Proceedings of WCITD'08.Google Scholar
- McMillan, L. 1997. An image-based approach to three-dimensional computer graphics. Ph.D. Dissertation. Department of Computer Science. University of North Carolina at Chapel Hill. Google Scholar
Digital Library
- McMillan, L. and Bishop, G. 1995. Plenoptic modeling: an image-based rendering system. In Proceedings of the 22nd Annual Conference on Computer Graphics and Internetive Techniques (SIGGRAPH'95). 39--46. Google Scholar
Digital Library
- Noimark, Y. and Cohen-Or, D. 2003. Streaming scenes to mpeg-4 video-enabled devices. IEEE Comput. Graph. Appl. 23, 58--64. Google Scholar
Digital Library
- Ohazama, C. 1999. Opengl vizserver. White paper, Silicon Graphics, Inc.Google Scholar
- Penta, S. K. and Narayanan, P. 2005. Compression of multiple depth maps for ibr. Vis. Comput. 21, 611--618. 10.1007/s00371-005-0337-8.Google Scholar
Cross Ref
- Redert, A., de Beeck, M. O., Fehn, C., IJsselsteijn, W., Pollefeys, M., Gool, L. J. V., Ofek, E., Sexton, I., and Surman, P. 2002. Attest: Advanced three-dimensional television system technologies. In Proceedings of 1st International Symposium on 3D Data Processing Visualization Transmission (3DPVT). IEEE Computer Society, 313--319.Google Scholar
- Riva, O. and Kangasharju, J. 2008. Challenges and lessons in developing middleware on smart phones. IEEE Computer 41, 10, 77--85. Google Scholar
Digital Library
- ScanView. 2003. A system for remote visualization of scanned 3D models. http://graphics.stanford.edu/software/scanview/.Google Scholar
- Schmalstieg, D. 1997. The remote rendering pipeline - managing geometry and bandwidth in distributed virtual environments. Ph.D. dissertation, Institute of Computer Graphics and Algorithms, Vienna University of Technology, Favoritenstrasse 9-11/186, A-1040 Vienna, Austria.Google Scholar
- Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. In Proceedings of the 28th Annual Conference on Computer Graphics and Internetive Techniques (SIGGRAPH'98). 231--242. Google Scholar
Digital Library
- Shi, S., Hsu, C.-H., Nahrstedt, K., and Campbell, R. 2011a. Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, NY, USA, 103--112. Google Scholar
Digital Library
- Shi, S., Nahrstedt, K., and Campbell, R. H. 2011b. Distortion over latency: Novel metric for measuring interactive performance in remote rendering systems. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'11). Google Scholar
Digital Library
- Singhal, S. and Zyda, M. 1999. Networked Virtual Environments: Design and Implementation. ACM Press/Addison-Wesley Publishing Co., New York, NY. Google Scholar
Digital Library
- Smit, F. A., van Liere, R., Beck, S., and Fröhlich, B. 2009. An image-warping architecture for Vr: Low latency versus image quality. In Proceedings of the IEEE Symposium on Virtual Reality (VR). IEEE, 27--34.Google Scholar
- Zhu, M., Mondet, S., Morin, G., Ooi, W. T., and Cheng, W. 2011. Towards peer-assisted rendering in networked virtual environments. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, 183--192. Google Scholar
Digital Library
Index Terms
A real-time remote rendering system for interactive mobile graphics
Recommendations
Real-time remote rendering of 3D video for mobile devices
MM '09: Proceedings of the 17th ACM international conference on MultimediaAt the convergence of computer vision, graphics, and multimedia, the emerging 3D video technology promises immersive experiences in a truly seamless environment. However, the requirements of huge network bandwidth and computing resources make it still a ...
Building low-latency remote rendering systems for interactive 3D graphics rendering on mobile devices
MM '11: Proceedings of the 19th ACM international conference on MultimediaThe recent explosion of mobile devices is changing people's computing behaviors and more and more applications are ported to mobile platforms. However, some applications, such as 3D video tele-immersion and 3D video gaming that require intensive ...
A Framework for Interactive 3D Rendering on Mobile Devices
Although mobile devices have now become an important computing platform, however most of them still lack hardware accelerated graphics. Therefore, interactive 3D rendering on these devices is a difficult task. This paper describes different approaches ...






Comments