A framework for cost-effective communication system for 3D data streaming and real-time 3D reconstruction

ABSTRACT

The methods commonly used for people to converse despite a large geographical distance are either text messaging or video conference. A more immersive communication method which is similar to in-person conversation is more desirable. The closest form is a conversation via the holographic projection of the participants and environment. Two features needed to achieve this include 3D reconstruction of the target and streaming of 3D data. We implement 3D reconstruction and 3D streaming using cost-effective depth cameras. The focus of this work is on how to approach the idea by using such devices to create a standardized platform for the implementation of the system with the features mentioned above. Specifically, the system can capture 3D data from multiple depth sensors, reconstruct a 3D model of the human target to create an avatar, and stream the changes acquired from the sensors to the client to control the avatar in real time.

References

1. Azzedine Boukerche and Richard Werner Nelem Pazzi. 2006. Remote Rendering and Streaming of Progressive Panoramas for Mobile Devices. In Proceedings of the 14th ACM International Conference on Multimedia (MM '06). ACM, New York, NY, USA, 691--694.
2. Yi Chao, Tao Wu, X. Wang, and Guizhou Zheng. 2015. The computation of delaunay triangulation of LiDAR point cloud based on GPU. In 2015 23rd International Conference on Geoinformatics. 1--4.
3. Andrew Feng, Dan Casas, and Ari Shapiro. 2015. Avatar Reshaping and Automatic Rigging Using a Deformable Model. In Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games (MIG '15). ACM, New York, NY, USA, 57--64.
4. Google. 2016. It's your turn. Build the future. (2016). https://developers.google.com/tango/
5. Intel. 2016. Overview of IntelÂő RealSenseâĎć SDK. (2016). https://software.intel.com/en-us/intel-realsense-sdk
6. M. Isenburg and P. Lindstrom. 2005. Streaming meshes. In VIS 05. IEEE Visualization, 2005. 231--238.
7. Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 559--568.
8. Raed Ghannam Jarrar. 2009. Image-based rendering protocols for remote interactive walkthroughs on mobile devices. Master's thesis. University of Ottawa, Canada.
9. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing (SGP '06). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 61--70. http://dl.acm.org/citation.cfm?id=1281957.1281965
10. H. R. Maamar, A. Boukerche, and E. Petriu. 2013. Streaming 3D meshes over thin mobile devices. IEEE Wireless Communications 20 (June 2013), 136--142.
11. Zoltan Csaba Marton, Radu Bogdan Rusu, and Michael Beetz. 2009. On Fast Surface Reconstruction Methods for Large and Noisy Datasets. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Kobe, Japan.
12. A. Ortiz, D. Oyarzun, I. Aizpurua, and J. Posada. 2004. Three-dimensional whole body of virtual character animation for its behavior in a virtual environment using H-Anim and Inverse kinematics. In Proceedings Computer Graphics International, 2004. 307--310.
13. Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Ming-song Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 741--754.
14. Denis Perevalov and Igor (Sodazot) Tatarnikov. 2015. openFrameworks Essentials. Packt Publishing.
15. S. Rusinkiewicz and M. Levoy. 2001. Efficient variants of the ICP algorithm. In Proceedings Third International Conference on 3-D Digital Imaging and Modeling. 145--152.
16. Radu Bogdan Rusu. 2009. Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments. Ph.D. Dissertation. Computer Science department, Technische Universitaet Muenchen, Germany.
17. Radu Bogdan Rusu and Steve Cousins. 2011. 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China.
18. Andy Schmeder. 2008. Everything you ever wanted to know about Open Sound Control. Technical Report.
19. H. Seo, C. Joslin, U. Berner, N. Magnenat-Thalmann, M. Jovovic, J. Esmerado, D. Thalmann, and I. Palmer. 2000. VPARK - a Windows NT software platform for a virtual networked amusement park. In Proceedings Computer Graphics International 2000. 309--315.
20. Ari Shapiro, Andrew Feng, Ruizhe Wang, Hao Li, Mark Bolas, Gerard Medioni, and Evan Suma. 2014. Rapid avatar capture and simulation using commodity depth sensors. Computer Animation and Virtual Worlds 25, 3--4 (2014), 201âĂŞ211.
21. D. Stanev and K. Moustakas. 2014. Virtual Human Behavioural Profile Extraction Using Kinect Based Motion Tracking. In 2014 International Conference on Cyberworlds. 411--414.
22. Oculus VR. 2016. A Closer Look at Oculus Avatars, Oculus First Contact, and Asynchronous Spacewarp. (2016). https://www.oculus.com/blog/a-closer-look/
23. A. Zarean and S. Kasaei. 2016. Human body 3D reconstruction in multiview soccer scenes by depth optimization. In 2016 24th Iranian Conference on Electrical Engineering (ICEE). 1591--1596.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Motion processing

    • Shape modeling

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Collaborative interaction

      • Graphical user interfaces

  • Visualization

    • Visualization design and evaluation methods

• Networks

  • Network protocols